Lithuanian keeper Eduardas Kurskis endured another calamitous afternoon to effectively hand the points to Caley Thistle.
Russell Duncan had handed Inverness a first-half lead and John Rankin doubled the advantage from the penalty spot on 53 minute
s.
But Christophe Berra pulled one back, before a crazy period ensued at the finish.
Kurskis – who gifted Rangers a winner last weekend – was handed a red card for a stupid second booking and all looked lost for Hearts.
But Andrius Velika hauled them level with a late penalty, before Bayne struck at the death to leave Hearts with just one win in their last nine league matches.
Caley Thistle served early notice of their intent with two Ian Black drives, before Kurskis illustrated early nerves as he hesitated to collect a loose ball.
But Hearts settled down and almost struck after six minutes when Inverness goalkeeper Michael Fraser failed to hold a crisp Velicka close-range strike and Phil McGuire hacked clear.
Four minutes later, Lee Wallace roared down the left, before cutting the ball across goal, where Kestutis Ivaskevicius was denied by a reflex Fraser stop. The ball quickly broke back to the midfielder, whose second effort was this time deflected wide.
Hearts were now in full flow and Audrius Ksanavicius came even closer right in the 11th minute, rattling the crossbar with a sweet 22-yard shot.
But Tynecastle was left stunned in the 21st minute when the visitors broke the deadlock. A neat passing move saw Rankin feed Black and he played in Duncan, who steered a 20-yard drive past the despairing Kurskis.
The goal had taken the wind from Hearts' sails and Rankin was next to go close when he fizzed a shot past.
Brewster's men quickly sought to protect the lead, getting numbers behind the ball yet threatening on the break.
Hearts were dealt another blow just after the half-hour mark, with the lively Ksanavicius limping off after a challenge from Ross Tokely, with Calum Elliot coming on.
But the hosts spurned a great chance to level after 37 minutes when Velicka's cross was dummied by Elliot and the onrushing Andrew Driver could only sidefoot into Fraser's grateful arms.
Despite the opening, Hearts, shorn of Laryea Kingston and Michael Stewart in midfield, were lacking creativity as they struggled to make inroads and were roundly booed off at half-time.
Inverness appeared comfortable to sit back and let Hearts come at them, but Velicka almost had some joy after 51 minutes when Fraser smothered his shot from the edge of the penalty area.
It was to prove a pivotal save as the Highlanders doubled their tally within two minutes.
Marius Zaliukas slipped as Duncan played a through ball and Romanian Marius Niculae raced into the box, only to be caught by Kurskis' outstretched leg.
Referee William Collum was perfectly placed to award a penalty – and caution Kurskis – before Rankin stepped up to tuck the spot-kick into the bottom left-hand corner.
Hearts looked dead and buried, but they gave themselves a lifeline in the 61st minute when Berra headed in Driver's corner.
Yet, bizarrely, Elliot – on the field less than 30 minutes – was replaced seconds later in a double change with Ivaskevicius for Michal Pospisil and Saulius Mikoliunas.
Hearts pushed forward for the equaliser, but they continued to look fragile in defence and the nervy Kurskis twice fumbled shots under his body before recovering.
Fraser denied Mikoliunas with another fine stop as time ticked on, before Berra prodded past.
The game was ebbing away from the Gorgie side and Kurskis failed to help their cause when he was stupidly red-carded five minutes from time.
The Lithuanian was trying to let the ball run out for a goal-kick under pressure from Duncan. He did so, but the pair then crazily squared up on the touchline, with Duncan booked and Kurskis sent packing for a second yellow.
Lee Wallace was drafted into nets with Hearts having used all their subs and it appeared game over.
But, remarkably, they levelled in the 88th minute when Velicka tucked away a penalty after he had been tugged back by Rankin.
The Tynecastle side looked to have rescued a remarkable point, but the visitors had the last laugh when Bayne lashed home a loose ball for the winner in injury-time.
