An own goal by Hoops' central defender Kaspars Gorkss, three minutes into injury-time, sealed the Pilgrims' comeback.
Argyle had trailed for the entire second half to Rangers' opener, two minutes before the interval, which itself had an element of fortune to it.
Heidar Helguson nodded home from close range after a shot from team-mate Adel Taarabt had cannoned off Argyle centre-back Krisztián Timár into his path.
It seemed as though that would be enough, until the home side won a late, late throw-in on their left-hand side.
The ball was propelled into the penalty area by Gary Sawyer and into the net, with home players Jamie Mackie and Rory Fallon both claiming the goal.
However, TV replays later confirmed that is was a faint touch by Gorkss that carried the ball past goalkeeper Radek Cerny.
Argyle were grateful to their own goalkeeper Romain Larrieu for keeping them in the game.
The Frenchman denied former team-mate Ákos Buzsáky by tipping his curling first-half free-kick around the post, before turning away a goal-bound drive from Wayne Routledge.
However, his most impressive and arguably most important stop came as Argyle left themselves open when they were chasing the game late on.
Rowan Vine raced into space behind Timár and bore down on goal before unleashing a shot that Larrieu got down to quickly.
The Pilgrims were denied a clear penalty when a header from Fallon, after fine wing work by Mackie and Jim Paterson, was deflected wide by a Rangers' player's hand.
After that, Argyle rallied strongly, and Timár saw two shots cleared from the goal-line in quick succession, from Gavin Mahon and substitute Hogan Ephraim, before the drama continued with the late equaliser.