The Cumbrians were forced to make an early substitution when skipper Paul Thirlwell failed to recover from a calf injury and was replaced by Danny Graham after just seven minutes. Duffy should have opened the scoring two minutes later when Leon Britton left him unmarked in front of goal but he miscued with only the keeper to beat.
Prompted by young Joe Allen in midfield, the Swans moved the ball around quickly with Britton and Andy Robinson providing the width down the flanks, but Duffy and strike partner Jason Scotland repeatedly failed to hit the target. Duffy headed another Britton cross over the bar from close range as the first half drew to a close.
Within two minutes of the restart the visitors were awarded a penalty when Britton brought down Jeff Smith. Graham confidently struck the spot-kick into the net to give his side a surprise lead.
As the game went into the final quarter, Swans boss Roberto Martinez sacrificed a defender for speedy winger Anderson in an effort to get back on terms, but it was former Middlesbrough striker Graham who once again went close with a rasping drive which skimmed the home crossbar.
The Swans equalised in the 81st minute when Carlisle keeper Keiren Westwood spilled a shot and Duffy reacted the quickest to slot home from an angle. Just over a minute later Anderson was in the right place to deflect in the winner to send the home crowd wild with delight.