The visitors stunned Carlisle by taking a 21st minute lead through Delroy Facey. The former Bolton Wanderers striker made the most of a rare chance when Peter Murphy was caught out ball watching and he lashed Michael Keane's diagonal ball past Keiren Westwood.
Keane had come close himself in the third minute testing Westwood from 20 yards, but that was all the Millers could muster in the first 45 minutes.
Carlisle had by far the better of the first half. Chris Lumsdon's long ball over the top found keeper Neil Cutler wanting, but somehow he blocked Karl Hawley's shot and then repeated the trick to deny Kevin Gall on the follow-up.
Gall then went on a solo run skipping past defenders to fire in a shot that Cutler tipped around the post.
United then produced a moment of magic when they broke down a Rotherham attack and Kevin Harper picked out Hawley, who breezed through the defence only to fire wide.
In the 32nd minute Gall turned his marker but saw his shot fly high and wide and United thought they had levelled when Hawley cut inside to lash a 20-yard shot that hit the bar and Gall couldn't find the target with the rebound.
Carlisle had a serious shout for a penalty just before the break when Harper played in Hawley - he worked space for a shot that appeared to come off a defender's hand but referee Graham Salisbury saw nothing wrong.
United ended the half with Simon Hackney firing a shot just over the bar from outside the area and they kept up the pressure in the second half.
Carlisle dominated the second half and got the goal they deserved in the 64th minute. Gray towered over the visiting defence to head home a Lumsdon corner after a Gall shot had been deflected around the post by Pablo Mills.
The Cumbrians almost made it two ten minutes later but Gray's header went wide.
A frantic last ten minutes saw Cutler make a magnificent instinctive save from Zigor Aranalde and another from Hackney, but it was Westwood who preserved Carlisle's point when he foiled Lee Williamson at the death and Facey blasted the rebound over the bar.