Danny Graham's 12th minute header settled the contest, although the Cumbrians could have won with more to spare.
But Abbott wasn't complaining after only taking charge of the side on Monday following Neil McDonald's shock dismissal.
"I want the job and I will try my hardest to get it," said McDonald's former second in command.
"I'm hoping to take the team on a happy journey and everyone is invited.
"It's been a tough time but I can't go into the reasons for Neil's departure because I don't know.
"But I spoke to him yesterday and I'm sure I will speak to him again."
Abbott certainly appears to be the fans' choice. They were singing and chanting his name throughout the match.
"It was very gratifying to hear my name," added the caretaker boss.
Despite the uncertainty and the upheaval, Carlisle responded magnificently.
They should have been 2-0 up inside the first ten minutes. Kevin Gall was denied by keeper Jim Provett, while Gall headed over from Zigor Aranalde's cross.
But there was no escape for the Shakers when Graham rose magnificently to plant a header from ten yards beyond Provett.
Bury, who were 2-1 winners at Milton Keynes last Saturday, struggled to repeat that form and even Andy Bishop couldn't turn the game their way.
Manager Chris Casper left it until the 64th minute to introduce the Dons' matchwinner
And the Shakers two-goal hero immediately lifted his team and the Gigg Lane crowd.
With his first touch after replacing the ineffective James Dean, last season's leading scorer created a chance for lively Nicky Adams.
Fellow substitute Andy Mangan scuffed another left-foot chance wide of Keiren Westwood's goal as the Shakers looked to take the game into extra time.
But Carlisle held on to give Abbott the perfect start to his new role.
"We were disappointed not to get something out of the game," said Casper. "We created some decent chances. Carlisle flew at us and their goal was well worked.
"We showed a good attitude to get back into the match and we've got to try and build on the result against Milton Keynes against Grimsby on Saturday."