However, he failed to break into the first-team under former Middlesbrough boss Steve McClaren and was subsequently loaned out to Blackpool, Carlisle, Bournemouth, Bury and Scunthorpe. Gulliver played a vital role in Rushden's central defence throughout the 2004-05 and 2005-06 seasons and was handed the 'Player of the Season' award in their relegation season from League Two.
Diamonds manager Garry Hill commented on the move : "I am really pleased to welcome Phil Gulliver back on board here at Nene Park and I know that the supporters will be celebrating having their one of their favourites returning to Rushden & Diamonds. Phil is here for the right reason - he wants to play for this special club. He has demonstrated his commitment by taking a pay cut to return, that is the type of player I want here, someone who really cares what Nene Park is all about."
In his time at Brunton Park, on loan from Middlesbrough for one month at the start of 2003, Phil made just one start, that appearance coming on New Years Day in a 2-1 defeat at Hartlepool. That performance seemingly not impressing then boss Roddy Collins as Gulliver then spent the next three games sitting on the bench before returning to the Riverside.