Hoey's dismissal follows the collapse of a £660 million ($906 million) bid to redevelop Wembley as the new English National Stadium.
"That's politics," the BBC quoted Hoey as saying.
Caborn, who has held his seat since 1983, said that his appointment "shows how seriously this government is taking sport".
"I am absolutely delighted," he added. "I want to see sports linked into the community in a much more positive way."
A former director of English first division soccer club Sheffield United, he watches rugby league and cricket and is keen on golf.
His first public engagement as sports minister will be to attend the one-day international between England and Pakistan on Tuesday at Lord's.
"I've been watching cricket since I saw Freddie Trueman bowling for Yorkshire at Bramall Lane," he said.
His Sheffield connections mean he is well known to the new Home Secretary, David Blunkett, which he said would help towards "better joined up government".
Caborn is also well connected with Sir Rodney Walker, the chairman of U.K. Sport, and with John Prescott, the deputy prime minister.
In 1991, Caborn was involved in the staging of the World Student Games in Sheffield. Although that was criticised as an expensive failure, Caborn believes such events can have a long-lasting effect on a region.
"Sport can be used as a major force for regeneration in this country," he said. "There are so many possible spin-offs from sport, using the intellectual property from sport, sport medicine and sport science to generate income for sport, just as they do in Australia.
"We have to have a clear vision of where we see the sports industry going in the next 10 to 15 years."
Hoey, 54, was an often controversial character in her two and a half years in office.
Unusually for a Labour member of parliament, she supports hunting. She also led a call for the reintroduction of standing areas - terracing - at English premier league soccer grounds.
Hoey is among several casualties among junior ministers as Prime Minister Tony Blair puts together a new team for his second-term administration.
On Friday, Blair named Tessa Jowell to replace Hoey's former boss at the DCMS, Chris Smith.
Kate Hoey’s replacement as UK’s Minister for Sport, following Tony Blair’s post election ministerial re-shuffle, is Richard Caborn, a 57-year-old Sheffield member of parliament and former minister at the Department of Trade and Industry.
