Mr Mnangagwa flew into Harare's Manyame airbase from South Africa and met key members of the ruling ZANU-PF there before heading to the nerve-centre of power, State House, for a briefing, his aide Larry Mavhima told AFP.
He will be sworn in as president at an inauguration ceremony on Friday, officials said.
Addressing crowds in Harare gathered outside the ruling ZANU-PF headquarters, Mr Mnangagwa said: "The people have spoken.
"Today we are witnessing the beginning of a new and unfolding full democracy in our country."
Crowds cheering for the return of Mr Mnangagwa to Zimbabwe. Source: AAP
He was last seen in public on Friday and gave a televised address on Sunday but neither he, nor his wife Mrs Grace Mugabe, have been seen since, with their whereabouts unknown.
On the streets, the news that his long and often brutal leadership was over sparked wild celebrations which lasted late into the night, with crowds dancing and cheering ecstatically amid a cacophony of car horns.
Mr Mnangagwa, 75, was sacked by the president on November 6 in a move that pushed infuriated army chiefs to intervene, triggering a series of events which led to Mr Mugabe's ouster.
Excited crowds
Ahead of his arrival, hundreds of people gathered outside ZANU-PF headquarters in Harare in the hope he would address them, some holding placards welcoming him home, while others wore shirts emblazoned with his likeness.
A former key ally of Mr Mugabe, Mr Mnangagwa had fled the country after his dismissal, saying he would not return without guarantees of his safety.
His sacking was the result of an increasingly bitter succession battle with the first lady, who had been pushing to take over from her ageing husband.
In a highly symbolic scene shortly after his resignation, a man took down a portrait of Mr Mugabe from a wall inside the building where MPs had assembled for the extraordinary session to impeach the defiant president.
Another person replaced it with an image of the ousted vice president.
Mr Mnangagwa is a political veteran long-time party loyalist who has served in a host of different cabinet positions since independence in 1980 and who has close ties with the military.
But critics describe him as a ruthless hardliner behind years of state-sponsored violence, warning he could prove just as authoritarian as his mentor.
And Rinaldo Depagne of the International Crisis Group said Mr Mugabe's departure "does not necessarily mean more democracy".
Crowds in Zimbabwe celebrate. Source: AAP
Meeting Zuma
At Harare's Manyame airbase, senior military commanders and a gaggle of journalists were waiting for him as his business jet touched down at around 1400 GMT, AFP correspondents at the scene said.
Before leaving South Africa where he had been staying, Pretoria published a photograph of Mr Mnangagwa shaking hands with President Jacob Zuma following a meeting earlier in the day, with both men grinning broadly.
Mr Mugabe's resignation capped a chaotic week in which the military seized control and tens of thousands of Zimbabweans took to the streets in an unprecedented show of dissent against Mr Mugabe.
"We want our new president to make sure power hungry gangs don't infiltrate," said Talent Chamunorwa, 37, a brick seller.
"We hope to be able to access our money from the bank come December and the US dollar must come back."
He was referring to Zimbabwe's chronic shortage of cash and a mistrusted "bond note" scheme intended to be pegged to the greenback but trading at a lower rate in reality.
Outstayed his welcome
Mr Mugabe had ruled Zimbabwe almost unopposed since independence and, until his exit, he was also the world's oldest serving head of state.
But efforts to position his 52-year-old wife as his successor were his undoing.
Although Mr Mugabe's fate remains unknown, ZANU-PF has said he deserved to be treated with respect after leading the country for nearly four decades.
"He deserves to rest and I believe every Zimbabwean agrees with this," said ruling party spokesman Simon Khaya Moyo.
"But I think he had overstayed the hospitality of the people of Zimbabwe."
Last week's military takeover had all the hallmarks of a coup, but the generals stopped short of forcing Mr Mugabe out.
As the crisis grew, the ZANU-PF party, an instrument of Mr Mugabe's brutal reign, removed him as party leader and began parliamentary proceedings to have him impeached.
'Extraordinary opportunity'
The international community hailed his exit as a chance to reshape Zimbabwe's future, with British Prime Minister Theresa May saying it offered "an opportunity to forge a new path free of the oppression" that characterised Mr Mugabe's rule.
US Secretary of State Rex Tillerson said it offered Zimbabwe "an extraordinary opportunity to set itself on a new path".
And Beijing, which became a major political and economic partner of Harare as it was shunned by the west, said it respected Mr Mugabe's decision, describing him as a "good friend of the Chinese people".
Most Zimbabweans had only known life under Mr Mugabe, whose time in power was defined by violent suppression, economic collapse and international isolation.
