Ahead of his arrival, state media said he would be sworn in as president at a ceremony on Friday.
Mugabe's iron grip ended in a shock announcement to parliament where MPs had convened to impeach the 93-year-old who dominated every aspect of Zimbabwean public life for decades.
On the streets, the news that his long and often brutal leadership was over sparked wild celebrations which lasted late into the night, with crowds dancing and cheering ecstatically amid a cacophony of car horns.
"Comrade Mnangagwa is coming back today," senior aide Larry Mavhima, told AFP, saying he was expected to brief the media after landing at a military airport outside Harare.
Mnangagwa, 75, was sacked by the president on November 6 in a move that pushed infuriated army chiefs to intervene, triggering a series of events which led to Mugabe's ouster.
A former key Mugabe ally, Mnangagwa fled the country after his dismissal, saying he would not return without guarantees of his safety.
His sacking was the result of an increasingly bitter succession battle with Mugabe's wife Grace, who had been pushing to take over from the ageing leader.
"My decision to resign is voluntary," Mugabe wrote in his resignation letter, expressing his "desire to ensure a smooth, peaceful and non-violent transfer of power".
Party hardliner
In a highly symbolic scene shortly after his resignation, a man took down a portrait of Mugabe from a wall inside the building where MPs had assembled for the extraordinary session to impeach the defiant president.
Another person replaced it with an image of Mnangagwa.
The ruling ZANU-PF party said Mnangagwa could swiftly be named interim president as the country charts a way through the turbulence.
"He will be the one who will be sworn in to be (interim) president for 90 days," said party spokesman Simon Khaya Moyo.
Mnangagwa is a long-time party loyalist who has close ties with the military, with critics describing him as a ruthless hardliner responsible for years of state-sponsored violence.
Mugabe's resignation capped a week in which the military seized control and tens of thousands of Zimbabweans took to the streets in an unprecedented show of dissent against Mugabe.
'He deserves to rest'
"I am so happy that Mugabe is gone, 37 years under dictatorship is not a joke. I am hoping for a new Zimbabwe ruled by the people," Tinashe Chakanetsa, 18, told AFP.
As the news began to sink in late Tuesday, crowds gathered brandishing national flags with many among them praising army chief General Constantino Chiwenga who led the military's power-grab.
Mugabe had ruled Zimbabwe almost unopposed since independence in 1980 -- but his efforts to position his 52-year-old wife Grace as his successor prompted intervention from the military that underpinned his regime.
The monolithic grip of Mugabe -- who was the world's oldest head of state -- was shattered last week when armoured military vehicles took to the streets, blockaded parliament and soldiers placed him under house arrest.
Mugabe's fate, and that of his wife, remain unknown, but ZANU-PF has said he deserved to be treated with respect after leading the country for nearly four decades.
"He deserves to rest and I believe every Zimbabwean agrees with this," said Moyo.
"But I think he had overstayed the hospitality of the people of Zimbabwe."
A new path
Last week's military takeover had all the hallmarks of a coup, but the generals stopped short of forcing Mugabe out.
As the crisis grew, the ZANU-PF party, an instrument of Mugabe's brutal reign, removed him as party leader and began parliamentary proceedings to have him impeached.
"When he saw the turnout (of lawmakers), he probably realised he'd better jump before he was pushed," said Derek Matyszak, an analyst at the Pretoria-based Institute for Security Studies.
British Prime Minister Theresa May said the resignation gave Zimbabwe "an opportunity to forge a new path free of the oppression that characterised (Mugabe's) rule".
And the European Union's diplomatic chief Federica Mogherini called for a solution that "respects the aspirations of the people of Zimbabwe for a more prosperous and democratic future".
The US State Department called it "a historic moment for the people of Zimbabwe... to put an end to Zimbabwe's isolation" saying the country's future should be decided by the people.
Most Zimbabweans have only known life under Mugabe, whose time in power was defined by violent suppression, economic collapse and international isolation.
