There has been a string of such agreements through Syria's six-year civil war. They have been touted by the government as the best way to end the fighting but have been controversial with the rebels who say they are starved out.
Critics say the population movements are permanently changing the ethnic and religious map, but in an exclusive interview with AFP on Wednesday President Bashar al-Assad insisted they were only temporary and people would return to their homes once the "terrorists" had been defeated.
The evacuation of the four towns - two besieged by the army, and two by the rebels - had been due to start on April 4.
But implementation of the deal brokered by rebel supporter Qatar and regime ally Iran late last month was repeatedly delayed.
At least 80 buses left the government-held towns of Fuaa and Kafraya in Idlib province in the northwest, an AFP correspondent in rebel-held territory said.
They arrived at a marshalling point in Rashidin, west of government-held second city Aleppo.
Most of the evacuees from the two mainly Shiite towns were women, children or elderly people.
Dozens of rebel fighters, including Sunni extremists of Al-Qaeda's former Syria affiliate, Fateh al-Sham Front, stood guard, the correspondent reported.
All of Idlib province bar the two towns is held by an increasingly uneasy alliance between the jihadists and Islamist rebels.
Waiting in the cold
A civilian who was travelling in one of the evacuation buses from the rebel-held towns of Madaya and Zabadani said the operation began at around 6.00 am (0300 GMT).
"We just left now - around 2200 people in around 65 buses," Madaya resident Amjad al-Maleh told AFP by telephone.
"Most of the passengers are women and children who started gathering yesterday evening and spent the night in the cold waiting."
He said that rebel fighters among the evacuees had been allowed to keep their light weapons.
More than 30,000 people are expected to be evacuated under the deal, which began on Wednesday with an exchange of prisoners between rebels and government forces.
Watch: Theresa May: Syria likely behind gas strike
All 16,000 residents of Fuaa and Kafraya are expected to leave, heading to government-held Aleppo, the coastal province of Latakia or Damascus.
Civilian residents of Madaya and Zabadani will reportedly be allowed to remain if they choose. Those who opt to leave with the fighters will head to rebel-held territory in Idlib.
The four towns are part of a longstanding agreement reached in 2015 that requires aid deliveries and evacuations to be carried out to all areas simultaneously.
But access has been limited, with food and medical shortages causing malnutrition, illness and even death among besieged residents.
The UN says 4.72 million Syrians are in so-called hard-to-reach areas, including 600,000 people under siege, mostly by the Syrian army, but also by rebels or the Islamic State group.
'Wouldn't worry about it'
There has been a series of evacuations in recent month, mostly around the capital Damascus but also from the last rebel-held district of Syria's third city Homs.
The mainly Suuni rebels have charged that Assad's Alawaite minority-dominated regime is deliberately forcing civilians to leave to alter the country's sectarian map in its favour.
But in Wednesday's interview, the president told AFP that it was the rebels who were driving people to leave their homes.
"We didn't choose it," he said.
"We wish that everyone could stay in his village and his city, but those people like many other civilians in different areas were surrounded and besieged by the terrorists, and they've been killed on [a] daily basis, so they had to leave.
"But of course they're going to go back to their cities after the liberation," he said.
"Talking about demographic changes is not in the sake or in the interest of the Syrian society when it's permanent. As long as it's temporary, we wouldn't worry about it."
