Crowds chanted "Allahu Akbar" (God is Greatest) as a number of initial visitors entered Jerusalem's Haram al-Sharif compound, known to Jews as the Temple Mount.
The flashpoint holy site includes the Al-Aqsa mosque and the Dome of the Rock.
A call to prayer rang out from Al-Aqsa, but Muslim worshippers held midday prayers outside the site in protest at the new security measures.
Dozens of worshippers gathered to pray at an entrance to the compound next to the Lions' Gate entry to the Old City.
"We reject the changes imposed by the Israeli government," Sheikh Omar Kiswani, Al-Aqsa director, told reporters outside.
"We will not enter through these metal detectors."
Some women wailed and cried while telling people not to enter.
Three Arab Israeli assailants opened fire on Israeli police Friday in Jerusalem's Old City before fleeing to the compound, where they were shot dead by security forces.
Israeli authorities said the gunmen had come from the flashpoint holy site to carry out the attack.
Israel took the highly unusual decision of closing the Al-Aqsa mosque compound for Friday prayers, triggering anger from Muslims and Jordan, the holy site's custodian.
The site remained closed on Saturday, while parts of Jerusalem's Old City were also under lockdown.
Israeli authorities said the closure was necessary to carry out security checks and announced they would reopen the compound Sunday.
Police said Sunday that so far two gates leading to the holy site had been opened, equipped with metal detectors, adding that more than 200 people had entered.
'Avoid any escalation'
Israeli Prime Minister Benjamin Netanyahu spoke of the security measures late Saturday before departing for a trip to Paris.
"This evening I held a discussion with the top security leadership and I instructed that metal detectors be placed at the entrance gates to the Temple Mount," he said.
"We will also install security cameras on poles outside the Temple Mount but which give almost complete control over what goes on there."
Netanyahu spoke by phone with Jordan's King Abdullah II on Saturday night, a statement from Amman said.
Abdullah condemned the attack, but also called on Netanyahu to reopen the Al-Aqsa compound and stressed the need to "avoid any escalation at the site".
Proposals to change security measures at the compound have sparked controversy in the past.
A plan developed in 2015 between Israel and Jordan to install cameras at the site itself fell apart amid disagreement over how they would be operated.
The Haram al-Sharif/Temple Mount is central to the Israeli-Palestinian conflict, with Palestinians fearing Israel may one day seek to assert further control over it.
It is located in east Jerusalem, occupied by Israel in 1967 and later annexed in a move never recognised by the international community.
It is considered the third holiest site in Islam and the most sacred for Jews.
Jews are allowed to visit but not pray there to avoid provoking tensions.
