A police helicopter hovered over the city as speakers slammed Canberra's strong support of Israeli Prime Minister Benjamin Netanyahu and his government.
"We are here to oppose Australia's support for Israel, for a racist apartheid nation," pro-Palestinian author and Australian lawyer Randa Abdel-Fattah told AFP.
"It's disgusting to see that some of our Australian leaders have rolled out the red carpet and welcomed a war criminal into Australia," she said.
"But there are so many Australians who are against this and we are raising our voices loudly and clearly today, to say (Prime Minister) Malcolm Turnbull, and (Foreign Minister) Julie Bishop, 'not in our name'," she added.
A large banner was unfurled depicting Netanyahu with a moustache as Hitler and the word "Fascist" written underneath.
Pro-Israel supporters were also on the streets and The Australian newspaper reported that riot squad officers removed a man who approached the crowd shouting "long live Israel".
The demonstrators, organised by the local Palestine Action Group, tried to march on Netanyahu's hotel but were cut off by police well short of the city centre building over looking the harbour.
Australian-Palestinian Rihab Charida told SBS: "I’ve come here to join the protest because Netanyahu, like every other Israeli prime minister before him, have just blatantly denied Palestinian basic rights and have continued to exploit Palestinian land, with no repercussions."
“It doesn’t surprise me that he’s welcome in Australia, because the Australian government has historically supported Israel," she said.
SBS senior political reporter Daniela Ritorto spoke with protesters:
When asked about the bipartisan support for a two-state solution and the peace process, Ms Charida said Israel's settlement expansion into Palestinian territory over decades had made that unviable.
"Decades of theft of land and denial of basic rights, and rights to access water for example, decades of that type of thing, has rendered the two-state solution impossible anyway," she said.
"So all the settlements that Israel has been building inside the West Bank – the West Bank is no longer territoriality continuous as a result.
"I think what we need to be talking about is resolving the problems that are created by Israeli occupation, for example, the refugee problem. Palestinians make up the largest number of refugees in the world and have been for the longest period. These ongoing issues are the issues that need to be resolved in a peace process."
On the second day of Mr Netanyahu's historic visit, the Israeli Prime Minister and Malcolm Turnbull signed an air services agreement to promote travel between the two countries, and agreed to bilateral cooperation on technological innovation, research and development.
They also released a statement expressing concern over Islamist terrorism in the Middle East.
Ahead of Netanyahu's arrival Wednesday, some 60 business leaders, academics, members of the clergy and former politicians signed a letter saying Australia should not welcome Mr Netanyahu, claiming his policies "provoke, intimidate and oppress" the Palestinians.
