The demonstration in an ultra-Orthodox neighbourhood of the city was organised by a particularly hardline group known as the Eda Haredit.
Rabbis gave speeches in Yiddish in the Mea Shearim neighbourhood, while a banner read: "We're Jews and therefore will not enlist in the Zionist army."
The protest became violent when demonstrators blocked roads and resisted efforts to disperse them by riot police, mounted officers and water cannon.
"Eight rioters who used violence against police were arrested," a police statement said in Hebrew.
"They lay down in the road, shouting slogans against the police, some of them threw stones at police," it added.
Last week's Israeli Supreme Court decision strikes down a law exempting ultra-Orthodox men engaged in religious study from military service, saying it undermines equality.
Israeli police use water cannon to disperse Ultra-Orthodox Jewish demonstrators as they protest against army recruitment Source: AAP
The decision raises the possibility that they could be forced into service, a highly contentious proposition with political implications.
However, the court suspended its decision for one year to allow for preparations for the new arrangement -- which also provides the government with the opportunity to pass a new law.
Ultra-Orthodox political parties and their allies in government are likely to draft new legislation that could seek to override the court ruling and keep the exemption in place.
The ultra-Orthodox parties form a key part of Prime Minister Benjamin Netanyahu's governing coalition, and have often acted as kingmakers in Israeli politics.
The issue is part of a decades-old debate over whether young ultra-Orthodox men studying at seminaries should undergo compulsory military service like the rest of Israel's Jewish population.
Israeli Border Police arrest an Ultra-Orthodox Jewish demonstrators as they protest against army recruitment in Jerusalem, Source: AAP
After reaching the age of 18, men must serve two years and eight months, and women must serve for two.
The ultra-Orthodox oppose serving for a variety of reasons, with the most extreme believing a Jewish state is not allowed before the coming of the Messiah.
Others argue that seminary study is just as important to Israel as military service or that ultra-Orthodox soldiers would be confronted with salty language and other irreligious behaviour.
