The 47-member Council of Europe, the continent's leading human-rights body, says the referendum was an uneven contest, citing what it calls an inadequate legal framework and late changes in ballot-counting.
Turkish president Recep Tayyip Erdogan has denounced European monitors' criticism of a weekend referendum as the West's "crusader mentality."
The criticism comes as a narrow and controversial victory in the referendum grants the president sweeping new powers in running the country.
Preliminary results show just over 51 per cent of voters backed what is the biggest overhaul of Turkish politics since the founding of Turkey's modern republic.
But Cezar Florin Preda, from the Organisation for Security and Cooperation in Europe (OSCE) that monitored the vote, says the vote was held on what he calls an "uneven playing field."
"Under the state of emergency put in place after the July 2016 failed coup attempt, fundamental freedoms essential to a generally democratic process were curtailed. One-side dominance in the coverage and restriction of the media reduces voter access to a variety of views."
The OSCE has also criticised the misuse of administrative resources, the obstruction of parties supporting the "no" campaign and a late change by the Turkish electoral board.
The board announced just before the vote it would count ballots not bearing official stamps as valid, undermining important safeguards which were in place against fraud.
Mr Erdogan has rejected international monitors' criticisms of the referendum, though, insisting the vote was the most democratic election witnessed in any Western country.
Addressing supporters outside his palace in Ankara, he said Turkey would ignore the OSCE monitors' findings.
"We have a lot of work to do, and I think all of us know this fact, because it was a struggle against the powers of the world. On the west, the crusaders attacked us. And inside the country, their servants attacked us. But we never gave up. We never gave up, and, as a nation, we stood straight."
Thousands of people have marched through neighbourhoods of Istanbul, some of them chanting "Thief, Erdogan," "No to the presidency" and "This is just the beginning."
Mr Erdogan says concentrating power in the hands of the president is vital to prevent instability.
But critics say the narrowness of his victory could have the opposite effect.
They say it could add to volatility in a country that has recently survived an attempted coup, militant attacks, a Kurdish insurgency, civil unrest and war across its Syrian border.
Mr Erdogan has also repeated at several public appearances he is ready to restore the death penalty, which essentially would end Turkey's long quest to join the European Union.
Germany's foreign minister, Sigmar Gabriel, warns introducing the death penalty would mean the end of EU accession negotiations for Turkey.
"We will only be able to help Turkey with their business and social development if the country stays in the line of democracy, not a non-democratic line. For instance, the introduction of the death penalty would be the end of the negotiations to enter the European Union."
The result also exposes the deep divide between Turkey's urban middle classes, who see their future within Europe, and the rural poor, who favour a strong ruler in Mr Erdogan.
In Istanbul, Turkish political analyst Ersin Kalaycioglu says the election result is likely to create more uncertainty in international relations with Turkey.
"Both economic and political development will suffer by less inclusive, more exclusive institutions, and less institutionalised, more personalised policymaking. So Turkey is moving towards more personalised policymaking. And personalised policymaking will create more uncertainty in Turkey's foreign relations."
President Erdogan survived the coup attempt last year and responded with a heavy crackdown.
More than 47,000 people were jailed, and more than 120,000 were sacked or suspended from government jobs.
The changes announced in the referendum could keep him in power until 2029 or beyond.
Under the changes, mostly to come into effect after the 2019 elections, the president will appoint the cabinet and an undefined number of vice-presidents.
The president will also be able to select and remove senior civil servants without parliamentary approval.
