Scott Morrison has come under fire for mounting an unusual argument about why 26 January is a tough date for some people.
The prime minister said the date was important to reflect on how far the country had come.
He spoke about the experience of those aboard the First Fleet, who raised the Union Jack for the first time on 26 January, 1788 after arriving the previous week.
"On Australia Day, it's all about acknowledging how far we've come," Mr Morrison told reporters on Thursday.
"You know, when those 12 ships turned up in Sydney, it wasn't a particularly flash day for the people on those vessels either."
The prime minister is a descendant of William Roberts, who came to Australia as a convict aboard the Scarborough in the First Fleet.
Critics have accused Mr Morrison of drawing a false equivalence between those aboard the First Fleet and the experience of Australia's Indigenous people.
Greens Senator Lidia Thorpe said it was disrespectful and offensive to compare the experience of First Nations people with those aboard the First Fleet ships.
“The prime minister has an opportunity to unite this country, not to divide it. And that starts with telling the truth about this country’s history," she said.
“This country is led by a man whose racism, ignorance and denial of this country’s history is on show.”
Labor's Indigenous Australians spokesperson Linda Burney also scolded the prime minister.
"Suffering is not a competition. What the prime minister has said makes no sense," he said.
"As the leader of the country, he has an example to set for the rest of the nation and he should know better.
"How can we expect to see real progress on issues such as reconciliation and closing the gap when he makes such ignorant and unhelpful comments like this?"
Aboriginal and Torres Strait Islander people were living on the continent for more than 60,000 years before the British arrived.
Indigenous people have since endured widespread massacres, oppression and dispossession.
For many Indigenous and non-Indigenous people, 26 January is a day of sorrow and mourning.
Another of Mr Morrison's critics - Labor MP Graham Perrett - also pointed out there were actually only 11 ships in the First Fleet.
"Luckily our prime minister doesn't have an electorate connected with this event," Mr Perrett quipped.
The prime minister represents the seat of Cook, named in honour of Captain James Cook, a navigator in the Royal Navy who is credited as the first European to discover the east coast of Australia in 1770.
