Scott Morrison has denied equating the conditions faced by First Fleet convicts with the experience of Indigenous Australians in his comments earlier this week, as he moved to calm growing backlash.
But the prime minister argues it is important to recognise the stories of all Australians on 26 January.
"Australia is more than 25 million stories - more than 25 million - and each of us here can trace our stories back into our experience in Australia," he told reporters in Brisbane on Friday.
"All the stories are important, all the stories should be respected and on Australia Day there is an opportunity to do that."
Indigenous Olympic legend Cathy Freeman lashed the prime minister on Thursday, when he spoke about the experience of those aboard the First Fleet, who raised the Union Jack for the first time on 26 January, 1788, after arriving the previous week.
"You know, when those 12 ships turned up in Sydney, it wasn't a particularly flash day for the people on those vessels either," the prime minister said.
For many, 26 January is a day of sorrow and mourning that signals the European invasion of the continent after more than 60,000 years of Indigenous occupation.
"You can't compare the experiences of those 12 (sic) ships that first arrived to this country to what their arrival meant for all generations of Australia's First Nations people," Ms Freeman wrote on Twitter on Friday.
She wasn't the first to criticise the prime minister's comments, with Labor's Indigenous Australians spokeswoman Linda Burney saying "suffering is not a competition".
"How can we expect to see real progress on issues such as Reconciliation and Closing the Gap when he makes such ignorant and unhelpful comments like this?," Ms Burney said on Twitter on Thursday.
But Finance Minister Simon Birmingham leapt to his leader's defence.
Asked whether it was fair to compare scurvy to genocide, Senator Birmingham said he did not want to focus on the past.
"That was more than 200 years ago, convicts being brought out under forced orders from the United Kingdom," he told the ABC.
"I don't want to reflect on what was happening more than 200 years ago in terms of the individual circumstances for many individuals.
"They were pretty rough times for lots of people."
Senator Birmingham said he wanted people to concentrate on what Australia had achieved and the potential for greater success.
"We will achieve that greater success by bringing people together not by dividing them, by embracing the Indigenous heritage of this national and the multicultural heritage of this nation, and by celebrating it together and not trying to segment one over the other."
Home Affairs Minister Peter Dutton has fired a warning shot for people planning to protest on 26 January.
Thousands of protesters have been told police will not tolerate criminal behaviour and will enforce coronavirus restrictions.
"If people are going to protest, then they need to do it within the law," Mr Dutton told the Nine Network.
"They need to do it peacefully and people need to abide by the health directions."
Mr Dutton said state and federal governments had spent the past year trying to protect remote Indigenous communities from COVID-19.
"We don't want to see an outbreak and particularly amongst Indigenous Australians," he said.
"A lot of us have put in a lot of work over the course of this last 12 months to make sure that Indigenous Australians are protected from the virus.
"We don't want all that success unwound."
