Peter Dutton took aim at Labor for pledging to nearly double the annual humanitarian intake to 27,000 and the Greens for wanting it raised to 50,000.
The coalition supports the existing intake of 13,750 that rises to 18,750 in 2018-19.
"For many people they won't be numerate or literate in their own language let alone English ... these people would be taking Australian jobs and there is no question about that," he said.
Labor and the Greens have labelled the comments xenophobic and deeply offensive.
Prime Minister Malcolm Turnbull didn't directly respond to the minister's remarks when quizzed by reporters in Cairns on Wednesday.
"We have the most successful multicultural society in the world," he said.
He acknowledged that resettling refugees came at a cost to the budget but the government "did not begrudge the money".
"For many people they won't be numerate or literate in their own language let alone English ... these people would be taking Australian jobs and there is no question about that."
'Dutton would make Hanson proud'
Labor accused Mr Dutton of trying to fan the flames of a debate around refugees and asylum seekers.
"This is very ugly indeed," opposition immigration spokesman Richard Marles said, citing the success of refugees such as businessmen Sidney Myer and Frank Lowy.
Opposition Leader Bill Shorten has accused Immigration Minister Peter Dutton of insulting Australia's great migrant history.
"Mr Dutton's comments are comments Pauline Hanson would have been proud to make," Mr Shorten told reporters in Sydney on Wednesday.
Turnbull must sack Dutton: Di Natale
Malcolm Turnbull should immediately sack Immigration Minister Peter Dutton over his controversial claims about refugees, says Greens leader Richard Di Natale.
Senator Di Natale described Mr Dutton's claim that refugees would take Australian jobs and be a burden on the welfare system as "shameful and appalling".
"(The comments) are not just an attack on refugees, they're an attack on families right around the country - families like mine," he told reporters in Sydney on Wednesday.
Former Labor immigration minister Chris Bowen called on Mr Dutton to apologise.
"There are hundreds of thousands of refugees in Australia who have worked hard, educated themselves and their children and they will be shaking their heads ... in disgust," he told ABC Radio.
Greens senator Sarah Hanson-Young was appalled by what she said was the minister's scaremongering.
But Foreign Minister Julie Bishop came to Mr Dutton's defence saying he was pointing out self-evident facts.
It was expensive to resettle people, and refugees should contribute to society by gaining employment, she said.
Cabinet colleague Mathias Cormann said what the Greens were proposing was neither affordable or sustainable.
Last week, government backbencher George Christensen declared he didn't want any of the 12,000 Syrians coming to Australia as part of a special intake resettled in his Queensland regional electorate because they would increase competition for local jobs.
But other rural areas are lining up the welcome mats keen for an economic stimulus and to fill worker shortages in agriculture and meat processing.
NSW Regional councils such as Wagga Wagga, Albury, Coffs Harbour, Wollongong and Newcastle are among those offering to resettle refugee families.
A study last year found resettlement of 170 Karen refugees from Myanmar (Burma) in the rural Victoria town of Nhill had created 70 jobs and added $40 million to the struggling community's economy.
Refugees in numbers
- Australia's annual refugee intake is 13,750
- That rises to 18,750 by 2018-19
- There is also a special intake of 12,000 from the Syrian conflict
- Labor wants to increase the annual intake to 27,000
- The Greens want to take in 50,000
