Speaking in Seoul where she's in high-level security talks, Senator Payne said responsibility for increasing tensions in the region sits squarely with Pyongyang.
"We cannot allow that kind of brinkmanship to succeed," she told the Seoul Defense Dialogue.
"This is a critical test for the international community."
Senator Payne again stressed China must do more and said Australia wanted the UN Security Council to consider further strong measures.
Her comments come as the federal government is considering how to evacuate the 200,000 Australians in north Asia should tensions with North Korea escalate into war.
Australians on holidays or working in Asia have been told to monitor the crisis closely.
The government has updated its travel advice for South Korea and Japan in response to North Korea's nuclear and ballistic missile tests.
The advice says further provocations by Pyongyang or reactions by other countries cannot be ruled out, and tensions could escalate with little warning.
Prime Minister Malcolm Turnbull has urged all Australians overseas, but especially those in or heading to China, Japan or South Korea, to register with the government's Smart Traveller website.
Mr Turnbull said the risk of war was the greatest since the end of the Korean War and the government was developing emergency plans for Australians in the area.
There would be massive military response if Kim Jong-un carried out his threats, he said.
"It would be a suicide note on the part of Kim Jong-un but it would be a complete catastrophe," Mr Turnbull said.
"Having said that, I remain confident the global community will put more economic pressure on North Korea and that will bring ... the regime to its senses."
