Five North Koreans are in the frame for last week's airport killing, drawing a furious response from Pyongyang which has accused Kuala Lumpur of conspiring with "hostile forces" to damage its reputation.
Malaysia recalled its envoy to Pyongyang and also summoned the North Korean ambassador to Kuala Lumpur, who was told his accusation was "baseless", the foreign ministry said in a statement.
"The ministry emphasised that as the death occurred on Malaysian soil under mysterious circumstances, it is the responsibility of the Malaysian government to conduct an investigation to identify the cause of death," it said.
The row erupted when Malaysian police rejected North Korean diplomats' demands to hand over the body of Kim Jong-nam after he was apparently poisoned at Kuala Lumpur's international airport.
Ambassador Kang Chol last week told reporters outside the morgue where Jong-nam's body is being held that Malaysia was being pressured by South Korea in a bid to defame its northern neighbour.
Seoul has pointed the finger of blame for the attack at the North, citing a "standing order" from Kim Jong-Un to kill his elder sibling and a failed assassination bid in 2012 after he criticised the regime.
"The Malaysian government takes very seriously any unfounded attempt to tarnish its reputation," the statement said, after the ambassador's meeting with Deputy Secretary General for Bilateral Affairs Raja Nushirwan Zainal Abidin.
Related reading
Malaysia rejects North Korean demand to hand over body
The Malaysian ambassador in Pyongyang has also been recalled to Kuala Lumpur for consultations, the statement said.
Police said Sunday they believed five North Koreans were involved in the killing, with four having fled the country on the day of the murder.
Officers have already arrested one North Korean living in Kuala Lumpur, an Indonesian woman and her Malaysian boyfriend, as well as a Vietnamese woman.
Three more North Koreans were wanted for questioning, police said.
The drama erupted last week as Jong-nam prepared to board a plane to Macau, where he has been living in recent years. Malaysian police say the 45-year-old was jumped by two women who squirted liquid in his face.
Jong-nam suffered a seizure and died before arriving at hospital.
He was once thought to be the natural successor to his father, the then-North Korean leader Kim Jong-Il.
But after Jong-Il's death in 2011 the succession went instead to his younger half-brother Kim Jong-Un.
Reports of purges and executions have emerged from the current regime as Jong-Un tries to strengthen his grip on power in the face of international pressure over his nuclear and missile programmes.
Related reading
Kim Jong-Nam, the 'Little General' who fell from grace
