Police say the bodies of boys aged 10 and three, a six-year-old girl and a one-year-old girl were found inside the property in Mantello Drive in southwest suburban Werribee after a blaze broke out around 1am on Sunday.
Their parents along with an eight-year-old boy escape the fire.
All three are in hospital, the man with serious smoke-related injuries, the woman suffering smoke inhalation and the boy with minor injuries.
Detective Senior Sergeant Ashley Ryan says police are unable to yet say whether the fire is suspicious or not.
"Members from the arson explosives squad are here conducting that investigation," he told reporters at the scene.
Investigators were also speaking to the surviving family members in hospital, he said.
"It is a terrible set of circumstances and one that is very difficult for members of the family, and our hearts go out to the family and to the community this morning."
Sgt Ryan said the scene had also been highly distressing for police and emergency services to attend given the circumstances.
He was unable to comment on reports it may have taken as long as 10 minutes to contact triple zero after the blaze erupted.
CFA Commander David Clancy says the premises were well alight by the time crews arrived.
"The firefighters had a fairly arduous task to combat the firefight," he said.
"They were unable to conduct an internal attack due to the heat."
Fire brigade Lieutenant Damien Molloy said it took about 40 firefighters just under an hour to bring the fire under control.
"We were presented with a house that was well alight," he said.
"Fire had already broken through the roof and was already coming out the front door."
Because of the intensity of the blaze, it was too hard for crews to gain access.
The blaze broke out shortly after 1am on Sunday. Source: Richard Carlesso
"With the roof already being compromised and the tiles collapsing, it was unsafe for our members to be able to proceed," he said.
The cave-in had also made it difficult to identify the location of the four deceased children.
With the distress of the parents and eight-year-old boy "it was very difficult to get accurate information early on to be able to focus our search areas initially, externally and then internally", Lt Molloy said.
He said there had been a lot of people in the street at the time but the blaze was well beyond anything a neighbour could help extinguish.
