A dozen people, four of them children, were seriously injured, Miguel Reis, chief surgeon at the hospital in Madeira's capital Funchal, told a press conference.
Four of the injured are foreigners, with French, German, Dutch and Hungarian citizenship, he said.
Tragedy struck at noon when the massive tree toppled into a crowd gathered near Funchal for the Lady of the Mount festival in honour of Madeira's patron saint, Mary.
Amateur video posted online showed the tree smashing down, prompting panicked worshippers to flee.
Catholics around the world mark Assumption Day on August 15, which according to Christian tradition marks the Virgin Mary's ascent into Heaven.
The Lady of the Mount festival honours Madeira's patron saint, Mary. (AAP) Source: AAP
The festival in Madeira draws large crowds every year from the island and beyond.
Worshippers had gathered to light candles in a botanical garden around five kilometres (three miles) from central Funchal.
'Panic'
"We heard a cracking sound and then the tree fell where people were lighting candles. People sitting on the benches near the fountain were also hit," witness Domingos Perestrelo told the SIC Noticias TV station.
"There was a big panic -- it happened in the middle of mass, around a half-hour before the start of the procession. There were a lot of people there."
The fountain is surrounded by several tall trees aged a hundred years or more.
Authorities have opened an investigation into what happened. The mayor of Funchal, Paulo Cafofo, said the tree had not been supported by cables because it was not considered to be at risk.
A security perimeter was set up around the scene, where worried worshippers gathered waiting anxiously for news and offering help to rescue teams.
Miguel Albuquerque, president of the regional government, announced three days of mourning.
Portuguese Prime Minister Antonio Costa offered his condolences to families and friends of the victims, while President Marcelo Rebelo de Sousa arrived at the scene to show support for the bereaved.
Local football team Maritimo Funchal expressed its "profound sadness at the tragedy in Monte".
Madeira, a popular tourist destination thanks to its dramatic coastlines and gentle climate, is the largest of several Portuguese islands in the north Atlantic Ocean.
Located in a volcanic archipelago off the coast of Morocco, it is prone to floods as well as landslides -- which in February 2010 left more than 50 people dead, with Funchal badly hit.
Last year the Lady of the Mount festival was cancelled due to a forest fire.
Madeira has a population of more than 250,000, the vast majority of them Catholic.
Related reading
Sunbathers killed as plane lands on packed Portugal beach
