Thiem, who lost to Nadal in the Monte Carlo quarter-finals last month, was the last player to beat the Spaniard on clay - at last year’s Rome Masters.
Nadal had set a record of winning 50 consecutive sets on a single surface by beating Argentine Diego Schwartzman on Thursday.
Nadal’s exit means Roger Federer will reclaim the number one spot when the revised ATP rankings are released on Monday.
Thiem stuck to the baseline and took his groundstrokes early to unsettle Nadal, who was unusually below par in the opening set on his favourite surface, making several unforced errors.
Although the 24-year-old Austrian failed to convert a set point in the ninth game, he pounced at the second time of asking to seal the first set with an ace.
A couple of mistimed forehands from Nadal handed Thiem an early 3-1 advantage in the second set but the defending champion fought back to draw level after six games.
But Thiem was not to be denied and won 10 of the next 11 points to snuff out any chance of a Nadal comeback, sealing the victory with a powerful cross-court winner in under two hours.
“I haven’t been good enough today,” Nadal told reporters. “He was better than me today. That’s the end of the story.
Rafael Nadal (L) and Dominic Thiem at the conclusion of their Madrid Open match. Source: AAP
“Some days you don’t play as good as you would like to play. Also when that happens it’s because your opponent is doing really well.”
The result means that fifth-seed Thiem becomes only the third player, after Argentine Gaston Gaudio and Serb Novak Djokovic, to beat Nadal on clay on three occasions.
“I had to really increase my level compared to Monte Carlo to beat Rafa here,” Thiem said. “He won 21 matches on clay and 50 sets. So I had to play an extraordinary match, and that’s what I did. I think a very important thing also was today that I went in with the attitude that I can beat him.”
Despite the defeat, Nadal will still be favourite to win the French Open for a record-extending 11th time next month.
In the Madrid semi-finals on Saturday, Thiem faces big-serving South African Kevin Anderson, who has beaten the Austrian in each of their six career encounters indoors and on hard court but with none played on clay.
Sixth seed Anderson, who was runner-up at last year’s U.S. Open, beat Serb Dusan Lajovic 7-6(3) 3-6 6-3 earlier on Friday.
