The shock defeat ends the Australian's incredible run of 10 successive grand slam singles match victories and 15-match winning streak overall, after claiming the French Open and Birmingham titles last month.
Barty, 23, was bidding to become the first Australian woman to make the quarter-finals at Wimbledon since Jelena Dokic 19 years ago.
But it was the world number 55 Riske, who withstood 12 aces from the Aussie to secure her tour-best 14th grasscourt win of the season.
"I'm so excited. Ultimately I'm so proud of myself how I handled today's match," Riske said after reaching a maiden grand slam quarter-final with her first-ever win over a world No.1.
"I had to play aggressive and take it to Ash and that's what I did today.
"My game is very well suited to (the grass).
"I hit a flat ball and love coming to the net and I feel at home here. I love being at Wimbledon.
"It's just so special."
Barty started brilliantly against the 29-year-old grasscourt specialist, serving four straight aces in the first game before going on to win the opening set 6-3, in just over half an hour.
But things quickly went downhill for the top seed, with the American powering back in the second set and claiming it 6-2 to turn the match on its head.
In a tense final set, both players held serve before Riske decisively broke at 4-3, before holding to advance to the quarter-finals - and book a potential clash with fellow American Serena Williams, who meets 30th seed Carla Suarez Navarro later on Monday night (AEST).
