The West Australian looked comfortable amongst the yellow jersey group through the first climb of the giant of Provence, but was dropped from the reduced group on the second ascent, 31 kilometres from the finish and still 10 kilometres from the summit.
With no team riders around him and the climb clearly taking its toll, O'Connor had to dig deep to fight onwards and salvage his top-10 chances, dropping three spots to fifth with his previous two minute gap to leader Tadej Pogačar (UAE Team Emirates) extending back out to five minutes and 58 seconds.
Pogačar managed to reel in a late attack by Jonas Vingegaard (Jumbo-Visma) to prevent any serious time being taken away from him on the stage, his lead now over five minutes on second placed Rigoberto Uran (EF Education-Nippo) as Wout van Aert (Jumbo-Visma) produced an incredible performance to win the day.
O'Connor admitted he was very fatigued on the second climb, naming the stage as the hardest of his career to date apart from combating an illness at the Giro d'Italia.
"I was obviously suffering, I think everyone could see that," he said after the stage.
"But I tried to do what I could do to stay within the time. I was exploding, but I did my best and that's what matters.
"I would say that was the hardest in-race moment. I think I was suffering both from just Ventoux itself and the heat a little bit, so here's to better days."
Despite his dreams of catching Pogačar taking a serious hit, O'Connor was upbeat about his situation at this point in the race, not getting too bogged down on results with 10 stages still left.
"It's still amazing," he said of his GC rank.
"It's a position I still didn't think I would be in, so hopefully when we get to the next couple of days I can improve and I can have a good one in Andorra."
The 25-year-old will be eyeing Stage 14's hilly journey from Carcassonne to Quillan as a big opportunity to turn up the heat on the four riders above him and hopefully rediscover the climbing legs that put him on everyone's radar in stage 9.
The Tour de France continues with Stage 12, a 159.4-kilometre stage from Saint-Paul-Trois-Châteaux to Nîmes, with Mark Cavendish potentially in with a chance of equalling the Tour de France all-time stage victory record. Watch from 2130 AEST on SBS, SBS VICELAND and the SKODA Tour Tracker for all the action from the race.
