Citing "dissent" with more hardline colleagues, Petry dropped her bombshell at a morning party press conference, catching other key AfD figures by surprise as she abruptly left the room.
The spectacle played out before the media put the spotlight on the tug-of-war within the party between radical and more moderate forces at the top, and raised questions on how far right it planned to position itself.
Although its beginnings as an anti-euro party were rooted in populism, the AfD's rhetoric veered further right in the run-up to Sunday's elections.
Key members challenged Germany's culture of atonement over World War II and the slaughter of six million Jews in the Holocaust.
It also rolled out provocative posters declaring "Burkas? We prefer bikinis" and "New Germans? Let's make them ourselves", featuring a heavily pregnant white woman, to push its Islamophobic and anti-migrant campaign.
Outraged mainstream politicians have heaped on criticism, including Foreign Minister Sigmar Gabriel who had labelled leaders of the party "real Nazis".
But even Petry herself had openly criticised one of her party's two key candidates Alexander Gauland over his claim that Germany should be proud of its war veterans, saying that would lead voters to shun the party.
On Monday, seated next to Gauland, she declared that "there is dissent in the AfD over the issues".
"I decided after careful reflection that I will not sit with the (AfD) parliamentary group" in the Bundestag, said Petry who added she will serve as an independent MP.
The open squabbling put a damper on the party's success, and pointed to a potentially rocky future for the party in parliament.
'Get our country back'
Alice Weidel, another top AfD candidate in the election, revealed that Petry had not spoken to either her nor Gauland "for months" even though the duo were the leading faces of the AfD's campaign.
For now, Weidel and Gauland appear to have prevailed, as they basked in the fact that the party has become Germany's third biggest political force with 12.6 percent of the vote.
It even came in just on top of Chancellor Angela Merkel's Christian Democratic Union in the eastern state of Saxony, home of the anti-Islam PEGIDA protest movement, as it lured away voters angry with the arrival in Germany of more than a million asylum seekers since 2015.
Gauland, who had also come under fire from Petry over his vow to "go after" Merkel and her government, stood firm on his tone.
"No one would be surprised in the British parliament if someone said they would go after the government," he said, adding that "of course one must go after a government, go after it in a parliamentary debate."
Gauland also minced no words about his party's anti-migrant platform, declaring that "we will get our country back."
"I don't want to lose Germany to an invasion of foreign people from foreign cultures," he vowed.
Reiterating his opposition to double nationality, Gauland took aim at ethnic Turks in Germany, saying he had "little understanding" for those who have German citizenship but who still voted in favour of a referendum in Turkey to increase President Recep Tayyip Erdogan's powers.
"On that, one can at least ask the question whether these people have really arrived in this country and whether they are really part of the German society," he said.
'Nazi smears are cheap'
Analysts said breaching such taboos over German identity could well become the norm in the Bundestag with dozens of AfD MPs seated on the opposition benches.
But the daily Sueddeutsche Zeitung cautioned against just dismissing the AfD as extremists.
"Nazi smears against the AfD are cheap -- dealing with them in parliament will require a great deal more energy and imagination," it said.
"The other parties must distance themselves from the AfD while endeavouring from day one to win back voters from the AfD," it said, adding that "this is a serious test for German democracy."
