Labor says Special Minister of State Mal Brough is teetering on the edge over the Ashby-Slipper affair.
And Opposition Leader Bill Shorten says sooner or later, Prime Minister Malcolm Turnbull will have to make a call.
"Mal Brough's position is untenable," Mr Shorten told ABC TV on Sunday.
But Liberal Peter Dutton does not believe that a case has yet been made to sack Mr Brough from the frontbench.
"The most important thing is for the police to conduct their investigation and for that to continue without pressure," he told Sky News.
"It will be resolved one way or another."
Labor is asking the Speaker for a parliamentary privileges committee inquiry into whether Mr Brough misled parliament when he denied asking James Ashby - a then staffer to speaker Peter Slipper - to procure copies of his boss's diary.
Federal police are investigating the Ashby-Brough matter, which carries a penalty of two years in jail.
