A draft resolution put forward by Japan would have extended the UN-led Joint Investigative Mechanism (JIM) for 30 days to allow time for negotiations on a wider compromise.
But Russia used its veto power to prevent adoption after 12 council members voted in favor of the measure. China abstained, while Bolivia voted no.
It was the 11th time that Russia has used its veto power to stop council action targeting its ally Syria.
"Russia is wasting our time," US Ambassador Nikki Haley told the council after the vote. "Russia has no interest in finding ground with the rest of this council to save the JIM."
"Russia will not agree to any mechanism that might shine a spotlight on the use of chemical weapons by its ally, the Syrian regime," she said.
"It's as simple and shameful as that."
A resolution requires nine votes to be adopted at the council, but five countries - Britain, China, France, Russia and the United States - can block adoption with their veto power.
Japan put its proposal forward after Russia on Thursday vetoed a US-drafted resolution that would have allowed the expert investigators to continue their work for a year.
A separate Russian draft resolution that called for changes to the JIM failed to garner enough support, with just four votes in favour.
"Any extension of the JIM's mandate for us is possible only provided fundamental flaws in its work are rectified," said Russian Ambassador Vassily Nebenzia.
Fictitious investigation
The Russian ambassador accused the JIM's leadership of having "disgraced itself with its fictitious investigation" of the sarin gas attack at the opposition-held village of Khan Sheikhun.
The panel "signed its name on baseless accusations against Syria," he charged.
In a report last month, the JIM blamed the Syrian air force for dropping the deadly nerve agent on Khan Sheikhun, leaving scores dead.
The April 4 attack triggered global outrage as images of dying children were shown worldwide, prompting the United States to launch missile strikes on a Syrian air base days later.
After the veto, the council met behind closed doors at Sweden's request to hear another appeal for a temporary extension, but Russia again refused, diplomats said.
Swedish Ambassador Olof Skoog said council members must "make sure that we are absolutely convinced that we have exhausted every avenue, every effort before the mandate of the JIM expires tonight."
Italian Ambassador Sebastiano Cardi, who holds the presidency, told reporters after the meeting that the council "will continue to work in the coming hours and days constructively to find a common position."
Diplomats, however, acknowledged that the chances of salvaging the panel were slim.
The final efforts turned to finding some technical ruse that would allow the JIM to avoid shutting down and would not require a resolution, diplomats said.
The row over the fate of the chemical weapons inquiry came as the United Nations was preparing a new round of peace talks to open on November 28 in Geneva to try to end the six-year war.
The joint UN-Organisation for the Prohibition of Chemical Weapons (OPCW) panel was set up by Russia and the United States in 2015 and unanimously endorsed by the council, which renewed its mandate last year.
Previous reports by the JIM have found that Syrian government forces were responsible for chlorine attacks on three villages in 2014 and 2015, and that the Islamic State group used mustard gas in 2015.
MORE NEWS:
Jesse Jackson reveals Parkinson's battle