The destroyer, the USS Stethem, sailed less than 12 nautical miles from tiny Triton Island in the Paracel Islands archipelago, which is claimed by China as well as Taiwan and Vietnam, a US official told AFP.
The distance is commonly accepted as consituting the territorial waters of a landmass.
The operation, meant to demonstrate freedom of navigation in disputed waters, came just hours before a scheduled phone call between President Donald Trump and his Chinese counterpart Xi Jinping.
China had dispatched military vessels and fighter planes in response, foreign ministry spokesman Lu Kang said in a statement late Sunday, according to state news agency Xinhua.
"The Chinese side strongly urges the US side to immediately stop such kind of provocative operations that violate China's sovereignty and threaten China's security," the spokesman said.
The statement added that Beijing would continue to take all necessary means to defend national sovereignty and security.
It was the second operation of its kind carried out by the United States since Trump took office and comes days after his administration took a number of steps that seemed sure to strain US-Chinese relations.
Trump on Thursday authorised a $1.3 billion arms sale to Taiwan, which China considers a rebel province. The same day, the US Treasury Department imposed sanctions on a Chinese bank accused of laundering North Korean cash.
Also Thursday, the State Department expressed concern about Beijing's respect for freedom in Hong Kong, on the 20th anniversary of Britain ceding the territory back to China.
And two days earlier, the State Department placed China on a list of the world's worst human trafficking offenders.
A sharp cooling
All those steps added up to a sharp reversal in tone from April, when Xi travelled to Trump's Mar-a-Lago resort in Florida for a first face-to-face meeting that Trump later said had helped build an "outstanding" relationship.
Further positive signs had followed, including an agreement in May on exporting US beef and natural gas to China.
Trump had praised China's efforts to bring pressure on North Korea over its nuclear and missile programs.
But when those efforts failed to produce results -- Pyongyang conducted new missile tests in violation of UN Security Council resolutions -- the American president made his frustration known.
Those efforts had "not worked out," Trump tweeted on June 20, adding, "At least I know China tried!"
Trump is scheduled to speak with Xi on Sunday at 8:45 pm (00h45 GMT Monday), 45 minutes after speaking with Japanese Prime Minister Shinzo Abe.
A growing Chinese presence
The latest US "freedom of navigation" exercise comes as Beijing continues muscular efforts to cement its claim to nearly all of the South China Sea, parts of which are also claimed by Taiwan and Southeast Asian nations including the Philippines, Brunei, Malaysia and Vietnam.
The United Nations says countries can establish the reach of their territorial waters up to a limit of 12 nautical miles.
China has rapidly built reefs in the area into artificial islands capable of hosting military planes.
Freedom of navigation operations are designed to challenge the sovereignty of countries with claims to disputed territory. Washington has challenged annexations of South China Sea islets while advocating for a diplomatic settlement.
On May 25, the USS Dewey guided-missile destroyer sailed less than 12 nautical miles from Mischief Reef -- part of the disputed Spratly Islands in the South China Sea, south of the Paracel Islands.
