One woman was killed and 19 injured during violent clashes amid a white nationalist protest.
Two state troopers monitoring from the sky were also killed when their helicopter crashed.
US President Donald Trump's national security adviser says the violence that broke out in Charlottesville "meets the definition of terrorism".
Heather Heyer was killed when a man drove his car into demonstrators protesting against a white nationalist rally in the city.
The driver, 20-year-old James Alex Fields Junior, was charged with multiple counts, including second-degree murder.
HR McMaster believes his actions must be condemned.
"Any time you commit an attack against people to incite fear, it is terrorism. It meets the definition of terrorism. But what this is, what you see here, is someone who is a criminal, who is committing a criminal act, against fellow Americans. A criminal act that may have been motivated, and we'll see what the -- what's turned up in this investigation, by this hatred and bigotry which I mentioned we have to extinguish in our nation. And we have to do that by asking ourselves, what are we teaching our children?"
The US President has been criticised for not explicitly condemning the white nationalists.
Donald Trump instead denounced what he called a display of hatred, bigotry and violence from "many sides".
His former communications director, Anthony Scaramucci, believes those words were too tame.
"Well, I wouldn't have recommended that statement. I think he needed to be much harsher as it related to the white supremacists and the nature of that. I applaud General McMaster for calling it out for what it is. It's actually terrorism."
Professor Larry Sabato, from the University of Virginia, has told the BBC he witnessed some of the protests.
He believes President Trump's response was disgraceful.
"This was outrageous and if you want the test of outrage, look at what the conservative Republican senators and House members are saying. They're all going after Trump, their own leader. They want to distinguish themselves from him. They're saying you need to call this what it is, evil, that's what white supremacist thinking is: evil."
The man who organised the white nationalist rally was mobbed as he tried to publicly respond to the rampage.
Crowds chanted "shame" as Jason Kessler, a blogger based in Charlottesville, began his press conference.
One man pushed Mr Kessler and a woman tackled him after he blamed police for what occured.
"The hate that you here around you, that is the anti-white hate that fuelled what happened yesterday. What happened yesterday was the result of the Charlottesville police officers refusing to do their jobs. They stood down and refused to follow through on security arrangements."
The racial divides in Charlottesville that fuelled Saturday's violence had been replaced by unity on Sunday.
Vigils were held across the city in honour of the victims.
Elected white officials received multiple standing ovations from the black congregation at Mount Zion Baptist Church.
At the service, Virginia Governor Terry McAuliffe preached unity, and offered this response to the white nationalists:
"To the white supremicists and the neo-Nazis who came to our beautiful state yesterday, there is no place for you here in Charlottesville. There is no place for you in Virginia, and there is no place for you in the United States of America. We deplore your hatred, your bigotry, and shame on you. You pretend you are patriots - you are not patriots. You are dividers."
