You're standing at a protest or rally waving your American flag. Perhaps you are there because you are in favor of common sense immigration reform. Maybe it's a protest against using tax money to fund Planned Parenthood. Or it could be a protest against the removal of a Robert E Lee statue, a move which you believe is revisionist and/or apologist.
And while you're standing there peacefully waving your American flag, people come stand next to you and they're waving Nazi flags. They may be wearing an American flag t-shirt or an In God We Trust baseball cap. But they are waving big red flags with swastikas on them. And there's more of them coming to join you in your protest/rally.
I don't agree with a lot of what some people say or believe. It isn't my place to censor that, just like it isn't anyone else's. If it is done peacefully without violence they have as much right to freely speak or show what they believe as anyone else. I would be offended by this, but I don't have the right not to be offended in the same way that others have the right to freely do this.
That doesn't mean there aren't consequences of a more personal nature that comes with having this right.
If it was clear they were there for more than protesting (promoting violence, etc), I'd contact the police as I would in any other rally or protest situation that appeared to be taking on a more violent tone, so to speak.
They wouldn't and don't represent my beliefs but if they're only there to protest peacefully, my option is to walk away from them and/or the entire rally if it's clear that there were more of them than myself.