This is a common criticism of the left, and it is designed to protect members of the right from real charges of racism. Let me be clear, your comment contains no element of racism. The statement "the Tea Party are racists" is inaccurate, but "some Tea Party members, including leaders in the movement, espouse racist views" is accurate.
I could point to Mike Williams, Glenn Beck's "deap-seated hatred of white people" comment (among many other comments), or a host of others for examples of the movement in general (liberal Joan Walsh seems to have a nice summary).
What these all have in common is they are specific examples of racism, not general "they're all racists!" kind of fear-mongering. I commend the Tea Party for dealing with its racist fringe the way it has. I think that is the proper way to handle these issues, not putting your head in the sand and shouting "both sides" (a la Rand Paul when a MoveOn protestor got stomped on by his volunteer).
Liberals weren't clamoring for McCain's birth certificate. We aren't accusing John Boehner of anti-colonial sentiment driving his decisions. Sure, there are boneheaded liberals. You're welcome to bring it up. I'll take a look and decide that Olbermann's off his rocker, or that Media Matters is too whiny (which sometimes I do find).
You have to allow for the possibility that, sometimes (or often), liberals are crying racism not because it's a political gambit to appeal to "stupid voters", but because someone on your side is being a racist.