If I want to blank, but blank makes people dislike me, then I might blank anyway if people are going to accuse me of blanking no matter what I do. Why not?
When blank is being racist, though, that ought to cause incur a pause. If your response to being called a racist no matter what you do is to become racist, that says something about you. It demonstrates to me that either (a) you really want to be a racist but are merely held back by social convention, or (b) you are cool being racist if it pisses the right people off. Neither of these is a flattering look, perceptively or morally. If we assume, at any rate, that racism is bad.
A third possibility is that racism, in and of itself, is orthogonal to our concerns. One of the theoretical problems with racism -- actual racism, I mean, and quite apart from the social / economic / legal sanctions presently applied against those accused of it -- is that it requires you to abjure potential allies in the fights you actually care about -- or worse, treat them as enemies unnecessarily.
Drill down on the Steve-o-sphere, and you will discover that the fight, the one we actually care about, is the fight to secure the blessings of liberty to ourselves and our posterity. A benign interpretation of the Civil Rights movement is the success of what Trumwill calls the "Sunday-Best" strategy: convincing the plurality of white opinion that equal rights for black Americans didn't actually threaten our prospects for the blessings of liberty or otherwise achieving a life well-lived, and might, in fact, enhance those prospects. I once agreed with this myself, much as I once thought immigration restriction didn't advance anything I cared about either.
Of course, if racism -- or, for that matter, not-racism, a la Trumwill -- is your moral postulate, then I don't really have the tools for arguing you out of your position in the context of shared values. If, on the other hand, racism and not-racism are competing instrumentalities, then I must point out that not-racism is failing today as surely as racism failed in 1964.