You're missing the point, which is why he pointed to the irony of it. You're stating you hold the moral high ground while at the same time threatening them with physical violence. The irony is that you're not being much better than them, you're behaving on the same level. You're misconstruing accepting someone else's view point as being a coward.
Speaking from personal experience, someone being racist towards me is funny, and I take it as an opportunity to crack jokes. You'd be surprised how quickly people disassociate you from skin color when they realize all the ideas they had of you in their had were just predispositions they had built up in themselves, and those all melt away over time. They stop looking at you for anything other than who you are when they discover there's a lot in common, while still holding them same racist beliefs. It's an interesting contradiction, but one that I've seen to be very common none-the-less.