I haven't met many people who are against the concept.
What I myself am against is deliberately picking a loaded word to represent a concept and then crying bigotry when people don't agree with your new definition for that word. I have no problem with the concept, but I'd like them to pick a different word for it to keep the discussion unambiguos.
Also, since you mentioned the Oxford Dictionary's definition, I'll give below a definition from the Dictionary of Gender Studies in the Oxford Reference Dictionary of English, as I happen to have free access to the entire reference due to my studies:
The notion of what it means to be male or female. In some languages such as French and German, words have a grammatical gender which may be feminine, masculine, or neutral. Within feminist theory, gender has been contrasted with sex. Gender here expressed the acculturation of an individual into femininity or masculinity as practised in a given culture; that is, it was regarded as socially constructed, whereas sex was viewed as biologically given through female or male bodily traits. This binary distinction has come under increasing pressure since the early 1990s when Judith Butler argued that both gender and sex are cultural constructions, as queer and LGBT people in particular proposed that there are more than two gender identities, and as a result of poststructuralist and postmodern theories that refused binarism as a way of structuring our understanding of the world."
Note, that it gives multiple definitions. It starts with the traditional meaning, i.e male vs female, and then goes on to explain what the word means in modern feminist theory. Even the dictionary of Gender Studies doesn't try to say that the old definition is somehow outdated or wrong. It's just different.