I'm not comfortable with the censorship, or use of trans issues by neoliberals or whatever (or censorious attitudes of some trans activists on twitter), but trans people are still people, who deserve consideration imv (and such a tiny number of people too (less than 1%), I wonder what the fuss is about). Feminism is used by neoliberals too, but I'm not abandoning that either. Just because neoliberals use a wedge issue, doesn't mean I'll abandon my principles (though it may mean I change political tactics to avoid the ground they prefer).
And 'the science' on the binary of sex (not gender) is not so simple (as I vaguely understand it). Eg sexual dimorphism exists in the brain in a few places - it just so happens that trans people's brains tend to match the gender they aspire to, even before any treatment. Also reliance on the X/Y chromosome arrangement won't help as the many sexual characteristics are spread over various chromosomes in a complex mixture of genes, hormones and epigenetics. These are averages and the science is not completely settled (being science), but it does seem to say that biological sex is some sort of complex spectrum (like gender expression), and so could be considered as giving support to taking trans people seriously (or at least not giving support to rejection of all trans people as poseurs).
Sometimes I feel just as bad for my trans friends these days as I used to for my gay friends in the 80s (with many of the same arguments being used). The bathroom talk is ridiculous - it's basically discriminating against trans people who don't pass (plus there'll be many naturally mannish women/effeminate men who might not pass a bathroom inspection - what then, genital checks?)) - I don't mind if this makes me unpopular here, I have to stick up for my friends.