"not much of a man"

That phrase, or variations on it, have been used to justify transitioning. I find myself kinda conflicted about it. On the one hand, at least for me, of course I was "not much of a man" - because, I wasnt one at all. On the other, it almost sounds like "well, you may have failed at being a man, but you can pass at being a woman, since that's easier". What do you guys think?