forbiden fruit?

Just a question for those who identify as cross-dresser. Do you think there is an element of "forbidden fruit" in your desire to cross-dress? I mean, if society had no more problem with a man in a dress than it does with a girl in pants, would it still have the same appeal? Just wondering.