Homophobia and Atheism

Are there atheists/agnostics who hate gays? I don't mean relatively harmless "gay jokes" (I know they're unacceptable, but I draw a distinction between true bigotry and engaging in offensive humor) or disliking the activity of gay sex, I mean actually hate the fact that a person may be gay.

I used to believe that the hatred of gays was based on religion, and if that was removed people would be more accepting. There are holes in that theory, however. People aren't exactly accepting of other "god-given" differences such as skin color, so I'm not sure why I ever thought "gay-ness" would be any more acceptable.