Oh, I definitely agree. So he likes hookers and whips and whatnot, it doesn't bother me one bit and I don't think it has anything to do with his job. I bet half of the people that kept bitching about him have similar dirty minds =)
The thing that did bother me was the nazi part, especially considering his background. I know he claimed that it didn't have anything to do with nazis, but it was pretty obvious and I hate the idea of the holocaust being ridiculed in that manner. While that still doesn't concern his ability to do his job, it's definitely morally reprehensible and a disgrace.
Personally, I don't really care wether he resigns or not. I try not to get into F1 politics if possible... they make my head hurt ^^