I found it a bit offensive though, although you meant nothing of that.

It's sort of like saying you can't stand or you hate fat people, just because they're fat and can't play basketball or something.