Age doesn't matter. My boyfriend is 13 years older than me. But it always seems strange when you are both younger. Like I think back and I was 5 years old when my boyfriend was 18. But I met him when I was 29 (two days before I turned 30) and he was 43. It would have been "not okay" when I was 17 and he was 30, ya know?
My friend is a few years older than her husband and I don't even think about it unless we are talking about high school. Then I remember - oh yeah, he was only a Freshman when we were Seniors.

Maybe you could ask him if he feels weird about the age thing. You might find out he is fine with it, but thinks you have an issue with it. And if you do find out he has an issue, then you will at least know and not wonder everyday.

Good luck!!