I am sure it would bother me too but it does seem like he is showing you he loves you in other ways, it may just be hard for him to say it.

Me and my husband would always say "i love you" at the end of a phone conversation, it was such a habit we would even say it when we didn't mean it at that moment and it became kind of a mumble So what I am saying is the words lost their meaning. Do you feel like he loves you when he looks at you?