Yeah, he probably meant it casually. Guys can be really obtuse sometimes. I think you should just try to get him to hang out with you. Not necessarily a date, but an after work thing or something.