yep it is each cat owners choice. but when an outdoor cat gets killed by a car, a dog, or some sicko, how will you feel?
also there are diseases they can catch from other cats.
I know you love your cat, and I would hate to see anything happen to him/her. maybe you could take the cat outside with you on a leash?