I totally agree, a horses halter should never be left on unless you are tacking them up or walking them around. My horses never have thier halters on wether they are in their stall or in the field. I have heard many horror stories of people leaving them on and horses getting caught on things, or foals getting their hind legs stuck in them.
In my opinion a halter should not be on an unsupervised horse.