It is possible to get an item hand cancelled, however that requires going to the PO and seeing the clerk in person. Most of the time people don't do that.

As far as the "different shaped" envelopes, those don't go through the machines. The machinery only handles "normal" envelopes. round, oval, etc envelopes don't run through the machinery. A while ago there was a group that was determined to cause the PO as much grief as possible by mailing things in wierd containers to defeat the machinery.
Haven't heard from them lately, but I'm sure they're still trying