I would be more worried if it was the same cat's fur on the floor EVERY morning.

The fact that sometimes it is Ally fur and sometimes Conner, suggests that they are deciding on the top cat in the house, rather than bullying or real fighting.

Also are you sure it isn't play fighting. I was shocked when Milly & Milo started play fighting (cos George & Milly didn't play at all)

As they would make noise and fur would fly but there was no blood or injury to either cat, so now I monitor fights but I try not to interfer.

Are they hurting each other ?

As a rule if they are not hurting each other and both cats take the lead in starting the fights, I would ignore it.

But if they are getting injured or you see signs of one cat bullying the other, seperate them when you are not there to supervise them.

All the best.