Please continue fostering!!! I think the reasons are obvious and were pointed out many times on this thread by Aly and others.

When I fostered kittens I kept them in a large, multi-level cage for half the day and then restricted them to one room when I was home. I felt guilty at first, but when I realized they make a playground out of any size space I just started to enjoy the sweeties and stopped feeling guilty. It wouldn't have been fair to stress out my kitty by having the little monsters invade the entire house and it was a choice I had to make for Nipo's sake.

As someone who has many cats and is fostering you bring loads of knowledge/experience to the process but at the same time you must make restrictions to keep your own cats healthy and safe. I think the good you are doing far outweighs what you may consider the "bad."