I think you need to do what you need to do. Perhaps he needs a lot of one-on-one that the lady can give him. Perhaps he sense that she needs him.

Go with what your heart is telling you. It will all work out in the end.