Karen said it perfectly.
I also understand they lost their son, and their grandson is sort of like filling that void. But he is YOUR son and he does need a routine, especially right now. They're going to have to accept seeing him on weekends.
I'm so sorry they are causing you more pain and stress.
Hugs!!