I have been in the same boat as you. However, I would choose going to your grandfather's birthday - if it was his last hurrah, I would go anyways - despite the fact he says these things - I would think, if you didn't attend, it would definitely NOT look great on your side with the family. I know it's tough, very tough, but family comes first.