You are his parent. IMO, you can take him out of school for any reason under the sun. Mental health days, cause you want to spend some time with him doing something that could not be addressed on the weekend, sick day, etc. I don't like it when the school tries to manipulate the parent/child relationship. If ONE day (or even two days) of school are going to make/break a child's learning, something is wrong.

If, as an adult, I can determine when I want/need to take time off work, then, why can't a parent make that same determination for thier child? Who is the boss here? The school system/teacher, or the parent?

To further antagonize- IMO, talking to the teacher is the WRONG move. Too many of them might take your criticism/direction out on the child.