if your not sure can you say "maybe?" For now? (#1 question) If you say "not this minute, but yes later.", does that mean all options for future contact are closed? (#2 question) And, lastly, what does he want? Have you asked what he wants? Can you make a list of questions for him? it would probably ease tension for both of you for the first meeting- bear in mind, he may not have answers or he may not have answers you expect. Maybe you could also make him a list of things you've done and been happy about in your life- things you'd want to share with him?
I often thought of meeting my father but never have- keep us posted on how this goes for you no matter how you decide! good luck!