No, you're not being a baby, far from it. You have a brain tumor that must be monitored closely and dealt with when the time comes. That is NOT insignificant. If you're in the US and work for a large enough employer, you may qualify for FMLA status. That allows you to take up to 12 weeks per year off (unpaid unless you have that much sick time saved up) due to a serious medical condition (consecutive or intermittent days). Of course, there's a pile of paperwork you and your doctor would have to do, and it won't stop your boss from being insensitive, but it will help protect your job and you might sleep better. Check with your HR or benefits office.
I understand your mixed feelings. I felt the same way and it was the root of most of my anxiety. Once I made the decision to move forward with treatment, my anxiety started to let up. I wanted to wait and watch, but it's just not right for me.
Will be thinking of you on the 13th. Please let us know how it goes.