There may be a woodruff key in there, some have them, some don't. Whatever is holding it would most certainly not need drilling out.
Edit, you learn something every day....Graham says if it's a pin, you may have to drill it out.😊😊
Graham says try tapping it on the other side to see if it pops up a bit enough to grab hold of it with plyers or drill a very small hole in the pin, insert a small self taping screw and pull.