Thank you for the reply. What I ended up doing was simply melting some of the plastic I had cut from the screen area into the hole, then sanding it smooth. After two passes of that I was able to redrill the holes and get a result that wasn't necessarily great, but passible.

(Don't mind the ziptie, it's only temporary)
Unfortunately just being careful wouldn't have been quite enough. I pulled the buttons and wells from a cheap Sega controller which I liked the layout of, and as a result the buttons simply wouldn't fit had I kept them lined up to the original holes.
Oh well, that's what I get for wanting to do things my own way instead of just using a premade board. Next time I won't be so stubborn, I think.