Another approach is to use a Dremel with a carbide burr, but it will take a steady hand and very careful work to remove the head of the pull rivet. Drilling it could be problematic as well, as you say, it would be easy to enlarge the hole. The problem is the shank is mostly fillling the hole and the drill will not want to center well.
Sonex/Corvair 2016 Finished, Sold 2020
RV-9A 2005 Purchased
IO-320 160 hp
Exempt but happily paid for 2021