Another idea..

..is that you could simply buy a rivet shaver bit that fits into your countersink cage. Then you could use your drill (decent ROM) or die grinder (more RPM) to do the shaving. The countersink cage will allow you to "fine tune" the depth of the bit...