Hi Rob,
I think you might have been asking about the edges, and not holes.
The bumps are a result of punching.
On the thin aluminum, the mini scotchbrite wheel on a die grinder will generally remove the bumps or smooth them down to the point where they won't cause any problems.
If you're talking about thicker aluminum, the game is a little different. On the thick stuff, I pull a flat smooth file across the edges. The technique is called draw filing. That very quickly knocks the bumps off and then you're home free to debur without worry.
The file that comes with the Avery kit is aggressive and will remove a whole bunch of material. So I rarely use it. Except where I have a whole lot of work to do. After using it, I come back with my regular smooth file just to smooth it out a bit more, before going on deburring.
I got my file at the "Aviation Dept"

in Lowes. Don't get a cross-cut file they can leave some nasty marks in their wake.
On the topic of draw filing, checkout this video. This guy uses the technique.
http://www.youtube.com/watch?v=Dec78RQsokw
Also, I put Boelube (or some other cutting oil) on my file before I use it. The guy in the video uses chalk. But if you use a cutting oil, the cuts will be cleaner/smoother and you'll be surprised at how easy it is to clean the file. It just takes a rag to wipe away the Aluminum.
Hope that helps,
Phil