You need a "palm drill" with a 1/4" chuck. If fact two palm drills are not a bad idea. Keep a sharp #40 jobber drill bit in one drill motor for 3/32" rivets and a #30 jobber bit in the other drill motor for 1/8" rivets. That way you don't have to change drill bits every time you need a different size hole.
Also, 1/4" palm drills have the high RPM's you need for clean holes in thin aluminum. Aircraft Tool Supply Co. has a nice little drill that sells for $79.95 which would be ideal. Part number ATS 6100