(I know you been building a long time, and have plenty of experience, so these are just things to consider, maybe spark an idea for you)
If you cant wobble the chuck in any direction by hand, its not bearings. Therefore its your drills or for some reason your chuck teeth, something on them etc. You can get new chucks if you deem it necessary vs buying a whole new handdrill.
If you still think chuck wobble, put it in a vice, make a small wire pointer and check its run out.
Way more likely you have a crappy batch of drills with the bit center ground off center, or the tip was over heated when ground and warped it. Happens all the time with smaller drills.
In anycase (for the noobs out there reading), you should be drilling up to the final size (some people ream, but I'm not a believer in that for small holes up to #30) I go thru tons of 1/8' drills as my starter hole makers. A #40 with a center point off center wont make an off center hole if its being used as an up-size hole maker, at least not that you can tell. You will see it if its a starter hole maker tho.
As a final, I have a dewalt cordless that has developed an off centered chuck to it after many years of use (the chuck jaws are off, not the bearings). I use it when I need to, mostly for big drill stuff, and never for starter holes where hole alignment is an absolute issue.
Also note (and I'm sure you know) that rivets have a squeeze range, not an absolute, so the hole doesnt have to be exact size or roundness (some common sense about limits is obviously needed here).