Define "slight". If you're enlarging the diameters 1/32", which is essentially taking the powder coating off, I agree, you should be fine. You may be able to get away with a little more than that.
Put some primer or paint on the bare steel to keep it from rusting. That bracket ain't easy to get to when it's mounted.