using "oops", or oversized rivets, for the skin to spar should have no impact on elevator fit. If you were doing this for the spar to rib rivets, you should still be fine as long as the manufactured head is on the spar side.
You will laugh when you attach the elevators and see how much clearance there really is. But it is good that you question the effects of even minor changes to the construction. Build on!