Assuming your talking about non-dimpled -4 rivets with #30 holes?
In a perfect world, a #30 drill bit leaves a .1285 hole A -4 rivet is .125. In other words, if it's perfect, the hole is only .0035 bigger than the rivet you're trying to stuff in it.
Obviously there's going to be some slop from holding the drill crooked, machining tolerance of the bits, rivets, etc. but yes, the primer can make the hole too tight to get a rivet to drop in. (FYI- per mil-spec, the allowable hole size for a -4 rivet is .125-.135)
I typically prime before match drilling, so there's no primer in the hole. But if you choose to do it the other way it's not a big deal. If you can't get a rivet in and you're sure the hole is final size and didn't get missed when you were match drilling, its pretty simple to just twirl the appropriate side drill bit between your fingers and clean the primer out of the hole.