You're doing the correct thing.
It's pretty much a given that any sharp 90 degree turn is bad. Doesn't matter if it's the edge of a thick doubler or the inside edge of a very cleanly drilled hole.
Even though there isn't a burr and the hole is absolutely perfect, it's also got an absolutely perfect 90 degree angle.
Just use 1.5 - 2 turns. It's just enough to knock that 90 degree sharp edge off. If there are actually burrs hanging onto the hole, you might need to give it another twist or so...
But that's the reason.
Phil