This is not a good assumption. I strongly recommend anyone not certain of the mods they're contemplating call the factory. I was told flatly that they're "structural". Good luck.
As someone not daring to stray from the design I don't advocate as such. I always presume the factory is both conservative and right. Don't know why someone would stray on an issue like this. But I also know the tech help on the phone are clueless on this kind of detail and will always defer to build as designed. They are not consulting a finite element model of the wing when they answer you.
Having said that, on my -4 the wing access panel joggle, which accounts for much of the strength of a hole, is discontinuous and rectangular. That means it is carrying little strength. The hole is bisected by a rib so the skin is not carrying much load in that area.