You are OK. Just fill when you are finishing the skins for paint. The skins do not move back and forth and the crease does not contact the edge of the skin.
When I first started, my partner and I got a bit too loose with the dimpler C-frame and puched a hole an inch from the hole we intended to dimple. After we started breathing again

we asked an "old pro" and his response was "Been there done that, just fill with epoxy or rivet. No big deal."
It is a part of the process and you will have the occasional ding while building. I have yet to meet anyone who has drilled, deburred, and set over 9,000 rivets ( rough avg of total) without 2 or 3 mishaps.
If you want to see how good (ie; experienced) a builder is, start by looking at his tail feathers. Sometimes you can tell which one they did first.
Happy riveting.