I had this problem when I used those "sub-structure" dimple dies. I stopped using them.
With the regular #40 dimple dies from Cleveland and the 3/32 clecos from Brown Tools, I haven't had any problems.
To rivet the stuff you've already dimpled, make up some small backers out of a piece of scrap from the trim bundle. You just cut some small squares and drill #40 holes, no need to dimple these temporary cleco backers. But, they'll hold everything together well enough for you to rivet.
Since the holes are now just the tiniest bit bigger, I'd recommend using a cleco in every hole, or at least every other hole to limit the amount of slop between pieces. And, make sure that every dimpled/countersunk hole is properly nested with no gaps. If you have to, get some hex-nut style clecos to draw stuff together tight, and also use cleco edge-clamps where needed.