If you are speaking of standard AD rivets, I would look first to your dimpling technique.
When I first started, wary of the "softness" of the Alu, I was always under forming the dimple. You have to "coin it" in the vernacular.
Play with some scrap and increase the force applied to your dimpling set up.
Stop short of marring the surface of the sheet around the die.
If this doesn't work for you I'd check your dimple dies.
FWIW
I found that when dimpling with the C-frame I got better results if I hit twice. First hit would set the dimple, then the second (harder) hit would ensure that it was properly formed.
[snip]