The rivet is aluminum. The pull shank is steel. They're all that way. Keep calm, build on.
A good friend who's built *many* planes showed me a nifty trick a year or so ago to get the shop end of some drilled-out pulled rivets removed from a RV-12 stabilator. Enlarged the tooling holes a little in the outboard rib, got a magnet on the outside of the structure, and dragged the shop heads with the magnet to the tooling hole. Beats taking the whole thing apart for sure. I'm going to call it a feature.
Here's 2 mins and 41 seconds of doing just that hah:
https://www.youtube.com/embed/C71PUvVTt_o