a couple of methods FWIW
if you are talking 12" N numbers, the 'eraser' is going to be slow.
It takes a little practice to scrub off the viinyl without burning the background....an air drill with the pressure turned down is good.
Here's how I tackle vinyl removal from vehicles, but there are a lot of variables, so experimentation is the word.
I would use a razor blade, to lift only the corner of the letter/number.
dull it first by scraping it on the concrete floor or driveway, ( like you are stropping a razor) and knock the sharp corners off ...which will keep it from digging in to the aluminum and leaving a big gouge. some plastic razor blades are good for this if you really want to be safe.
If the lettering has been in the direct sun, it is probably warm enough for trial one. try to peel the tab you've lifted with the razor at 90 degrees to the surface.
If it snaps, or the adhesive stays on the surface, try a different angle, speed, and heat. the ideal of course is when the vinyl takes all the adhesive with it! If the vinyl is old, thin, and breaks a lot, you may be unable to peel it.
Another method is to coat the vinyl with a solvent that softens the vinyl, soaks thru and softens the adhesive. ( check your yellow pages for
'sign supply'....or automotive graphics/detailing?) 'Vinyl-Off' 'Rapid-Remover' and 3M all make similar products.
You now have a real gooey mess, but most of the orange based cleaners like 'goo gone' will clean it up pretty well. Good ol mineral spirits or other slow thinners will also allow you to scrape the adhesive off with a plastic squeegee blade onto a paper towel.
spray, scrape, rinse, repeat.
If you are going to paint afterward, be aware that a scrupulous cleaning will be required to get all this adhesive snot out of the rivets and seams!