This is an old trick that might help. If you can get a hold of a ball bearing (1/2" or larger). Set the ball bearing on the hole. Back the other side up. Gently tap the bearing. This will cause the hole to close up on one side. Repeat the process on the other side. Then you may have solved your problem.
The material drilled is either 6061 or another soft aluminum so it is not that uncommon to have an enlarged hole. I reamed mine to .375 and found the bearing slipped in without much friction. After the ball bearing trick they pressed in just fine.

Adding the loctite (Red) wouldn't hurt in addition.
Paul