Back in the day, Van's didn't prime their demo aircraft. The paint on those seemed to hold up reasonably well. If you're not going to prime, paint adhesion probably comes down to how well you clean, scotchbrite, and etch the surfaces prior to paint.
One alternative to no primer under your paint is a VERY thin coat of self-etching primer.