I belong to both, and on occasion, I have been quite displeased with both. However, walking away has never been a realistic option.
The sensational side of the free press, the fearful public, our need for airports, the battle for airspace, and the huge overreach of the security establishment all threaten our ability to fly. Failure to remain organized and present our case is simply capitulation.
So please, don't walk away, at least at the national level. When you're unhappy, don't complain...organize! Both EAA and AOPA are membership organizations, and if enough members feel strongly about something, they can change it.