My kids didn't fly with me until they were around 6 or 7. Couple of point here:
1. Don't take them up until they can effectively equalize the pressure in their ears. Inflict that kind of pain and you may burn them on flying for life. My 13 yr olds still have problems with the issue, so I am very careful with altitude variation rates.
2. I see no good point in taking a child up for a pleasure flight (excluding travel) that cannot sit up, wear a headset and look outside the plane. Not being able to see out, and their very likely going to get vertigo and become ill. More chance on souring them on flying, not to mention inflicting physical discomfort on your child.