another future RV pilot
My kids were about 3 and 6 years old when I started my -6A project. By the time of first flight we had added another one; now ages 12, 15, and 18. So they don't really know what life is like without an airplane project in the garage (or house for the years we didn't have a garage...). Family is one of the reasons this project took me so long, and I wouldn't change a thing.
Here's my son checking on progress during wing construction many years ago (he has done a lot of flying with me, including some Cessna 140 airplane camping, and definitely wants to get his license and fly this plane, but I haven't convinced his sisters yet - by the way, the Cessna 140 trips did slow my RV progress as well but the memories we share were worth it and again, I wouldn't change a thing):
You can see in the picture above that the wing construction was one of those phases without a garage - that airplane shop was a bedroom in the house we lived in at that time.
And the picture below, taken just after my first flight on 25 October 2008, shows my now much taller/bigger son with his little sister (she's younger than the tail section of my RV):
Airplane building I hope will be a great retirement activity for me someday but for those still raising a family, take heart - it can be done (slowly) without completely giving up your family time!