A lot of you have stated a very good point. The RV-10 as designed seems to be a very capable aircraft. Sure we all could build it by the numbers and end up with a very nice plane that performs the mission. There's nothing wrong with that.
I feel a lot of us choose to build our own plane so we can build it to our desire and put our own personal touch in it. That is at different levels, some choose to only change the paint scheme or interior design, some choose to add mods like rudder trim or A/C. Some people get a little more extreme and start customizing engines, fuel tanks, or adding recovery chutes, etc.
For me personally, I'm starting with the -10
because of it's reputation, build process/quality, and the community. However, I have a desire to experiment with a different engine conversion. I do plan to setup the engine on a test stand with prop and cowling and run it as much as possible while I build the rest of the plane. I realize there will points along the way, that I may change my mind and go with an aircraft engine. But, I will give this a try.
I'm not trying to make the -10 go faster. My third build may be a jet, LOL, and that would start with an airframe that is
designed to go fast. My second build will probably be a STOL aircraft.
I am not trying to be a jerk but you should consider the advice.
You are talking desires and reality. First your RV10 will not be fast with a VW Audi V6 TDI diesel or any other car engine I can think of, I can guarantee it. The reputation of the RV3, 4, 6, 7, 8, 10 are all based on Lycoming power. Deviations from Lytc power have had some mixed results, mostly poor to bad, with rare individuals like RV6guy and his buddies up in Canada... So if slowing down and reducing payload carrying ability in your dream RV10, is your goal, you will succeed with a prop bolted to a PSRU driven by a boat anchor, water cooled, car engine. There is no experiment. It has been done and the results are known. Why do you think your results will be different?
Before working on your 2nd STOL plane build, you might want to try your 1st to plane. The most sage advice given to new builders is
"stick to the plans". You assume you will have good performance with a VW/Audi TDI V6. (BTW I have been driving a VW TDI CR 2.0L Diesel for 10 years and still do, love diesel, but many companies with vast resources have tried to make diesel work specifically for light planes with various degrees of success and failures, sometimes technical success and just sales failures. Diesel option is more expensive, is always heavier and does not offer a huge advantaged in a plane. In a car, pickup truck, semi truck or large boat diesel rules, but then weight is not as big of an issue.)
BTW I don't know your finances, but airplanes are expensive you may have heard. I assume you are a pilot. I recommend sometimes buying a fixer upper C140C150/152/170/172, Piper (any Pacer, Cherokee, Tomahawk) and being an aircraft owner. If you want a equivalent RV10 then a Mooney, Beech Bonanza, C182. If you never owned a plane it might enlighten you to the reality of airplane care and feeding. To spread the cost out, getting a partner to split the fixed cost (tie down/hanger) and expendables (oil, tires, brakes) is also a good thing. Some people who build never owned a plane and might be surprised. It is like the old joke a man is happy they day he buys his boat.... but that man is truly happiest when he sells his boat.
The down side of owning and flying is this is a distraction in time and money from building. Obviously you can't experiment with a certified plane as an amateur built kit plane, without getting the FAA involved. Bottom line work on your airmanship and have fun... or stay on the ground dreaming. Many have had the same thoughts as you and spent a lot of time an money not flying. However many people prefer to not fly and dream, build and tinker. I have seen some spectacular car engine conversions in planes collecting dust as they work on it, run it up in front of the hanger but never fly.
Up to you. I like both building and flying.... but flying is the goal not nursing a hanger "experiment" queen.
Only plane I know of in "large" (qualified as a few dozen planes a year) production with diesels is the Diamond DA62 light twin with two Austro Engine AE 330 turbocharged common-rail injected 2.0 liter diesel engines rated at 180 HP each, using EECU single lever control system. Keep in mind this is 180HP. It seems many "aircraft" diesels shoot for the under lower 1800 HP range when designing a diesel. The nominal HP a Lyc 540 puts out is 250 to 350 HP, more with tweaking. (Note a 2021 Diamond DA62 is $1,135,000. Yes over a million. Dimond now offers the diesel in a single engine DA50 for $700,000.)
You are 100% out on your own designing a new. Do you have the background, skill, equipment, time and money to make it work. Dreaming is one thing and flying is another. Many have dreamed and failed. However the dream with a Lycoming is less risk and most likely to lead to success and known excellent performance (real world performance not pipe dreams). Also you will have a valuable plane you could sell. Most car engine planes do not sell well or are discounted well below a Lyc powered example.
There are so many challenges to adapting a car engine intro a plane, the PSRU, cooling, fit, prop, weight, fit, mount, cowl.... With a Van's kit it is all designed and laid out (for a Lyc). Even then sticking to plans, keeping it simple, with diligent constant work you will have 2000 hours building it (if you are an efficient builder. Part time work that means a few (to many years). Now going your own you will make many parts over and over to make it work. Is it worth building? YES!!! However at the end you want a plane that performs and is valuable to you and even in resale, since you will put in well over $100,000 into it plus your time.
With that said UP TO YOU. Good luck.
By the way Lycoming's and Continental's can easily be turbocharged for high altitude. As was said a RV10 with weight and runway limits can operate at high density altitudes with descent payload. The B737's and other Boeing jets I flew out of Denver
were also weight and runway limited when ambient was hot. Are you going to do gross weight T/O's at 9000 ft often if ever? If you want Jet A get a P&W PT6. However it is cheaper to buy a used TBM700.... and you would have much more of a plane than a RV10. They can be bought for less than $1M.
Do you want to fly or dream. The quickest way into the air besides buying a finished RV is buy the kit, pound rivets, stick to the plans and drawings, keep it simple.
If you have your heart set on putting a car diesel in a plane do your research. First find someone who has done it and has 100's of hours flying it. I almost guarantee you will have a hard time finding this person. This should be a hint? (I know of a 2.0L VW TDI in a Cessna in Europe and has been flying for a few years. How much it has flown besides the YouTube Video of first flight, performance don't know.)
All the best and keep us informed of your progress...