Dynon Secondary ADAHRS - What Redundancy Really?

[jaredyates]

I've recently installed my second Skyview system and come across a very troubling situation regarding the non-reversion of a secondary ADAHRS.

The short version of the story is that I installed the system in an already flying airplane, flew a validation/calibration flight a month ago, and all went well. The system includes two screens and both a primary and secondary ADAHRS, among other things The next day I went out to the airplane to fly and around 30 seconds after startup everything that connected to the ADAHRS put up red Xs and there was a message that said "ADAHRS Fail Internal Error: Multiple Sensors".

After some back and forth with tech support, including lots of network configuration iterations, making some new test cables, half a day of unbundling and testing wires, we arrived at the only way this message could occur is if both ADAHRS units failed. I sent them both back and they arrived at Dynon on 6/22.

It took until today (7/11) for them to finish repairing the units (no stock for replacements, so they say). On the phone I asked what they found wrong with the units, and they said that the primary had several sensors failed, which they repaired.

What was wrong with the secondary? Absolutely nothing, so they say now. Before sending in the units, the only way that this error could have been presented was if both units were completely failed.

So I asked, why do I even have a secondary? He said, "sometimes it happens."

We trust our lives to Dynon, I don't think it should happen, and I think they owe us a lot more information about what a secondary ADAHRS is actually going to do and when. I don't think a "sometimes (the primary breaks in a way such that there is no useable secondary but we aren't going to tell you what that is because maybe we don't know or maybe we just don't want to) is an acceptable answer.

How careful am I going to have to think about taking the airplane somewhere and shutting down the engine, only to have it be stuck there for a month because of a single ADAHRS failure?

I don't think Dynon has done well in this situation for several reasons, most of which I can live with because I know Garmin might be just as bad or worse, but the non-reversion issue is not something I can disregard.

Feel free to tell me if I'm being too whiny about this. If you are running a Dynon system and think you are getting any redundancy from a secondary ADAHRS, I'm curious to hear what you know or think about its reversion and diagnostic capabilities.

 

[rleffler]

I agree that Dynon should be more transparent about any known failure scenarios and how to mitigate them.

When I completed my recent panel upgrade, I installed a Garmin G5. AFS allows the G5 to function as a third adahrs. I don’t know if the same functionality is available on the HDX.

It does give me more confidence that the third device is truly a tie breaker if another adahrs goes bad.

 

[lr172]

I've recently installed my second Skyview system and come across a very troubling situation regarding the non-reversion of a secondary ADAHRS.

The short version of the story is that I installed the system in an already flying airplane, flew a validation/calibration flight a month ago, and all went well. The system includes two screens and both a primary and secondary ADAHRS, among other things The next day I went out to the airplane to fly and around 30 seconds after startup everything that connected to the ADAHRS put up red Xs and there was a message that said "ADAHRS Fail Internal Error: Multiple Sensors".

After some back and forth with tech support, including lots of network configuration iterations, making some new test cables, half a day of unbundling and testing wires, we arrived at the only way this message could occur is if both ADAHRS units failed. I sent them both back and they arrived at Dynon on 6/22.

It took until today (7/11) for them to finish repairing the units (no stock for replacements, so they say). On the phone I asked what they found wrong with the units, and they said that the primary had several sensors failed, which they repaired.

What was wrong with the secondary? Absolutely nothing, so they say now. Before sending in the units, the only way that this error could have been presented was if both units were completely failed.

So I asked, why do I even have a secondary? He said, "sometimes it happens."

We trust our lives to Dynon, I don't think it should happen, and I think they owe us a lot more information about what a secondary ADAHRS is actually going to do and when. I don't think a "sometimes (the primary breaks in a way such that there is no useable secondary but we aren't going to tell you what that is because maybe we don't know or maybe we just don't want to) is an acceptable answer.

How careful am I going to have to think about taking the airplane somewhere and shutting down the engine, only to have it be stuck there for a month because of a single ADAHRS failure?

I don't think Dynon has done well in this situation for several reasons, most of which I can live with because I know Garmin might be just as bad or worse, but the non-reversion issue is not something I can disregard.

Feel free to tell me if I'm being too whiny about this. If you are running a Dynon system and think you are getting any redundancy from a secondary ADAHRS, I'm curious to hear what you know or think about its reversion and diagnostic capabilities.

Really only two possibilities. 1: the reversion software doesn't work that well and you really only had one ahrs failure but the s/w couldn't deal with it. 2: there is a problem with the second one and the dynon folks couldn't find it. With advanced electronics, NTF (no trouble found) on units with acutal issues is far more common than you might think. This is why warranty replacements (they pull from the NTF return pile) is such a lottery, sometimes needing two or three to get a good one.

If two units truly failed at once shortly after installation, some type of wiring or other elec system issues must also be investigated, as that is just too big of a coincidence.

 

[Untainted123]

When I completed my recent panel upgrade, I installed a Garmin G5.

Between software, hardware and environment, detecting an error and reverting to backup is a hard problem to solve.

I was told that even at Garmin, they are concerned about cross-contamination (for lack of a better term) that they keep the development team and the software for the G5 completely separate from the rest of the ecosystem (G3x etc) to help mitigate this issue. They don't want a bug introduced in one to affect the other.

Personally, I would forego the 2nd ADAHRS and buy a separate device, possibly from another vendor altogether (G5 being my preference, it incorporates so much redundancy in one package). You still have to know and design around the single point of failure behind THAT (power, pitot-static etc), but it at least gives some redundancy in the software and hardware stack.

 

[jaredyates]

Really only two possibilities. 1: the reversion software doesn't work that well and you really only had one ahrs failure but the s/w couldn't deal with it.

I think from a horses/zebras standpoint this is the only plausible explanation, which is the motivation for the rant.

 

[dmattmul]

2nd ADAHRS

Between software, hardware and environment, detecting an error and reverting to backup is a hard problem to solve.

I was told that even at Garmin, they are concerned about cross-contamination (for lack of a better term) that they keep the development team and the software for the G5 completely separate from the rest of the ecosystem (G3x etc) to help mitigate this issue. They don't want a bug introduced in one to affect the other.

Personally, I would forego the 2nd ADAHRS and buy a separate device, possibly from another vendor altogether (G5 being my preference, it incorporates so much redundancy in one package). You still have to know and design around the single point of failure behind THAT (power, pitot-static etc), but it at least gives some redundancy in the software and hardware stack.

Hopefully I’m not whistling by the graveyard but I like my second ADAHRS as I can (and do) get a miscompare which I then check my G5 to see which one is not being truthful. Seems to work pretty well. (So far)

 

[BobTurner]

I think from a horses/zebras standpoint this is the only plausible explanation, which is the motivation for the rant.

I agree. I think there can be too much automation. e.g., there should be a one or two, at most, switches or buttons to force a change over from #1 to #2 AHRS, if for no other reason than to occasionally test #2. Some time ago someone here on VAF reported a similar problem with a mini EFIS, a D-6 iirc. That box needed either airspeed, or gps, to reach a stable attitude solution. Its programming used airspeed, and switched to gps if an airspeed error was detected. But this pilot experienced a partially blocked pitot. The airspeed was obviously too low, but not low enough to activate the auto-switchover. But low enough to cause a false horizon indication. And, iirc, Garmin suffered a software failure more than a decade ago, where KLVK, KORD, etc all worked fine, but any non-all letter ID (e.g., C83) crashed the system including all interconnected efis boxes). This scared me enough that I put in a backup mini-EFIS from a different manufacturer than my main efis, and that has zero electrical interconnects. Yes, this creates a bit more work (I need to enter new altimeter settings by hand into both boxes) but gives me more hope that I can avoid some of these rare but potentially serious problems.

 

[Walt]

Interesting youtube video on the Garmin GI275. I removed my G5 and installed one myself for a number of reasons but mainly I wanted total independence from the G3X. It talks to the G3X via 232 only (no CAN bus), and HSDB to the GTN. Like the G5, I always install the suggested VFR GPS antenna for aiding.

https://www.youtube.com/watch?v=dYch_pDjJzA

 

[PaulvS]

The initial post does not state whether the 2nd ADAHRS was confirmed as being recognized and configured into the network, so I wonder about that.

I don't "trust my life" to Dynon or any other component manufacturer and believe that it's up to us as owners and pilots to take responsibility. Any component can potentially fail e.g. simultaneous loss of both displays (another brand, not Dynon) as experienced by a fellow pilot.

My SV system has a single display and a single ADAHRS and is supplemented with an analog ASI, which is sufficient to fly and land safely if the digital stuff quits working.

 

[jaredyates]

The initial setup did include having all of the components configured into the network, thank you for the suggestion. Sometimes it's the easy things and while anything is possible, in this case I didn't elaborate on how many steps were involved in ruling out the easy things, only to find out that I had done my part as the installer. Perhaps it is usually worthwhile to blame the installer when the customer base is a bunch of amateurs. In this case there were many subsequent network configuration attempts in various arrangements (and more) to try and make it work.

Your point about responsibility is fair, I do share culpability by trusting Dynon to create a robust system, and to keep replacement stock on hand, and to provide solid tech support. I gave them that power by selecting their system and now I'm complaining that they have too much power. Even knowing what I know now, I don't think I can justify sustaining completely non-dynon redundancy for 91.205 in the space I have. I realize that is my choice.

In my 250 hours on the last Skyview system I had three different instances of screens becoming inop, but they always came back after rebooting and never shut down the whole system. Interestingly I never had a hiccup on the d180/100.

 

[N804RV]

Now I'm curious. Is it routine to verify that both ADAHARS are recognized during the pre-taxi checklist?

I was reading in the install manual that you can confirm that each ADAHARS is online and recognized, that an ADAHARS that is not functioning will not show a status as "ACTV" or "STBY", and that you can disable the primary ADAHARS to verify the Standby ADAHARS will switch to active.

Is it realist to expect to be able to do that as part of a pre-flight checklist? And would that have made any difference in this case?

 

[jaredyates]

The menus can lead you to a comparison screen to see thr output of both units. Here is a photo of mine in the failed condition, when the secondary is supposedly perfectly fine:

https://photos.app.goo.gl/a9XjNj1FqjFLMJws7

 

[PaulvS]

There is another (SV software glitch) possibility that the system did not automatically failover because the 2nd ADAHRS was also flagged as faulty. So there was nothing to failover to, as far as the system was concerned, even if the redundant ADAHRS was actually OK!

Only Dynon would be able to potentially discover the cause and whether there is a software glitch that causes both ADAHRS to be flagged as failed even if only one of them actually generated a fault.

The older systems e.g. D180 (I also have one in my old plane) seem simpler and more robust than the current range of integrated systems, which have many more features than before and probably thousands and thousands of lines of software code. Unfortunately the KISS technology is no longer available!

 

[RVDan]

I set my dual adahrs up with the SV network cable for #1 ADAHRS going directly to the pilots display, and#2going directly to the cp display. This provides some level of wiring separation reducing single point wiring or hub failures from taking out both ADAHRS inputs. I then set up the Pilots display to use ADAHRS #1 as primary and CP display to use ADAHRS#2 as primary. This always gives me a view of what each ADAHRS is doing. In the event of a single failure the other ADAHRS is always displayed cross side.

 

[jaredyates]

Interesting Dan, thank you. I hadn't caught on that the ADAHRS priority was a per-screen setting but I like your strategy.

 

[ronrapp]

I've got glass from a single manufacturer (G3X Touch with G5 standby) and have never really thought anything else was needed, just by virtue of statistics.

Could a critical software bug in one box also be present in another? Sure. But after 10,000 hours of flying, I've got 385 hours of actual IMC, so were I to encounter such a failure, the odds are better than 96% that I'd be in VMC and less that 4% that I'd be in actual instrument conditions.

I think Garmin is also kind of unique in that a lot of this software and hardware is based on or descended from previous units like the G1000, so there's a considerable base of installations and users who have god only knows how many hours behind this stuff. I flew early G1000 panels, later upgrades like the Perspective suite in the Cirrus fleet, etc and never had a problem with any of them. To me, the best of all worlds is a non-certified system from a company that has a long history of making certified EFIS panels.

I'm not sure it's possible to engineer a perfect system, but even the most bug-prone of these EFIS units is light years ahead of the mechanical gyros I learned to fly instruments on. Partial panel NDB approaches, anyone? No?

--Ron

 

[jaredyates]

The big difference I hadn't fully internalized before now is that in the old days, if your vacuum pump quit, you could order a new vacuum pump and get the plane back in the air. With an integrated panel, if the one manufacturer gets a little sloppy with their software, supply chain, and management in general, we are totally stuck until they find themselves. This is going to be at least 5 weeks of downtime, and it could have easily been 2 weeks or even less if they managed things better.

In this case even if it isn't their fault that the ADAHRS broke, they have failed by having the software bug, by not having replacement stock, and not having reversionary capability for dispatch while the primary is out, and not having a tech support team that is trained/enabled to handle the problem quickly.

We've lost five days just in the logistics of them opening the box on the front end, and boxing/invoicing/shipping on the back end. Nothing to blame here but management.

Is this the same bug as someone else found? Maybe? Probably? The tech support attitude to what I'd consider a pretty catastrophic problem was "meh, it happens", not omg we need to fix that. Is it even going to be logged or addressed?

 

[rleffler]

The big difference I hadn't fully internalized before now is that in the old days, if your vacuum pump quit, you could order a new vacuum pump and get the plane back in the air. With an integrated panel, if the one manufacturer gets a little sloppy with their software, supply chain, and management in general, we are totally stuck until they find themselves. This is going to be at least 5 weeks of downtime, and it could have easily been 2 weeks or even less if they managed things better.

In this case even if it isn't their fault that the ADAHRS broke, they have failed by having the software bug, by not having replacement stock, and not having reversionary capability for dispatch while the primary is out, and not having a tech support team that is trained/enabled to handle the problem quickly.

We've lost five days just in the logistics of them opening the box on the front end, and boxing/invoicing/shipping on the back end. Nothing to blame here but management.

Is this the same bug as someone else found? Maybe? Probably? The tech support attitude to what I'd consider a pretty catastrophic problem was "meh, it happens", not omg we need to fix that. Is it even going to be logged or addressed?

I guess your comments depend on which vendor you buy products from. I’ve experienced both sides of customer support.

I have a minor user data issue on my gps. I’ve called customer support a half dozen times. Even stood in line at their tent at OSH for over an hour to talk with tech support face to face. All state ignorance of the issue even though I’ve logged incidents on previous calls. But then all state for $1000 I could send it in and they would resolve the issue.

Then there is an EFIS vendor that I’ve found several bugs in the software. One major bug ended up with tech support, the lead programmer, and president of the company on the call working the issue. I had a workaround by the end of day and new code that fixed the issue in a day two. I’ve also had a couple minor bugs. Usually they are addressed and code available within a week. There were no fees for their support or resolution. They also have crossed shipped hardware when there was speculation of a hardware issue.

One of these vendors are on my do not buy list. Although I broke down a bought an item from them in my recent panel upgrade.

The other vendor, I’m now on the third generation of their products. Both efis and adahrs. It’s probably apparent which vendor I prefer and why.

Buying avionics is not only about the right specs, it’s also about selecting vendors that consistently deliver quality customer service.

 

[Bavafa]

I guess your comments depend on which vendor you buy products from. I’ve experienced both sides of customer support.

I have a minor user data issue on my gps. I’ve called customer support a half dozen times. Even stood in line at their tent at OSH for over an hour to talk with tech support face to face. All state ignorance of the issue even though I’ve logged incidents on previous calls. But then all state for $1000 I could send it in and they would resolve the issue.

Then there is an EFIS vendor that I’ve found several bugs in the software. One major bug ended up with tech support, the lead programmer, and president of the company on the call working the issue. I had a workaround by the end of day and new code that fixed the issue in a day two. I’ve also had a couple minor bugs. Usually they are addressed and code available within a week. There were no fees for their support or resolution. They also have crossed shipped hardware when there was speculation of a hardware issue.

One of these vendors are on my do not buy list. Although I broke down a bought an item from them in my recent panel upgrade.

The other vendor, I’m now on the third generation of their products. Both efis and adahrs. It’s probably apparent which vendor I prefer and why.

Buying avionics is not only about the right specs, it’s also about selecting vendors that consistently deliver quality customer service.

I all with you in your choice of the vendor, though you have not named it. I have had the same experience and think very highly of them.

 

[Derek.plough]

You guys are killin me! I think it’s important to share our experiences both positive and negative. Spill the beans!

 

[MacCool]

You guys are killin me! I think it’s important to share our experiences both positive and negative. Spill the beans!

On the positive end....Advanced Flight, I'm guessing. I recognize the described excellent customer communication and support, having experienced it myself.

 

[RVDan]

As a person who has worked extensively in the certification of avionics, I have a couple of thoughts on SW. In the certified world, equipment manufacturers have to develop and maintain software using the rigorous systems for software development in standards like RTCA DO-178B or C. This adds immensely to the cost of software life cycle beginning with the requirements to verification and validation. There are levels of SW design assurance under DO-178 that create 4 levels of expected reliability. Level A is the highest, and Level D is the lowest for functions that impact safety of flight. The level of development and testing rigor increases from level D to Level A. Attitude, Airspeed and Altitude are typically level A systems. That means that systems are not expected to have SW failures, but yet, they do. This is why we look for separately developed independent systems for the standby, and why Garmin developed the G5 with an independent team.
Some SW for basic Part 23 aircraft is developed with in-house procedures that mimic DO-178, such as Dynon Skyview Certified SW and Garmin G3. For larger aircraft ( more bodies aboard) the SW meets the higher standard of DO-178

SW for Exp systems that is a child of certified SW has typically been modified which can have a significant impact on the SW reliability.

That said, the old insidious vacuum pump failures which happened all too frequently have been eliminated with electronic systems and system reliability improved for us little guys.

 

[Walt]

In ref to DO-178 the GI-275 for class I/II spec is shown below (they also have a class III/Part 25 unit), could not locate similar information on any of the other "standby" instruments. Here's some history from Flying Magazine for part 23:

"When the FAA a couple of years ago relaxed approval standards for certain avionics in certified Part 23 airplanes, it opened a pathway for manufacturers to skip the lengthy and expensive TSO certification pathway and create new products for general aviation based on ASTM standards rather than the cumbersome DO-178 standards for software, in the process sometimes slashing millions of dollars from the development costs of a single product."

Now you know a bit more about why I chose the GI275 to replace the infamous G5.

 

[Walt]

Plus the 275 is beautiful No more square peg in a round hole!

 

[akschu]

All state ignorance of the issue even though I’ve logged incidents on previous calls. But then all state for $1000 I could send it in and they would resolve the issue.

Which vendor is this? PM is okay if you require....

One major bug ended up with tech support, the lead programmer, and president of the company on the call working the issue.

That can only be Advanced Flight Systems, but from what I understand they use the Dynon AHARS now..... I wonder if their EFIS software swings to the secondary AHARS correctly?

My panel is all garmin now, canbus is really the best way to do this type of thing. I also like the RS-232 backup.

schu

 

[jaredyates]

I got an email last night from Robert at Dynon. If I'm summarizing fairly he says that they are indeed investigating further, and that my vibe from tech support about them not taking the non-reversion seriously was a miscommunication rather than the reality.

This was all welcome news and reassuring. He also said that they shipped replacement units, which is the first I've heard of. Tech support was pretty clear about repairing and sending back the original, and that was why it took so long.

UPS should be delivering the boxes on Wednesday which will give me a day or two if the weather is good to put in a bunch of hours before it is time to decide whether to fly or drive to Oshkosh.