unladenswallow
Active Member
Earlier this year a fellow builder completed the avionics section of his RV12iS including of course fitting the AV-60000. When performing system tests he discovered that the trim wasn't working. After excluding the usual wiring and autopilot configuration issues with the help of another RV12 owner and myself, we established that the AV-60000 wasn't producing the correct control signals. I suspected that the Arduino in the AV-60000 might not be receiving power, so I asked him to test the dimmer knob and the stall warner as well. It turned out that neither was working, confirming that the Arduino wasn't working, which could due to absence of power or another reason. Lights and avionics were working, so it was clear that the AV-60000 was receiving power.
I am an embedded electronics and software engineer with access to a variety of test equipment, so I offered to look at his AV-60000 in case it had a simple fault, or there was some wiring error on his part that was causing the trouble. On receiving it I powered it with 12V from a bench supply and tested the various functions. I confirmed that none of the trim output, the stall warner audio output or the dimmer output were working when the correct inputs were applied to the AV-60000. I also confirmed the the Arduino was receiving power from the voltage regulator in the AV-60000.
At this stage I suggested that the user contact Vans. As we are both in Europe and getting US-made kit repaired can incur substantial carriage, duty and VAT charges, I suggested he asked Vans to supply a replacement pre-programmed Arduino for me to fit; or (even better) provide him with the programming file so that I could program a locally-sourced replacement Arduino.
The response from Vans was unfortunately not positive. The builder would have been happy to pay a reasonable price of a replacement Arduino. However, for several weeks he only received delaying responses.
While the builder was waiting for a proper response, I decided to investigate further. I extracted the program from the Arduino, which was straightforward. Imagine my surprise when I found that the first 128-byte page of the processor was un-programmed! There is no likely way that this could have occurred after the unit left the factory, barring a faulty microcontroller (possible, although unlikely). My interpretation is that either the Arduino was not programmed correctly in the first place and was never tested for correct functionality, or that after testing the first page of flash memory was erased. The first possibility could only occur if Vans or the subcontractor shipped an Arduino or AV-60000 unit without testing it, or after it didn't pass tests. The second possibility might conceivably occur if the unit was connected to a rig that programmed and tested it, but the program command was given again accidentally just before disconnecting the Arduino from the rig or powering it down.
Eventually, Vans eventually told the builder that he would have to buy a new AV-60000 at a cost of more than $500 + carriage + import VAT. They refused to replace or repair the unit under warranty on the grounds that the builder had not purchased the AV-60000 directly from Vans, rather he has purchased a part-completed kit from the previous builder - as is very common. This refusal to honour warranty would almost certainly have been illegal had the kit been purchased from a European seller. Maybe it's dubious under US law too, I don't know. Vans also refuse to supply just a new pre-programmed Arduino, or to supply the program file.
I am a firm believer in right to repair, so I investigated further. I disassembled the extracted program and saw that it comprised a standard Arduino bootloader and an Arduino program. Much of the program comprised standard Arduino library code. By constructing Arduino programs that used the same library functions and comparing the resulting code to the AV-60000 code, I was able to reconstruct the missing 128 bytes of program code, as this was comprised entirely of interrupt vectors and tables from the Arduino library. I programmed this code into a new Arduino (since it was possible that the original one might need to be returned to Vans) and the AV-60000 then worked properly. BTW the Arduino installed by Vans was a not a genuine Arduino, it was a Chinese clone.
I returned it to the builder suggesting that he/she consulted his build supervisor as to whether this repair was acceptable and to repeat the tests I had done to ensure that the functions controlled by the Arduino were working in all respects, including e.g. behaviour when both trim buttons are pressed, or one is held down for an extended period of time.
While investigating this I found a few issues that disturbed me a little:
- I don't expect software in experimental aircraft to conform to the DO-178C standard, with which I have some experience, and had a small part in developing. However, for any critical software such as the trim control, if the software is developed in a manifestly dangerous programming language such as C++ as here, I do expect it to confirm to basic good software practices. This should include at least significant compliance to the MISRA-C++ standard. Programs constructed using the Arduino IDE fall woefully short of this. Fortunately, the software functions performed by the Arduino are so simple that it is not too likely that it contains serious bugs. If I had the source code then I would conduct a proper analysis.
- The AV-60000 design omits the mandatory input decoupling capacitor specified by the manufacturer of the voltage regulator that feeds the Arduino. It only works because the Pololu trim motor driver module happens to include a suitable capacitor across its power input connections. If the Pololu module is removed, the regulator oscillates.
- All common microcontrollers including the ATMETGA328PB used in the clone Arduino implement a watchdog timer. The purpose of this is to reset the MCU If it hangs due to a voltage transient, cosmic ray or other cause; so that any safety-critical functions it performs (e.g. trim control and stall warner) are restarted. The code in the AV-60000 Arduino does not, as far as I can tell (I haven't completely reverse-engineered it) enable the watchdog timer. This is inexcusable IMO.
In summary, Vans needs to:
- Listen to their customers more, and help them instead of fobbing them off and refusing warranty;
- Use qualified personnel to design their electronics and write the critical software. It appears to me that it has been designed and written by amateurs.
I am an embedded electronics and software engineer with access to a variety of test equipment, so I offered to look at his AV-60000 in case it had a simple fault, or there was some wiring error on his part that was causing the trouble. On receiving it I powered it with 12V from a bench supply and tested the various functions. I confirmed that none of the trim output, the stall warner audio output or the dimmer output were working when the correct inputs were applied to the AV-60000. I also confirmed the the Arduino was receiving power from the voltage regulator in the AV-60000.
At this stage I suggested that the user contact Vans. As we are both in Europe and getting US-made kit repaired can incur substantial carriage, duty and VAT charges, I suggested he asked Vans to supply a replacement pre-programmed Arduino for me to fit; or (even better) provide him with the programming file so that I could program a locally-sourced replacement Arduino.
The response from Vans was unfortunately not positive. The builder would have been happy to pay a reasonable price of a replacement Arduino. However, for several weeks he only received delaying responses.
While the builder was waiting for a proper response, I decided to investigate further. I extracted the program from the Arduino, which was straightforward. Imagine my surprise when I found that the first 128-byte page of the processor was un-programmed! There is no likely way that this could have occurred after the unit left the factory, barring a faulty microcontroller (possible, although unlikely). My interpretation is that either the Arduino was not programmed correctly in the first place and was never tested for correct functionality, or that after testing the first page of flash memory was erased. The first possibility could only occur if Vans or the subcontractor shipped an Arduino or AV-60000 unit without testing it, or after it didn't pass tests. The second possibility might conceivably occur if the unit was connected to a rig that programmed and tested it, but the program command was given again accidentally just before disconnecting the Arduino from the rig or powering it down.
Eventually, Vans eventually told the builder that he would have to buy a new AV-60000 at a cost of more than $500 + carriage + import VAT. They refused to replace or repair the unit under warranty on the grounds that the builder had not purchased the AV-60000 directly from Vans, rather he has purchased a part-completed kit from the previous builder - as is very common. This refusal to honour warranty would almost certainly have been illegal had the kit been purchased from a European seller. Maybe it's dubious under US law too, I don't know. Vans also refuse to supply just a new pre-programmed Arduino, or to supply the program file.
I am a firm believer in right to repair, so I investigated further. I disassembled the extracted program and saw that it comprised a standard Arduino bootloader and an Arduino program. Much of the program comprised standard Arduino library code. By constructing Arduino programs that used the same library functions and comparing the resulting code to the AV-60000 code, I was able to reconstruct the missing 128 bytes of program code, as this was comprised entirely of interrupt vectors and tables from the Arduino library. I programmed this code into a new Arduino (since it was possible that the original one might need to be returned to Vans) and the AV-60000 then worked properly. BTW the Arduino installed by Vans was a not a genuine Arduino, it was a Chinese clone.
I returned it to the builder suggesting that he/she consulted his build supervisor as to whether this repair was acceptable and to repeat the tests I had done to ensure that the functions controlled by the Arduino were working in all respects, including e.g. behaviour when both trim buttons are pressed, or one is held down for an extended period of time.
While investigating this I found a few issues that disturbed me a little:
- I don't expect software in experimental aircraft to conform to the DO-178C standard, with which I have some experience, and had a small part in developing. However, for any critical software such as the trim control, if the software is developed in a manifestly dangerous programming language such as C++ as here, I do expect it to confirm to basic good software practices. This should include at least significant compliance to the MISRA-C++ standard. Programs constructed using the Arduino IDE fall woefully short of this. Fortunately, the software functions performed by the Arduino are so simple that it is not too likely that it contains serious bugs. If I had the source code then I would conduct a proper analysis.
- The AV-60000 design omits the mandatory input decoupling capacitor specified by the manufacturer of the voltage regulator that feeds the Arduino. It only works because the Pololu trim motor driver module happens to include a suitable capacitor across its power input connections. If the Pololu module is removed, the regulator oscillates.
- All common microcontrollers including the ATMETGA328PB used in the clone Arduino implement a watchdog timer. The purpose of this is to reset the MCU If it hangs due to a voltage transient, cosmic ray or other cause; so that any safety-critical functions it performs (e.g. trim control and stall warner) are restarted. The code in the AV-60000 Arduino does not, as far as I can tell (I haven't completely reverse-engineered it) enable the watchdog timer. This is inexcusable IMO.
In summary, Vans needs to:
- Listen to their customers more, and help them instead of fobbing them off and refusing warranty;
- Use qualified personnel to design their electronics and write the critical software. It appears to me that it has been designed and written by amateurs.
