Why we have Backups!

Anyone who reads about me and my airplane should have figured out by now that I am a committed Glass Cockpit IFR pilot. I believe that the use of integrated avionics, advanced displays, and reliable solid-state gyro platforms (along with good pilot training and judgment) can make GA flying safer and more reliable. I make that pre-amble so that folks don't take this to be a "bashing" thread - rather, this is an item of interest that points out a fundamental requirement when we implement new technology - rock-solid, dissimilar backup systems. Don't put all your eggs in one basket. Plan for failure. Have a system or plan that gets you out of it without breaking a sweat.

And with that said, here is a little report that crossed my desk the other day. This is real, it happened just a couple of weeks ago, and goes to show that no matter how much money and time you have to spend on development, you will still never have a perfectly reliable system. And you handle that by.....having a backup plan!

BTW, the software problem mentioned below has been fixed and already implemented in the fleet. It is hard to to think of everythign to test in a multi-billion-dollar development program - and somehow, the longitude goign from +180 to -180 (or somethign like that) didn't occur to anyone. RV pilot's being who they are, I wouldn't be surprised if we don't' have one or two F-22 drivers among us that have already been exposed to this...(and fortunately, this flight was on the wing of a tanker at the time!)

Subject: F-22 AEF Deployment
Date: 12 Feb 07
Narrative:

1. A 1st Fighter Wing AEF 6-ship (Petro 91) departed Hickam AFB
enroute to AEF location on 10 Feb. Approximately 4 hours into the
mission and coincidental with crossing over the International Date
Line, all six aircraft experienced a significant avionics failure
including:

Both GINS 1 and 2 Fail
FLCS Degrade
Radar Fail
Fuel Degrade
Loss of all attitude references
Loss of Flight Path marker
Loss of all navigation aides (TACAN, ILS, Computed, etc.)
Loss of all heading indications

2. Aircraft communications were available via backup radio only.
Only navigation available was via cockpit airspeed and altitude
indications (both deemed accurate). All other aircraft systems, to
include engines, electrical system and air refueling, were nominal.

3. Flight Lead, initiated via the tanker a
CONFERENCE HOTEL (CH) call with LM Aero. All CH team recommended
workarounds (avionics restarts, date and time resets, etc.) did not
resolve the problem.

4. Flight Lead assessed pressing to the AEF location but decided
to turn back and return to Hickam. He also directed the second
deployment cell, a 2-ship approximately one hour behind him, to return
to Hickam. NOTE: This 2-ship never crossed the International Date
Line.

5. Enroute back to Hickam, after crossing back over the International
Date Line, avionics restarts were unsuccessfully attempted.

6. All aircraft successfully recovered at Hickam, shut down (cold
iron), restarted engines and all avionics malfunctions cleared.

Wow...amazing what TIME can do to a system like that, huh? I would hate to conjecture as to the failure but I would guess they were keeping the internal clocks synchronized via GPS and when they jumped the date line everything got out of whack. At any rate...this is VERY interesting from a software engineer's perspective.

Many (ok, ALL) of the real-time systems I have worked on are very sensitive to time changes and the entire systems must be reset if the internal clock is updated.

I'm amazed that the F-22's systems weren't more fault tolerant. I'm even more amazed that this wasn't caught. I know that the F-22's testing program is incredibly detailed and they test every imaginable scenario. Weird.

But....?

Very, very interesting read and great work by all to recover.

Paul, my question now is, "Will our Garmin GPS's operate okay as we cross the Int'l date line"? Jon Johannson comes to mind with his three round-the-world trips.

Thanks,

It was the Longitude

Jamie - from further notes that I read, it appears that it was the longitude transition (not the time/date) that caused the problem. I'm obviously not familiar with the guts of the F22 software, so can't speculate deeper - but apparently they had tested in crossing over the poles and over the Prime Meridian....just not the "other side" of the world!

Single Point Failure

Sounds like that output from the computer was essentially a single point failure...

I'm surprised the reliability guys didn't find it a long time ago....

I worked with the guys designing that computer in El Segundo, and I did notice that the main mission computer did not drive the flight controls - different computer built by a different company.

You will notice none of the failures mentioned above hurt the pilot's ability to fly....

Just blame it on the SW folks.

gil in Tucson

IIRC the F-15(?) rolled inverted (or wold have) when it crossed the equator due to a + vs - problem in the software... you would think the SW testers would have worked out the stressing "limit" conditions by now and tested for them....

Ironflight said:

Don't put all your eggs in one basket. Plan for failure. Have a system or plan that gets you out of it without breaking a sweat.

Click to expand...

I think you've said before that dissimilar redundancy is your friend. Your backup systems should be of a sufficiently different architecture to avoid failing from the same event. Consider the impacts of following and how likely/not they are:

* Alternator failure
* Battery failure / runaway
* Wiring fault / failure
* Loss of GPS signal due to interference / rain / jamming / Area 51 / aliens
* Pitot or static icing / blockage
* Lightning strike
* Software bug

As the incident notes, even the most expensive EFIS/FMS/ETC can have software bugs that crop up and bite. Some of the failure modes are very bad in that the display / indications simply freeze - how long would it take you to figure that out? Who wants to be like those Airbus pilots and have the PDF go blank and then say "Please wait..."

Anyway, this begs the question that with the move towards EFIS in GA, whether we still need mechanical backup instruments. I will vote Yes with my dollars and include backup AS, ALT and perhaps electric AI in my RV. YMMV

I do stuff like this for a living. You would be surprised how difficult it is to actually perform an exhaustive test of your software.

The only way to really do it is to perform exhaustive code reviews, nit pick over EVERY line that goes in, test boundary conditions etc etc etc etc. After you've done ALL of that and you're absolutely sure that everything is correct, then you perform an exhaustive post-integration test as best you can. At this point, if something fails it's a big deal since this isn't really the "test"...it's merely a confirmation that all your previous testing was done properly.

This takes a LONG time and it makes even the most trivial of changes take days to make. This is the process I instituted and drive, though, and as a result we have extremely reliable software. Still, stuff falls through the cracks.

Look at it this way...if you had a mechanism with 100,000 little gears and cams and things, and they were all interconnected, how easy would it be to get the device to function perfectly in all circumstances? Now consider that it's not unusual to have 100,000 lines of code just driving one black box. Now consider how many little black boxes you have scattered around a typical fighter/sattelite etc. It's actually quite amazing the software reliability we're able to achieve given the level of precision and review it takes to get there.

And it's not like I haven't introduced a midnight-rollover bug once or twice in my life....LOL.

I have backup for my backup...

Sometime I think that I went overboard with my panel design, but when I read this I think that I did it right.
Two different EFIS units (same company BMA), EFIS Sport, EFIS Lite G3.
Those backed up by round versions of ASI, ALT, Turn & Bank.

Maybe I should tie a piece of yarn to the canopy just in case....

Kent

kentb said:

Maybe I should tie a piece of yarn to the canopy just in case....

Click to expand...

Is that in case your butt falls asleep and you can't coordinate turns by the "seat of your pants"? What's the MTBF on that?

On second thought, let's drop that idea - this is a family show...

Redundancy

Enjoying this thread. I fly IFR now and plan to in my -8 as well. Current plan is Dynon D10A, AFS 3400 (different AHRS and OS's) backed up by ASI, Alt, and TC. The Tru Trak Dig. II VSGV provides a stabilized heading reference, whether GPS signal being received or not. Finally there is a Garmin handheld (296/496) available with panel page which allows easy control of the airplane as we have demonstrated to our own satisfaction numerous times (as long as GPS signal is good). Both EFIS's have internal battery back up. I have two alternators. If this all gets off the ground I'll have more redundancy than ever in my life! For you software/computer gurus, are there many common failure pathways for both EIFS's you can imagine? What a great time we live in! Bill

Sounds like someone's FMEA was missing a couple of lines!

But like Paul said, it's never possible to think of every possible failure. Heck, look at how much trouble the early Daylight Savings Time cutover is causing network admins this year! Now who would'a ever thought of that failure mode?

This thread is excellent 'food for thought'. Given my limited experience behind glass and the fact that I'm building a glass panel, it's threads like this that guide my design process. What I find most interesting however is that every individual builder of a glass panel has a different rationale for failure modes and flight safety and will defend it to their death. With few exceptions, I find validity in everyones rationale but quite often only to the extent that they have not or will not listen to the opinions and experience of others. While NTSB reports will guide our future designs, I believe it's threads like this that will keep the reports to a minimun. Thanks to everyone for your help and guidance.

R.E.'Ernie' Butcher
N99SU -8 Flying

Further validation

This is an excellent validation for my plans to go steam gauge backup. I spend my days developing redundent electronic systems but you won't find me flying GA without my E6 and traditional gauges as a backup.

Even on the ground where weight and size are not an issue we have problems balancing geographic location, power, environment, communications, etc for fully redundent digital systems. Solid state gyros, glass screens have the ability to last longer and be more accurate than traditional gauges but they can not escape MTBF and software errors.
I personally like the story about the lunar lander module and it's computer being overloaded on the first moon mission. Can you think of a more scrutinized chunk of code? Sure it was many years back but it was also simpler than todays calculators! Todays software with custom OS, Graphic Engines, along with unknow libraries of functions run 100,000's of lines of source code.

Bill Dicus said:

For you software/computer gurus, are there many common failure pathways for both EIFS's you can imagine?

Click to expand...

Lots

But I can imagine lots of failure modes for mechnical contraptions too.

Based on the number of problems I've seen monitoring a bunch of different forums, the EFIS guys all really seem to have their act together. Most avionics companies, in fact, seem to have their act together. Maybe others don't think so but I happen to think it's pretty impressive how reliable the current batch avionics/moving maps etc actually are.

As someone else already said, though, if you need it to live, then you need a backup because nothing is ever gonig to be 100% reliable.

Simulations

jdeas said:

.....
I personally like the story about the lunar lander module and it's computer being overloaded on the first moon mission. Can you think of a more scrutinized chunk of code? Sure it was many years back but it was also simpler than todays calculators!
......

Click to expand...

JD... one major difference... everything is simulated now before any flight or even assembly.... digital simulators didn't really exist when the lunar lander was being developed... the computers were too small....

They didn't know the flight conditions at the moon ahead of time....

TEGAS ... one of the first digital logic simulators wasn't around until the late 70's.

Not true now... I bet every bit of the F-22 was simulated well before use... we delivered boards to use for software development at least 5 years before the first F-22 flight.... even if it did use a copier/scanner microprocessor...

It sounds like they didn't explore all or the boundary conditions during simulation... they should have run a virtual flight around the world...

gil in Tucson

PS ...my first boss at Hughes Aircraft was a MIT PhD who was on the design team for the lander computer... he was a strange guy who worked 24 hrs non-stop and then went home and slept for 16 hrs.... he was just as likely to be working at 3 am as 3 pm....

Simulators

Gil,
While I don't disagree with the basic context of your reply, I must say I have software updates for every EPLD, FPGA, software compiler, and simulator I have every owned. These updates were not sent to me to allow more functionality, they were bug fixes in the tools we use to build and test digital devices. No matter how good you are, your testing can never be better than your tools. Remember the Pentium math bug a few years back? Wonder how many sims were run at Intel before that one got away!

Judging from my early days, my bet is the "lander' code was 'hard coded' op code so the items above do not even come into play! While the program was sound, the operating environment was not fully simulated and that is most likely what caused the problem. (Sounds like you would know this better than I)

I am not trying to knock anyones profession. I have made a good living designing embedded devices for 30 years now. Many of these devices run for 10+ years only to be shut down when being removed from service. All I want to convey is that the more different your primary and backup systems are, the better chance you have for surviving an unforeseen incident. For me this is vacum gauge front and center and a space for the newest homebrew or purchased digital widget to my right.

BTW I have a friend who is part of the testing program for the F22 avionics. After speaking with her It would be foolish to think that the gear we have access to in GA has been tested to the extent of the systems designed for transport jets or the military yet notice the title of this thread.

Collective Wisdom! Where has it gone?

Interesting trigger for a redundancy discussion, Paul.
Regarding the F-22 longitude 180 incident, rather than accepting it is too complicated to test everything, I think we should be looking further up wind and asking, ?Why, was the 180? problem missed being incorporated and why was it missed in flight testing?? Crossing 180? longitude is hardly a unusual occurrence. There is a queue of heavy metal flowing both directions over the Dateline 24 hrs a day.
In the days when three generations of US engineers worked for companies like Douglas, those sort of things where handed down through the generations in the work force. ?Never put the Flap switch near the gear lever, son. One day some dumb pilot will select gear up after landing!?
The world has lost that continuity. You would be painfully aware of how much wisdom was lost to NASA with the demise of the Apollo Program. The problem here isn?t faulty code, it probably worked as designed. It?s that it did not occur to anyone in the design specification stage or to look at that aspect of the Flight Test Program.
?Collective Wisdom? is a great part of Aviation, but it needs to be disseminated. And how divorced geographically and interpersonally were the software engineers from the hardware engineers?

RV redundancy.
As I have said before, there are two different requirements.
VFR. Map, compass, watch, Mk I eyeball and the ability to fly without reference to ASI or Altimeter.
IFR. Pick your level, considering budget, weight and probable failure. Even a B747-400 can only have ONE Radar Antenna; and if that fails (and I have had it happen over India in CBs); much drawing of air through teeth.

Pete.

Expectations

And you can't design for ALL eventualities.

Loss of "Corporate Knowledge"

Peter has a great point, which might be a little off topic, but since I started the thread, I can take it any way I wish, right? It is very true that it is hard to pass on all of those little lessons that we learn in a long career in any business, even though we might have the best of intentions. That is part of why I stress the philosophy and rationale behind design decisions to young engineers, rather than just implementations. You have to ask "why" something is done a certain way to learn the real lessons of design.

I know that I am not the only one worried about the potential loss of knowledge in Manned Space Flight between the time that we retire the Shuttle in 2010, and the new generation vehicles are ready to fly in (maybe) 2015. Operations folks are high energy go-getters to whom five years of waiting can be an eternity. We are doing our best to plan for ways to keep them active and around so that we have the right mix of experience, knowledge, and skills for the next program.

To answer a little bit about Apollo testing (and no....I am not that old - but I was trained well by the guys that were here then!), a lot was done with analog simulation. Anyone that has flown over the Johnson Space Center has seen the large, arrow-shaped concrete pad on the "back 40" - it is a good quarter mile long. This is the old radar test facility, where targets could be mounted on a gantry and moved around, while the lunar radar was sitting in a building looking out at the field. This would have been hooked up to the computer simulator and software. The Apollo 11 overload was partially due to "unexpected" conditions" - something that just wasn't anticipated.

Which brings us back full circle to the theme of this thread....you can NEVER rule out all failures, because no human is smart enough to anticipate them all. It's a waste of time to try and build perfection - so instead, plan for failure, and have a back-up....

Paul

Ironflight said:

Which brings us back full circle to the theme of this thread....you can NEVER rule out all failures, because no human is smart enough to anticipate them all. It's a waste of time to try and build perfection - so instead, plan for failure, and have a back-up....

Paul

Click to expand...

Yep - you have to have systems that are resistant to failure modes that are frequent and have high impact. Alternator failures have a high frequency (at least with respect to risk analysis) - it happens a lot. Being struck by falling objects (blue ice, skydiver) is low frequency - it happens, but rarely.

The problem is the stuff inbetween. Lightning strikes - high impact, but what's the probability of this happening? If you never fly near TS, it's pretty low, but it can happen many miles from TS.

You can't prevent all risks, but the key is deciding what risks you're going to mitigate and being comfortable with that.

Yes.... but...

fodrv7 said:

And you can't design for ALL eventualities.

Click to expand...

That's true, but crossing the equator or the date line is a very well known - and expected - operational mode. I call that a boundary condition - one were something reaches a limit or changes sign....

The simulations performed really should have done this - I'm sure they did "virtual flights" in all sorts of modes and places....

I spent the last 10 years trying to get engineers to design more efficiently... and the biggest efficiency is not making any mistakes during the design process, and having to correct them later....

Speaking from a biased view as a hardware designer, the software guys acted differently...

gil in Tucson

Speaking of IFR

I passed my written this afternoon and my checkride (yes in the RV7a) is set for March 15th.

I have a lot of practising to do!

I am glad I now longer have to know about errors in mechanical instruments that were outdated before I was born...

Frank

Backups

My turn and bank is electric. I have the standard 6 pack with Vac DG and VAC art horiz. I am putting in an electric DG and elec Art horizon(I hope the gauges work). I also have a VOR/ILS LOC indicator and dual gps units with battery backup... so if my vac pump goes.... I have elec backup... if my electric goes... I have vac backup. and if the multitude of multicolored and multitextured matter hits the oscilating device... The GPS has a 5 pack and terrain and vertical guidance and wx with battery backup... even my gps manuf are different. If my software does not load... no big deal... my mags are both mechanical... no electric mags for me. I've seen way too much failure for Hard IMC under just one system. I run Full VAC and ELEC. Of course... if there is a loose nut behind the yoke... all is naught...
Best
Brian

Avionics software - don't get me started!!!

Ironflight said:

Anyone who reads about me and my airplane should have figured out by now that I am a committed Glass Cockpit IFR pilot

. . .

Subject: F-22 AEF Deployment
Date: 12 Feb 07
Narrative:

1. A 1st Fighter Wing AEF 6-ship (Petro 91) departed Hickam AFB
enroute to AEF location on 10 Feb. Approximately 4 hours into the
mission and coincidental with crossing over the International Date
Line, all six aircraft experienced a significant avionics failure
including:

Both GINS 1 and 2 Fail
FLCS Degrade
Radar Fail
Fuel Degrade
Loss of all attitude references
Loss of Flight Path marker
Loss of all navigation aides (TACAN, ILS, Computed, etc.)
Loss of all heading indications

. . .

Click to expand...

Paul, Paul, Paul,

Don't get me started about avionics software development. I already have a blood pressure problem!

The F-22 software program is a nightmare and has been from way back in the 90's - I have worked on it at Lockheed and have interviewed with Northrup for F-22 radar and again with Lockheed for flight displays years later. I have friends that have worked F-22. I cannot even begin tell you how absurd the situation is for developing software for that program. Wasn't it a few years back and they had something like a 45 minute average between flight software resets (as reported in Aviation Week)? That flight software is allowed to fly that EVER needs to reset amazes me.

Let me say this about error detection and correction in most modern avionics - it doesn't exist in any useful form. In most avionics systems I have seen, the logic is simple - if a "serious error" occurs then log it and shut down the system. If a less serious error occurs several time then log the occurrences and shut down the system. The point is that little or no effort is made to diagnose and recover to a degraded mode - just shutdown the system.

The Ada language did a beautiful thing years ago introducing the exception handler to the language along with strong type checking. With just a little smart programming and design, one could design software that could give a great deal of fault tolerance and recovery. But most programs have coding rules that eliminate the exception handeler from accepted programming practice. In some cases, the runtime checking of data values is also disabled - why? Because some (probably non-software literate) manager decided that they are too hard to understand and cannot be allowed in flight software. ARGH!!!

Hard to say where to begin with what is wrong with avionics software development today. Absurd management. Rediculously overstaffed programs. Lack of real software design talent. Beaurcracy beyond belief.

Let ME design F-22 software and it would NOT have done that.

180 over 117.

JCB

(BTW, I am strongly leaning towards a primary system of steam gauges augmented with fold down / fold up electronic panels in my own airplane. I just have never really seen flight displays that I really really liked. And I spent a lot of my career helping develop LCD color flight displays)

HW vs. SW

OldAndBold said:

.....
Hard to say where to begin with what is wrong with avionics software development today. Absurd management. Rediculously overstaffed programs. Lack of real software design talent. Beaurcracy beyond belief.

Let ME design F-22 software and it would NOT have done that.

180 over 117.

JCB

Click to expand...

Often, if the SW folks made a mistake (oops... coding defect), it would be fixed by a change in the HW, simply because the HW beauracy was much smaller than the SW bureaucracy mentioned above.

Program managers really liked the move to VHDL and FPGAs... fast HW changes! VHDL was not classified as SW... luckily for the HW designers...

I believe a lot of the F-22 SW was done in Ada, but a move to C++ was occurring...

gil in Tucson

Was a designer on the first generation F-22 mission computers...

az_gila said:

. . .

gil in Tucson

Was a designer on the first generation F-22 mission computers...

Click to expand...

I really got a kick out of this when I encountered it: It was 2002. I was interviewing for a job at Northrop. They were doing a NEW radar for the F-22 - and it was to replace the OLD radar for the F-22 that had gone obsolete while the F-22 was STILL BEING BUILT(!)

It's your tax dollars at work...

Using it wisely

Paul thanks for starring a great thread and attracting very astute comments from some very impressive resume-holders. I learned a great deal from this discussion even though I used to be in the IT business (business applications, not avionics).

Although I don't have any, the advantage of mechanical instruments is that they have less tendency to fail as a group (except where driven by common energy supply, vacuum or electric). Of course, the mechanical gyro's and indicators can be pretty sneaky so that's an offset. I'd rather, for instance, have GRT EFIS/EIS backed up by autopilot with internal compass and GPS 496 and backed up by a Dynon. Just an example and one where all the stuff mentioned can have its own backup electrical supply. OK, it's still not perfect and I know that. So here's what to do. IMHO.

Use it wisely. My first instrument instructor told me to always ask during the briefing where the nearest VFR is and never fly over fog. These are good examples of using it wisely; there are many more. What's your back door?

Any system of devices, especially with human interface in the chain, can fail. Multiple systems with different energy sources and different architectures can be less likely to fail together. But, common sources of data can take out otherwise independent systems. I mean by that pitot-static, GPS signals, and so on. In a reasonably priced single engine aircraft such as an RV, the final safety device is limiting yourself to flights where you can handle the failure when it happens.

Technology

OldAndBold said:

I really got a kick out of this when I encountered it: It was 2002. I was interviewing for a job at Northrop. They were doing a NEW radar for the F-22 - and it was to replace the OLD radar for the F-22 that had gone obsolete while the F-22 was STILL BEING BUILT(!)

It's your tax dollars at work...

Click to expand...

Yep.. the technology in Integrated Circuits/Microprocessors evolves a lot faster than fighter aircraft development (see Moore's Law)

It was funny to see your work go obsolete before the plane has even flown....

gil in Tucson

Test question:

Q: How many SW engineers does it take to change a light bulb?
A: None. It's a HW problem.

Interesting....

az_gila said:

Often, if the SW folks made a mistake (oops... coding defect), it would be fixed by a change in the HW, simply because the HW beauracy was much smaller than the SW bureaucracy mentioned above.

Click to expand...

In the private sector (at least at my company [Tektronix]), the hardware is developed first, with the software close behind. So hardware group will declare that their work is finished before us software types. If a hardware defect is found after this, there is an exception that the software engineers will develop a way to compensate for the hardware defect.

Kent

Feature creep :{

I work in a small shop where I must develop hardware and software. The hardware guys have it best. Once a design is on the board and running you have a very specific set of operational parameters to meet. Changing them cost time and money to outside vendors. Most short run products I have been involved in have less than 10 hardware revisions. Being that FPGA and EPLDs design are tied in so tightly to hardware, their revision level is also low.
Now watchout, here comes marketing and they just got a first look at 'the product'!
You know, with the raw data your receiving, you could also do X,Y, and Z. Make all the folders look like wings, and while your at it, lets put shadows on all the graphics. We could also ....(etc etc etc)

It never seems to end! I can't tell you how many clients I have worked with that are on their third or more revision of "new features" while the test proceedures for the prototype get clipped because of a single sucessful test!

Until the marketing/sale department learn the value of solid stable software (not the MS "who cares if it crashes as long as it boots quick) we will be stuck with software packages that are below par.
Those who have been in the buisness long enough to remember mainframes and their testing procedures must be just shaking their heads.....

JD

jdeas said:

I work in a small shop where I must develop hardware and software. The hardware guys have it best. Once a design is on the board and running you have a very specific set of operational parameters to meet. Changing them cost time and money to outside vendors. Most short run products I have been involved in have less than 10 hardware revisions. Being that FPGA and EPLDs design are tied in so tightly to hardware, their revision level is also low.
Now watchout, here comes marketing and they just got a first look at 'the product'!
You know, with the raw data your receiving, you could also do X,Y, and Z. Make all the folders look like wings, and while your at it, lets put shadows on all the graphics. We could also ....(etc etc etc)

It never seems to end! I can't tell you how many clients I have worked with that are on their third or more revision of "new features" while the test proceedures for the prototype get clipped because of a single sucessful test!

Until the marketing/sale department learn the value of solid stable software (not the MS "who cares if it crashes as long as it boots quick) we will be stuck with software packages that are below par.
Those who have been in the buisness long enough to remember mainframes and their testing procedures must be just shaking their heads.....

JD

Click to expand...

This is VERY true. When I did some contracting on the side (mostly 'cause the projects interested me), I would clearly lay everything out and provide free support for BUGS. It was very clear that new features, different behavior etc... would require another contract and more $$$. Period. End of story. I lost a contract because I refused to allow her to coerce me into free work, anymore than my painter will repaint my house for free because I changed my mind on color. 3 years later, she gave it back to me and I implemented her features for a fair price....in the long run, at about 1/10 what it cost the other guy to royally mess it up over my 3 year haitus.

No $$$ = no accountability = lousy engineering = WAY more $$$ to fix it = bad business.

They'll never learn.