Haven't found that software glitch, Toyota? Keep trying - l...
http://www.latimes.com/news/opinion/opinionla/la-oew-...
OPINION
LOCAL
U.S. & WORLD
EDITORIALS
OP-ED
BUSINESS
SPORTS
LETTERS
OPINION L.A.
IN THE NEWS: HEALTHCARE
LAKERS-SUNS
ENTERTAINMENT
VACCINES-AUTISM
HEALTH
TOYOTA
LIVING
TRAVEL
OPINION
MORE
Search
GO
PACQUIAO-CLOTTEY
adv er ti s em ent
BLOWBACK
Haven't found that software glitch, Toyota? Keep
trying
An electronics problem isn't to blame for the sudden acceleration, say the carmaker's engineers.
That's nearly impossible to conclusively determine through laboratory tests.
ADS BY GOOGLE
By David M. Cummings
May the Best Car Win
Find Peace of Mind in a New
Chevy Malibu. Offers and Details
Here.
March 11, 2010 | 3:23 p.m.
www.Chevrolet.com/MalibuWins
Testing Laboratory
Chemical, Mechanical,
Metallurgical NDT, and
Environmental Testing
www.LabInc.com
E-mail
Print
Share
Text Size
There has been a lot of speculation recently that Toyota's
problems with sudden acceleration may be caused by a
problem in the vehicles' electronics systems. The
"electronics" includes millions of lines of software running
on the automobiles' computers. As The Times reported on
March 3, Toyota's chief engineer testified to Congress that
the company has done extensive testing on its cars'
electronics and believes they are not the cause of the
sudden acceleration.
Acai Berry
EXPOSED
(Consumer Report)
Rochester Mom
Discovers $3
Whitening Trick
Rochester W arning! Health
Reporter Discovers The
Shocking Truth!
Dentists DO NOT want
you to know about THIS
teeth whitening secret!
Read More
Read More
The Latest | NEWS AS IT HAPPENS
Having owned a Toyota myself, I have always been a fan of what I perceived to be the
automaker's high standards for quality. I also happen to have more than three decades of
experience designing, building and researching reliable computer systems, many of which are
embedded inside other devices. Based on this experience, I find it very difficult to accept the
statements from Toyota's chief engineer. And the implications extend beyond Toyota, to all other
companies that rely on software for their product safety.
As anyone with experience in embedded systems will tell you, there are nasty software bugs that
can be extremely difficult to reproduce in a laboratory test environment. To illustrate, I'd like to
describe one such bug we encountered at the Jet Propulsion Laboratory while developing the
flight software for NASA's Mars Pathfinder spacecraft.
Because of Pathfinder's high reliability requirements and the probability of unpredictable hardware
errors due to the increased radiation effects in space, we adopted a highly "defensive"
programming style. This included performing extensive error checks in the software to detect the
possible side effects of radiation-induced hardware glitches and certain software bugs.
One member of our team, Steve Stolper, had a simple arithmetic computation in his software that
was guaranteed to produce an even result (2, 4, 6 and so on) if the computer was working
correctly. Many programmers would not bother to check the result of such a simple computation.
Stolper, however, put in an explicit test to see if the result was even. We referred to this test as
his "two-plus-two-equals-five check." We never expected to see it fail.
Lo and behold, during software testing we saw Stolper's error message indicating the check had
failed. We saw it just once. We were never able to reproduce the failure, despite repeated
1 of 4
03/13/2010 01:22 PM
Haven't found that software glitch, Toyota? Keep trying - l...
http://www.latimes.com/news/opinion/opinionla/la-oew-...
attempts over many thousands if not millions of iterations. We scratched our heads. How could
this happen, especially in the benign environment of our software test lab, where radiation effects
were virtually nonexistent? We looked carefully at Stolper's code, and it was sound.
The only viable theory we could come up with was that an interrupt (an external hardware
stimulus such as a timer going off) had occurred at just the right microsecond within the execution
of Stolper's software. Furthermore, we theorized, the operating system (the equivalent of
Windows on the flight computer) had a bug that caused it to misremember whether an arithmetic
carry had occurred just before the interrupt. Although highly unlikely, it was the only credible
explanation we could come up with. Because this was a new version of the operating system built
for Pathfinder, still not yet fully tested itself, this theory had some credibility.
We reviewed the operating system code and consulted with the company that developed it. Much
to our surprise (and relief), we found that there was indeed a bug in the interrupt handling
software as we had theorized. If Stolper had not put in his "two-plus-two-equals-five check," we
might not have found the problem until it was too late -- that is, until it caused a catastrophic error
en route to Mars. (There were other subtle bugs that we found and fixed before and after launch,
some arguably subtler than this one.)
So what's my point? First, I don't know if Toyota's engineers embrace the software reliability
approaches we embraced on Pathfinder, which allowed us to catch these subtle bugs. Second,
even if the Toyota engineers do everything we did on Pathfinder and more, I'm still s keptical when
I hear an engineer declare a complex software system to be bug-free based on laboratory
testing. It is extremely difficult to make such a determination through laboratory tests. I'm quite
certain none of the members of the Pathfinder software team would have declared the s oftware to
be bug-free prior to launch, despite our best efforts to make it so.
If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my
response is simple: Keep trying. Find new ways to instrument the software, and come up with
more creative tests. The odds are that there are still bugs in the code, which may or may not be
related to unintended acceleration. Until these bugs are identified, how can you be certain they
are not related to sudden acceleration?
My last point is this: Whatever the final outcome of the Toyota saga, this should serve as a
wake-up call to all industries that increasingly rely on software for safety. It is probably only a
matter of time before a software error results in injury or death, if it has not happened already
(there are some who say it has). We need to minimize that possibility by enforcing extremely
stringent standards on the development and testing of software in all safety-critical systems,
including, but not limited to, automobiles.
David M. Cummings, executive vice president of the Santa Barbara-based Kelly Technology
Group, spent nine years as a consultant for the Jet Propulsion Laboratory, where he worked on
the Mars Pathfinder spacecraft.
Copyright © 2010, The Los Angeles Times
RELATED STORIES
From the L.A. Times
Toyota plans live webcast from Torrance to support its findings on sudden acceleration
Lawmakers ask Toyota to hand over results of electronics tests
Toyotas’ speed cited in deaths
Stay on top of your credit score
Get 3 Free Credit Scores with 3 Bureau Credit Monitoring!
www.Equifax.com
2 of 4
Get Listed
03/13/2010 01:22 PM
Haven't found that software glitch, Toyota? Keep trying - l...
E-mail
Print
Digg
Twitter
Facebook
Stum bleUpon
http://www.latimes.com/news/opinion/opinionla/la-oew-...
Share
COMMENTS (11) | Add Comment
It seems like a bad idea to begin with to link the gas pedal to software. Is it too s implistic to
devise a 'kill' switch for emergencies that could stop acceleration while allowing power steering
and braking to still work?
karma2 (03/12/2010, 5:53 PM )
Report Comment
Why do we need fly by wire for cars period, the old cable system works great in my Chevy
Cavalier with 75 thousand trouble free miles on the odometer. Toyota's quality control has
been slipping for about ten years now and seems to be getting worse every day, GO BACK
TO THE WIRE CABLE AND ONLY ALLOW THE COMPUTER TO MONITOR AND ADJUST
AIR FUEL RATIO ETC. The japanese are notorious for over engineering some products, the
Honda I had was the same way to much redundancy it would cause its own ghost problems
that could not be duplicated by the dealer so I got rid of it. KEEP IT SIMPLE STUPID, every
company needs a KISS program and stop taking control of the car away from direct input by
the driver.
TFEDUP (03/12/2010, 5:27 PM )
Report Comment
The problem is that the "absence of evidence is not the same as the evidence of absence"
and it is impossible to for Toyota to demonstrate such evidence of absence, i.e. software or
electronic bug does NOT exist. It is logically impossible, just like it is impossible to demonstrate
to a child that the boogeyman isn't going to get them. All you can show is that the boogeyman
isn't in the closet or under the bed. It is up to car buying public to decide as whic h point they
are satisfied. On the other hand Toyota has not be straight or forthcoming with their
information and tried to brush aside complaints, along with profit driven motives well justified by
the capitalist system, puts them in a very difficult position. However, if we really believe in the
principle of "innocent until proven guilty" that is beyond "reasonable doubt", and then to
excruciate Toyota and insist they demonstrate that their electronic or software is bug free is
the same as assuming Toyota is guilty. Thus forcing Toyota to prove their innocence, i.e. their
system is flawless, which has about a much chance as O.J. proving that he didn't kill his wife.
g35xfun (03/12/2010, 1:54 PM )
Report Comment
See All comments
If you are under 13 years of age you may read this message board, but you may not
participate. Here are the full legal terms you agree to by using this comment form.
Comments are filtered for language and registration is required. Web and/or e-mail
addresses are not permitted. Note: Comments are moderated and will not appear until they
have been reviewed by Los Angeles Times staff.
Google Maps adds bike
routes for L.A., other cities
Designers and celebrities
collaborate for Oscars
Playing along with the
Mozart effect
Plaschke: Lakers aren't
playing like champions
Loss brings Taiwanese
traditions back | Photos
More spotlights...
3 of 4
03/13/2010 01:22 PM
Haven't found that software glitch, Toyota? Keep trying - l...
Coastline Pilot
Daily Pilot
Baltimore Sun
Chicago Tribune
Huntington Beach Independent
CTnow
Daily Press
Valley Sun
Burbank Leader
Hartford Courant
http://www.latimes.com/news/opinion/opinionla/la-oew-...
News Press
Los Angeles Times
KTLA
Hoy
Orlando Sentinel
Brand X
Sun Sentinel
LA, Los Angeles Times Magazine
ZAP2it
The Morning Call
Terms of Service | Privacy Policy | Los Angeles Times, 202 West 1st Street, Los Angeles, California, 90012 | Copyright 2010
A Tribune Web site
4 of 4
03/13/2010 01:22 PM