AI Brittleness Is A Go-To Fail: AI Driverless Cars Vulnerable Too

Dr. Lance B. Eliot, AI Insider

Image for post
Image for post
Famous software bug made it onto T-shirts

I’m guessing that you’ve likely heard or read the famous tale of the Dutch boy that plugged a hole in a leaking dam via his finger and was able to save the entire country by doing so. There are two lessons rolled into one tale.

The first part is that the weakest link in a chain can become broken or severed and thus the whole chain no longer exists as a continuous chain.

The second part of the lesson learned involves the cascading impact and how severe it can be as a consequence of the weakest link giving way.

If you are not convinced that a single weakest link could undermine a much larger overall system, I’d like to share with you the now-famous account of the so-called “goto fail goto fail” saga that played out in February 2014. This is a true story. The crux of the story is that one line of code, a single “Go To” statement in a software routine, led to the undermining of a vital aspect of computer security regarding Apple related devices.

It turns out that the inadvertent use of a somewhat errant “Go to” statement created an unreachable part of the program, which is often referred to as an area of code known as dead code. It is dead code because it will never be brought to life, in the sense that it will never be executed during the course of the program being run.

There is a strong possibility that the programmer goofed-up and didn’t intend to have dead code. Our inspection of the code won’t immediately tell us whether the programmer put the dead code there for a purposeful reason, or they might have accidentally formulated a circumstance of dead code and not even realize they did so.

Infamous Dead Code Example

You are now ready to see the infamous code (it’s an excerpt, the entire program is available as open source online at many code repositories).

Here it is:

if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)

goto fail;

goto fail;

if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)

goto fail;

err = sslRawVerify(ctx,

ctx->peerPubKey,

dataToSign, /* plaintext */

dataToSignLen, /* plaintext length */

signature,

signatureLen);

if(err) {

sslErrorLog(“SSLDecodeSignedServerKeyExchange: sslRawVerify “

“returned %d\n“, (int)err);

goto fail;

}

fail:

SSLFreeBuffer(&signedHashes);

SSLFreeBuffer(&hashCtx);

return err;

Observe that there appear to be five IF statements, one after another. Each of the IF statements seems to be somewhat the same, namely each tests a condition and if the condition is true then the code is going to jump to the label of “fail” that is further down in the code.

All of this would otherwise not be especially worth discussing, except for the fact that there is a “goto fail” hidden among that set of a series of five IF statements.

It is actually on its own and not part of any of those IF statements. It is sitting in there, among those IF statements, and will be executed unconditionally, meaning that once it is reached, the program will do as instructed and jump to the label “fail” that appears further down in the code.

One line of code, a seemingly extraneous GO TO statement, which is placed in a manner that it creates a chunk of unreachable code. This is the weakest link in this chain. And it creates a lot of troubles.

By the way, most people tend to refer to this as the “goto fail goto fail” because it has two such statements together. There were T-shirts, bumper stickers, coffee mugs, and the like, all quickly put into the marketplace at the time of this incident.

What was the overall purpose of this program, you might be wondering? It was an essential part of the software that does security verification for various Apple devices like their smartphones, iPad, etc.

Unfortunately, regrettably, the dead code is quite important to the act of validating an SSL/TSL certificate and other factors. Essentially, by bypassing an important part of the code, this program is going to be falsely reporting that the certificate is OK, under circumstances when it is not.

Some would describe this as an exemplar of software being brittle.

One theory is that it was purposely planted into the code, doing so at the request of someone such as perhaps the NSA. Another theory is that the programmer or programmers (we don’t know for sure if it was one programmer, and so maybe it was several that got together on this), opted to plant the GO TO statement and keep it in their back pocket. This is the kind of thing you might try to sell on the dark web.

There’s a slew of theories about the matter being a mistake. You are a harried programmer. You are churning out gobs of code. While writing those IF statements, you accidentally fat finger an extra “goto fail” into the code. At the time, you’ve indented it and so it appears to be in the right spot.

Another theory is that the programmer had intended to put another IF statement into that segment of the code and had typed the “goto fail” portion, but then somehow got distracted or interrupted and neglected to put the first part, the IF statement part itself, into the code.

Yet another variation is that there was an IF statement there, but the programmer for some reason opted to delete it, but when the programmer did the delete, they mistakenly did not remove the “goto fail” which would have been easy to miss because it was on the next physical line.

On a similar front, there is a bunch of IF statements earlier in the code. Maybe those IF statements were copied and used for this set of IF statements, and when the programmer or programmers were cleaning up the copied IF statements, they inadvertently added the unconditional GO TO statement.

This then takes us to another perspective, namely if any of those aforementioned theories about the mistaken action or the intentional action are true, how come it wasn’t caught during a code review, or during testing?

You might wonder how the compiler itself missed this aspect. Some compilers can do a kind of static analysis trying to find things that might be awry, such as dead code. Apparently, at the time, there was speculation that the compiler could have helped, but it had options that were either confusing to use, or when used were often mistaken in what they found.

Ongoing Debate on Dangers of GO TO Statements

There are some that have attacked the use of the GO TO statements throughout the code passage. You might be aware that there has been an ongoing debate about the “dangers” of using GO TO statements. Some have said it is a construct that should be banned entirely. Perhaps the debate was most vividly started when Edgar Dijkstra had his letter published in the Communications of the ACM in March of 1968. The debate about the merits versus the downsides of the GO TO have continued since then.

You could restructure this code to eliminate the GO TO statements, in which case, the extra GO TO would never have gotten into the mix, presumably.

Another aspect involves the notion that the “goto fail” is repeated in the offending portion, which some would say should have actually made it visually standout. Would your eye tend to catch the same line of code repeated twice like this, especially a somewhat naked GO TO statement? Apparently, presumably, it did not. Some say the compiler should have issued a warning about a seemingly repeated line, even if it wasn’t set to detect dead code.

You might also point out that this code doesn’t seem to have much built-in self-checking going on.

In the software field, we often speak of the smell of a piece of code. Code-smell means that the code might be poorly written or suspect in one manner or another, and upon taking a sniff or a whiff of it (by looking at the code), one might detect a foul odor, possibly even a stench.

Software developers also refer to technical debt. This means that when you right somewhat foul code, your creating a kind of debt that will someday be due. It’s like taking out a loan, and eventually the loan will need to be paid back. Bad code will almost always boomerang and eventually come back to haunt.

Suppose the programmer was hampered and not being provided by sufficient tools to do their work. Suppose the manager was pushing the programmer to just get the work done. Suppose the schedule was unrealistic and shortcuts were taken. It takes a village to develop software. If the village is not of the right culture and approach, you are going to get software that matches to that culture.

Potential Impact on AI Self-Driving Car Software

What does this have to do with AI self-driving cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. The auto makers and tech firms doing likewise are hopefully doing the right thing in terms of how they are developing their software, meaning that they need to recognize the dangers of the brittleness of the AI systems they are crafting.

Brittleness of the AI for an AI self-driving car is quite serious. If the AI encounters a weak link, imagine if it happens when the self-driving car is doing 65 miles per hour on a crowded freeway. Lives are at stake. This AI is a real-time system involving multi-ton cars that can quickly and in a deadly manner determine the life or death of humans.

I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved.

For self-driving cars less than a Level 5, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car.

Returning to the topic of the “goto fail” and AI brittleness, we all need to realize that such a one line of code could upset the AI self-driving car cart, so to speak.

In theory, the AI systems of AI self-driving cars should have numerous checks-and-balances. The chances of any single line of code causing havoc should be extremely low. There should be fail-safe capabilities. The testing should be extremely extensive and exhaustive. Simulations should be used to help ferret out such anomalies prior to getting into the code of a roadway running self-driving car. And so on.

That’s the theory of it.

The real-world is different. In many of these AI systems there are tons of third-party code that is being used, and other packages and open source being used. For the AI developers tasked with developing the AI of the self-driving cars, they are likely assuming that those other bodies of code are already well-tested and will work as intended.

Maybe yes, maybe no.

There is such tremendous pressure to get AI self-driving cars onto the streets, pushed by this relentless idea that whomever is first will somehow win this moonshot race, there is likely a substantial amount of cutting of corners in terms of code reviews, tools being used, and the like.

I realize that some will say that this is yet another reason to rely upon Machine Learning and Deep Learning. Rather than writing code, you presumably can base your AI system for a self-driving car on the use of packages that can emit a large-scale artificial neural network and let that be the core of your AI for the driving task.

At this time, the AI stack for self-driving cars is still primarily of a more traditional nature and the Machine Learning and Deep Learning is mainly for only selected elements, most notably for the sensor data analyses. The rest of the AI is done the old-fashioned way, and for which the single line of code and weak link are a real possibility.

I don’t want to leave you with the impression that somehow the Machine Learning and Deep Learning is a silver bullet in this matter. It is not.

The packages used for the Machine Learning and Deep Learning could certainly have their own weaknesses in them. The resultant runnable neural network might be flawed due to some flaw within the Machine Learning or Deep Learning code itself. The executable might be flawed. We already know that the neural network itself can be “flawed” in that you can do various sensory trickery to fool some of the large-scale neural networks being constructed.

Conclusion

The Dutch boy stopped the dam from breaking by plugging the hole with his finger. Heroic! We can all rejoice in the tale. It provides us with the realization that sometimes small things can have a big impact. There is the lesson that the weakest link, this seemingly inconsequential hole, could lead to much greater ruin.

How many of today’s budding AI self-driving cars are right now underway and have a hole somewhere deep inside them, waiting to become the spigot that regrettably causes the rest of the AI system to go awry and a terrible result occurs. Nobody knows.

How much effort are the auto makers and tech firms putting toward finding the hole or holes beforehand?

How many are putting in place error handling and error processing that once a hole arises during actual use and after deployment, the AI will be able to recognize and deal safely with the hole?

I hope that the tale of the Dutch boy will spark attention to this matter. I tried to showcase how this can happen in the real-world by making use of the infamous “goto fail goto fail” incident. It is a nice choice for this purpose since it is easily understood and readily publicly discussed. No need to search high and far to find some seemingly arcane example that most would try to write-off as inconsequential.

There is a huge body of water sitting at the dam, which we’ll say is the public and their already nervous qualms about AI self-driving cars. If even one hole opens up in that dam, I assure you the water is going to gush through it, and we’ll likely see a tsunami of regulation and backlash against the advent of AI self-driving cars. I don’t want that. I hope the rest of you don’t want that. Let’s make sure to put in place the appropriate efforts to seek out the weakest links in our AI systems and find them before it finds your AI self-driving car system, so we can keep it from destroying the whole dam.

For free podcast of this story, visit: http://ai-selfdriving-cars.libsyn.com/website

The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.

More info about AI self-driving cars, see: www.ai-selfdriving-cars.guru

To follow Lance Eliot on Twitter: @LanceEliot

Copyright 2019 Dr. Lance Eliot

Written by

Dr. Lance B. Eliot is a renowned global expert on AI, Stanford Fellow at Stanford University, was a professor at USC, headed an AI Lab, top exec at a major VC.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store