Detecting Traffic Lights By Self-Driving Cars Is A Lot Harder Than You Think

Dr. Lance Eliot, AI Insider

Image for post
Image for post

Quick quiz for you.

There are approximately 300,000 of these in the United States alone.

You see them every day.

You notice them, but you don’t notice them.

They save lives, but are generally unheralded for what they do. Sometimes you curse them. Sometimes you thank them.

Can you guess what I am referring to?

Answer: Traffic lights.

Yes, as human drivers you are surrounded by traffic lights.

Most major intersections have them. You watch for that green light so that you can zoom through the intersection. When you see a yellow light, you need to judge whether to hit the brakes before the light turns red, or maybe hit the accelerator to try and make it before the light turns red. It’s a game.

It’s actually a deadly game.

There are an estimated 700 deaths per year due to red-light running crashes. That’s sad and preventable.

Equally horrific, there are an estimated 126,000 injuries due to red-light running. That’s also sad and preventable.

Considering The Nature Of Traffic Lights

When you think about it for a moment, you’ll realize that traffic signals are kind of a scary mechanism since they don’t do anything to actually physically stop anyone from driving crazily.

It is all a voluntary system.

It is our collective belief in the traffic signal that makes it real.

Imagine though the chaos without the traffic signals.

We would either all be having continual near misses, or the intersections would need to have stop signs or some other control mechanism, which would likely stall traffic and make our driving times longer. You could put traffic officers at intersections, which is the way things used to be, prior to the advent of traffic signals.

AI Autonomous Cars And Traffic Lights Aspects

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars and consider the traffic signal aspects as a crucial capability for self-driving cars.

Some automakers and tech firms consider the advanced traffic signal problem to be an “edge” problem in that it is not core per se of the act of driving a car. Allow me to qualify that point. Certainly we all agree that being able to detect a traffic signal is at the core of the driving task. But, this can be done in a rather simplistic manner, or it can be done in a more advanced manner. For some, once the simplistic version has been figured out, they move on to other self-driving car capabilities and consider the traffic signal problem entirely solved.

Let’s take a deeper dive into the topic of traffic signals.

First, how do you know that a traffic signal exists?

I am sure you are thinking that my question seems kind of silly and the answer is obvious. You look out your windshield and you see the traffic signal. Duh.

Well, as humans, we have an incredible ability of our eyeballs and our ability to see. We can look at a scene and find the gorilla that’s hidden over there behind the stack of boxes. Similarly, we can look at the street scene up ahead and “know” where the traffic signal is.

The sensors on the AI self-driving car have to put in a bit more work.

The cameras capture images of what’s ahead of the self-driving car. The image needs to be analyzed by the system. The image might contain not only a traffic signal, but perhaps there’s a plane flying through the sky that can be seen behind the traffic signal, maybe there’s a few birds resting on the traffic signal, maybe there’s rain coming down and the traffic signal is partially obscured by the heavy rain. And so on.

It’s not such an easy thing to find the traffic signal in a picture. Yes, the traffic signal is likely the same kind of shape nearly all of the time. It’s usually on a post of some kind. It’s got the three lights. It’s hanging over the intersection. These are all valuable clues. I’m not saying it is rocket science per se to find the traffic signal, but it is more work than you think.

So, the first step involves capturing images via the sensors and analyzing those images to find the traffic signal. You want to find the traffic signal and also not be fooled by something that might resemble a traffic signal. There could be other nearby lights such as for a lit-up billboard or maybe lights on the exterior of a building. Those might be red lights, yellow lights, green lights, and so you cannot just look for a particular color of a light.

You also might be faced with the circumstance of an intersection that does not have a traffic light. If the image analysis says that it cannot find a traffic signal, does this mean for sure that there isn’t one there? Maybe yes, maybe no. It could be that the image analyzer capability could not find the traffic signal. If the sensor analysis reports to the sensor fusion that there isn’t a traffic signal, and if turns out there is one there, the result could be catastrophic.

What Happens During Detection

Let’s pretend then that the visual sensor and analysis did not find a traffic signal, but that the traffic signal does exist. The sensor feeds its results into the sensor fusion. The sensor fusion compares each of the sensory device indications to try and triangulate and make sure that no one sensor is misleading or maybe has failed to do its job.

Would the radar on-board of the AI self-driving car detect the traffic signal? This can be tricky in that the radar might not have enough of a profile to bounce a radar signal off the traffic signal and detect it. Would the LIDAR detect it? This depends on whether there is even LIDAR available (for Tesla’s they aren’t using LIDAR).

Anyway, the point is that a traffic signal could exist but not be detected by any of the sensors of the AI self-driving car.

Imagine then that the self-driving car believes that it can just barrel through the intersection.

This could be wrong to do and perhaps there is a red light and the cross traffic is going through the intersection. Crash!

Now I realize you might object and say that the AI self-driving car has hopefully detected the other cars that are going through the intersection and so would realize that something is afoot. Furthermore, if there is other traffic next to the AI self-driving car, it would presumably be slowing down and stopping, due to the presumed red light, and so the AI self-driving car should notice that the other cars nearby it are stopping and so it should consider stopping too.

This brings up my earlier point about simplistic traffic signal capabilities versus more advanced traffic signal detection capabilities. If the AI and the self-driving car is solely programmed to visually find the traffic signal, this can be a significant risk as to being able to properly determine when a traffic signal is present or not. Some AI systems for self-driving cars only do the visual detection. Adding the sensor fusion and the other sensors data is considered “more advanced” — and likewise, comparing to the traffic situation is even more advanced. In other words, if you are only thinking of the traffic signal as just an object, you would only care whether you detected that particular object or not.

We know though as humans that we use all sorts of other clues to figure things out.

Edge Cases Upon Edge Cases

I’ll add more twists to the traffic signal problem.

Suppose the traffic signal is there but it is not working?

Maybe the power is out, maybe it has had a failure, etc. I know you might think that it low odds that a traffic signal exists but is not working — it is though a possibility.

Another variant to deal with involves the visibility of seeing the traffic signal.

Suppose the camera has dirt on the lenses and only gets a partial image. The lighting near the traffic signal can also impact detecting the status of the traffic signal. Have you ever driven up to an intersection and the sun was directly in your eyes? You could barely see the traffic signal lights. You knew that there was a traffic signal there, but it was nearly impossible to see if the light was red, yellow or green.

Indeed, you’ve undoubtedly noticed that most modern traffic signals make use of hoods to help protect the light and make the light stand-out. Often, a traffic signal light is aimed at a particular angle to try and make it more visibly apparent. These elements can help the image processing. They can also make the image processing more difficult, depending upon how much the hood hides the light or that angle of the light is a kilter of where the camera on the car is.

Future Of Traffic Lights Is Not Here Today

One hope is that the future traffic signals will be “smart” instead of dumb.

The smart traffic signal has more advanced capabilities than the traditional “dumb” one does.

For example, a smart traffic signal might emit an electronic signal indicating the status of the traffic light. In that case, the AI self-driving car can potentially receive an electronic signal rather than relying only on a beam of light. This can significantly aid the detection of the traffic signal.

Those that are looking further into the future would even say that the traffic signal as we know it today will ultimately no longer exist. If we later on opt to get rid of all human driven cars, and we had only AI self-driving cars, presumably the use of lights to signal the status of the traffic signal is no longer needed. It could be just an electronic signal. Also, there would not be a need to have the large pole that currently houses the traffic signal. You could put the traffic signal electronic emitter in a squat box near the intersection instead.

I like to point out to those futurists that it will be a long time before we have only AI self-driving cars.

For the foreseeable future, we will have a mix of human driven cars and AI self-driving cars (there are around 200 million conventional cars today in the United States alone). As such, we’ll need to keep the light emitting traffic signals for those “darned” human drivers. There are experiments currently involving sending an electronic signal to your smartphone, thus, in theory, we might be able to have human drivers that rely upon their smartphones to let them know the intersection status, rather than looking at a traffic light. I don’t think that’s going to hold-up though and we are likely to continue with the traffic signals as they are.

Where we will likely have advances towards smart traffic signals will be the ability of the traffic signal to adjust the timing of the lights based on more informed traffic analytics. A city might have traffic detecting sensors throughout the city, some being on buildings, some embedded in the roadway, some collected via flying drones, and so on. These sensors will collect traffic info. The traffic info will be analyzed and fed to the traffic signals so that the traffic signals can adjust their timing. This could greatly reduce gridlock.

Your AI self-driving car might also get connected to these traffic analytics.

V2I Will Eventually Occur

There will be the advent of V2I (vehicle to infrastructure) for AI self-driving cars.

This means that AI self-driving cars will be able to electronically communicate with the street infrastructure, including traffic signals, and with bridges (which can indicate how crowded they are), and tolls, etc. All of that will aid in navigating traffic signals. There is also going to be V2V (vehicle to vehicle communications), involving self-driving cars electronically communicating with other self-driving cars.

Smart traffic signals might have some unintended consequences.

Suppose a hacker was able to connect to a traffic signal and force it to show whatever red/yellow/green the hacker wanted to display. Suppose the traffic signals are all interconnected so as to allow for timed activity across the entire city, but then the hacker can opt to control all of those at once. There is also the privacy aspects that maybe a traffic signal is observing traffic and capturing license plates or other info about the cars passing through the intersection — would this be a good thing or a bad thing? It might help when trying to find criminals, but it might be used for nefarious violations of privacy.

Here’s one that might be somewhat chilling too.

Should your AI self-driving car be allowed to run a red light or not?

You might argue that the AI should never be allowed to carry out an illegal act. But, suppose that you are bleeding to death as a human occupant in an AI self-driving car and there are no other cars nearby, wouldn’t it be OK to run the red light so as to get to a hospital? There are numerous scenarios involving situations of these kinds. Also, if your AI self-driving car does run a red light, should it tell the police? Should it issue you a ticket?


AI self-driving cars need to make sure they are well versed in dealing with traffic signals.

The simplistic approach might be OK for the moment, when we have only a few AI self-driving cars on the roadways, and while they are used only in carefully mapped and geo-fenced areas.

Once AI self-driving cars become more prevalent, it will be a life or death matter as to whether they can handle traffic signals. This means that the AI needs to become more advanced.

I’m giving a green light for that to happen.

For free podcast of this story, visit:

The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.

More info about AI self-driving cars, see:

To follow Lance Eliot on Twitter:

For his blog, see:

For his Medium blog, see:

For Dr. Eliot’s books, see:

Copyright © 2019 Dr. Lance B. Eliot

Written by

Dr. Lance B. Eliot is a renowned global expert on AI, Stanford Fellow at Stanford University, was a professor at USC, headed an AI Lab, top exec at a major VC.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store