Car Sounds Can Be Helpful For Self-Driving Cars That Are Paying Attention

Dr. Lance Eliot, AI Insider

Image for post
Image for post

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column:]

The sounds of silence.

You might have thought that I was referring to the famous song by Simon and Garfunkel, but I am actually referring to the moments in which silence is golden. In particular, I’m referring to cars that are “silent” in that they aren’t making any rattling, grinding, clanging, or banging sounds as you happen to be riding in them.

Car Noises Usually Are A Bad Sign

Usually, any such ear catching sounds are bound to mean that there is something wrong with the car and you are likely at risk while riding in the vehicle.

At any moment, the engine might seize up, the axle might crack and fail, or any number of maladies might emerge.

Of course, sometimes the disturbing noise coincides with the actual failure event itself, in which case, the car problem is likely more apparent than the noise that you might have heard (the car sharply swerving, or the engine halting tends to draw more attention than the noise itself).

Car Noises Are A Helpful Sign

For me, I’m appreciative when a car makes those seemingly unwelcome sounds or noises.

How lucky that the car has kindly chosen to let me know something is amiss.

The perturbing aspect is that suppose you opt to take your car into a mechanic, and yet the sound perchance no longer is being emitted.

The car mechanic might look at you as though you are crazy.

Equally frustrating involves the mechanic not only unable to hear the sound, and an assertion that without the sound they cannot tell you what might be wrong, but they then offer they must do a complete diagnosis of your car.

On the topic of trying to use the car’s sounds for diagnosis purposes, it at times makes sense that the mechanic might not hear the noises that you heard. Suppose the sound only arises when the car is actually in-motion. If you take your car to the repair shop and while it is parked you turn on the car, this might not be a sufficient replication of the circumstances under which the sound is generated. There is a strong possibility that the noise only arises when the car is in actual motion.

Another factor can involve whether the car is heated-up.

Sometimes a noise might occur only when the engine is cold and upon first starting the car. In other cases, the noise only arises after the car has been going for a solid 30 minutes or more and the engine and the car components are heated-up.

Maybe the sound only occurs while shifting gears.

Perhaps the sound only happens when the car is taking a sharp turn. The noise might arise when going uphill, or only when going downhill. This highlights that where the car is being driven can make a difference in stoking the noise, along with how the car is being driven.

A key aspect about car sounds is that you usually need to know when they arose.

Yet another factor is whether the sound is a one-time instance, or whether it is intermittent or periodic.

Patterns And Car Diagnosis Via Noises

You might be thinking that these various car sounds and noises that relate to something untoward about the car are all ad hoc and have no pattern to them.

Not so!

Car mechanics can often make a pretty good guess about what might be wrong with your car via hearing the sound that it is making.

The car mechanic doesn’t necessarily need to hear the actual sound and can use your description of the sound to make a guess about what is wrong.

Today, there are several online databases of car sounds and noises, along with indicating what the noise or sound might suggest about any problems of your car. You can go to the database and listen to the sounds, trying to see if any match to the sound you hear in your car. You can also try recording the sound that occurs in your car and try to get it to match to the database, doing a calculation-type of matching using a sound conversion algorithm.

Car Sounds Detection And AI Autonomous Cars

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars.

Detecting car sounds and noises is a helpful and some might say essential added element that the AI ought to be providing in a self-driving car.

Detection Then Diagnosis

It is likely that true AI self-driving cars will be equipped with internal microphones for the primary purpose of interacting with human passengers.

Humans inside an AI self-driving car will most certainly need to have an ongoing dialogue with the AI system, using an AI-equipped Natural Language Processing (NLP) capability, so that the human can indicate where they want to go.

With those microphones already inside an AI self-driving car, the potential for also using that equipment as listening devices to detect car sounds and noises is readily heightened.

One issue though that has yet to be resolved involves the privacy elements of passengers inside true AI self-driving cars.

While you are a passenger in an AI self-driving car, should the microphones be activated and possibly recording your every word?

For some, this is a significant potential intrusion into their privacy.

Another aspect to be considered is whether the “normal” microphones used to interact with the passengers will be sufficient for purposes of identifying and “hearing” the car sounds and noises that might occur.

If the microphones are positioned in a manner to pick-up the human voice, they might not be well-positioned for picking up car sounds or noises, which would usually emanate from below the flooring of the car. This might necessitate adding additional microphones into the self-driving car, which also adds cost, weight, and potentially impacts the interior design and space available.

This does bring up another facet of the car sounds and noises.

Suppose a human passenger hears a car sound or noise, and they wonder what it might mean. It is conceivable the human passenger might ask the AI about the sound or noise. Did you hear that, the human asks the AI. What was that sound, the humans asks the AI.

Human Passenger Interaction With The AI System

It certainly seems likely that a human passenger would ask the AI about the car sounds or noises.

If you were in a ridesharing car or a cab driven by a human, and you heard a seemingly disturbing sound coming from the car, I would bet that you’d ask the human driver about it. You’d want to validate that they heard the sound too.

Human passengers in an AI self-driving car are going to react in the same manner, beseeching the AI to verify having heard the sound, along with aiming to get some explanation from the AI about it.

This raises the important question about how the AI should interact with humans about the sounds or noises that were heard.

Do you want an AI that offers soothing words?

Do you want an AI that tells it like it is, though the indication might be perceived as brash or harsh?

A human driver would likely alter their response to a human passenger depending upon a multitude of factors, such as their own perceived seriousness about the noise, and the nature of the passenger and how they might cope with such a discussion.

One potential advantage of having the AI detecting for car sounds and noises would be that it might be able to do an extensive analysis on the car sound.

A human driver of a ridesharing service might not know much about cars. For them, any sounds are just sounds, and they might not have a clue as to what a particular ping or pop signifies. The AI could look-up the sounds in a database and potentially have a better means of discerning what the sound portends.

We could carry this capability to a more advanced level for the AI.

Suppose the AI system had a knowledge-based component that could do an in-depth diagnosis to a level similar to a car mechanic hearing a noise. This more advanced capability could aid the self-driving car in diagnosing the potential problem and would also aid the ongoing driving of the self-driving car by the AI.

If the sound occurs whenever the AI takes a sharp left turn, the AI action planning portion might opt to be especially careful when making left turns. This could avoid for the moment further harming whatever mechanical issue has occurred.

Via OTA (Over-the-Air) electronic communications, which the AI usually uses to pump data up to the cloud of the automaker or tech firm, and gets patches and updates pushed down to the on-board AI system, the AI could share the sounds or noises into the cloud. This might be handy as there might be other similar noises or sounds that have been heard in other AI self-driving cars of this particular fleet.

AI Planning Action Steps

In terms of the types of actions that the AI might take, doing so after detecting and diagnosing a car sound or noise, these are potential AI planning action steps:

  • Stop use of the self-driving car “immediately” and pull over safely
  • Stop use of the self-driving car as soon as practical and seek to get to a repair shop
  • Adjust the driving of the self-driving car based on the potential issue
  • Continue normal use of the self-driving car and note to have the matter checked
  • Consider the sound or noise irrelevant and take no other action due to it
  • Etc.

Another potential recourse would be for the AI self-driving car to institute an electronic dialogue with other nearby AI self-driving cars.

Using V2V (vehicle-to-vehicle) electronic communications, the AI self-driving car that has detected an untoward sound might request that other nearby AI self-driving cars take a look at the AI self-driving car to see if they detect anything untoward. For example, if the tailpipe is dragging on the street, an AI self-driving car that is behind this AI self-driving car could detect the dragging tailpipe via the use of its cameras.

The V2V dialogue might also be used to warn other nearby AI self-driving cars that the AI self-driving car initiating the dialogue might soon be exhibiting driving issues. Perhaps the self-driving car is going to start weaving, due to a mechanical breakdown, and the AI will be doing whatever it can to minimize the weaving.

Besides using a knowledge-based system and a database to try and diagnose the car sounds, another approach would be via the use of Machine Learning (ML) and Deep Learning (DL).

If an entire fleet of AI self-driving cars are reporting various sounds and noises, along with the later-on determination by car mechanics of what it signified, this kind of data could be used for pattern matching.

Using a large-scale or “deep” artificial neural network, we could feed this data into the ML or DL and see if it can train on the data. By doing so, whenever an AI self-driving car perchance encounters such a sound or noise, the ML or DL could be used to more definitively diagnose it. This could either be an on-board ML or DL that is placed inside the AI self-driving car or might be a cloud-based version and the AI self-driving car would access it via OTA or the equivalent.


Bang. Hiss. Ping. Pop. Clunk. Clack. Buzz. Rumble. Screech.

If your car is making these sounds, and if it isn’t because you are playing some music or having a wild party inside the car, you ought to consider what your car is trying to secretly tell you. Might be a loose fan belt. Maybe your brakes are wearing out. The muffler on your car might be busted. Your bumper might be dragging on the ground.

AI self-driving cars will have lots of sensors that are pointing outward to detect what is happening around the AI self-driving car. I’ve repeatedly called for AI self-driving cars to be outfitted with more sensors for purposes of AI self-awareness. By this, I mean that the AI self-driving car needs to know its own internal status.

One important way to determine the status of the AI self-driving car will be via the sounds and noises that the car makes. Humans use those sounds and find them useful as a premonition or forewarning that something might soon go horribly awry. We ought to expect the AI to do the same.

With the AI doing sound detection, there is a good chance of boosting how these sounds will be dealt with. The AI self-driving car can be tracking how often the sound occurs, along with the circumstances such as going uphill or downhill, making turns or going straight ahead, speeding up or slowing down, and so on. With an advanced knowledge-based system, and perhaps Machine Learning or Deep Learning, when also combined with OTA, V2V, and V2I, we can substantially boost the diagnostic capabilities for AI self-driving cars.

It would also be hoped that this will further translate into safer driving by the AI system since it will be using essential clues in determining how best to safety drive the self-driving car.

That’s music to my ears, or, I might say are the sounds that I want to hear.

For free podcast of this story, visit:

The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.

More info about AI self-driving cars, see:

To follow Lance Eliot on Twitter:

For his blog, see:

For his Medium blog, see:

For Dr. Eliot’s books, see:

Copyright © 2019 Dr. Lance B. Eliot

Written by

Dr. Lance B. Eliot is a renowned global expert on AI, Stanford Fellow at Stanford University, was a professor at USC, headed an AI Lab, top exec at a major VC.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store