AI Self-Driving Cars Still Grappling With Jaywalkers

Dr. Lance Eliot, AI Insider

Image for post
Image for post

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column:]

Being from California, I remember one of the first times that I visited New York City (NYC) and made the mistake of renting a car to get around the famous metropolis. I had figured that driving a car around the avenues and streets would give me a good sense of how the city that never sleeps was laid out and where the most notable restaurants, bars, and shops could be found.

Turns out that I mainly discovered how much New Yorkers seemed to delight in jaywalking.

It was as though there weren’t any rules against jaywalking.

I also found out about the techniques involved in making a devoted stare or gaze that appeared to be a local custom. This might be the same kind of thing you’d do when you encounter a wild animal in the woods. Given them a strong stare might say that you are mighty and the animal should not try to take you on.

Some of the car drivers that were locals or that were used to the local customs would often give a stern stare back. On a few occasions, it would get really testy and the jaywalker would wave an arm and act as though they might try to slay the dragon of a car coming down the street.

I admit that after I turned in the rental car and became more of a traditional pedestrian on my visits to NYC, I adopted the jaywalking habit.

Locals often insisted that it was actually more dangerous to cross at a marked crosswalk, at least in NYC, than it was to jaywalk.

I doubt that there are any actual statistics to back the claim.

Where I grew up in California, jaywalking was generally frowned upon and only undertaken as some kind of last resort.

Styles of Jaywalking

I noticed for example in NYC that the time of day seemed to make a difference in terms of the volume and nature of the jaywalking.

Mornings, when pedestrians were trying to get to work, often stoked a lot of jaywalking, perhaps to try and get to work promptly and minimize the time required to get to the office.

There was also the amount of traffic that played a role in the jaywalking. If the traffic on a given street was completely backed-up and stuck, jaywalkers would in droves weave in and around the cars, doing so without a care in the world since they perceived that the wild animals (the cars and car drivers) were jammed in place and couldn’t do much to run them over. As soon as a green light allowed the traffic to flow, the jaywalkers became more cautious and realized it was now “game on” in terms of trying to time when to best engage in jaywalking.

The weather also played a part in the jaywalking ritual.

Rainy days meant that the jaywalkers had an even greater incentive to jaywalk. Why waste time and get wet in the rain, when you can scoot across a street and do so quickly enough that perhaps rain drops themselves won’t touch you.

Driver Attention And Jaywalkers

In my estimation, a sizable chunk of drivers were not attuned to the presence of the jaywalkers.

This made sense since the drivers were having to contend with bigger game, such as large trucks trying to make deliveries and rapidly exiting and entering unexpectedly into the street and avenues. There were other crazed car drivers jockeying for position. There were often obstacles on the roadway such as a pallet of liquor bottles being delivered to a liquor store.

Another factor involved sizing up the jaywalker.

How was the jaywalker dressed and what kind of look did they have?

If a driver saw a jaywalker that seemed like a seasoned New Yorker, it suggested that the jaywalker could take care of themselves and no further driver attention was needed. If the jaywalker looked like a wide-eyed tourist, well, this might present a problem because the “amateur” jaywalker might foul things up. The “professional” jaywalkers knew how to assiduously cross a street. Those out-of-town jaywalkers were bound to mess-up the delicate dance of true jaywalkers and NYC drivers.

Herd Mentality Of Jaywalking

In most cases, I observed individuals acting as jaywalkers.

This though was not always the case and there were frequently situations of multiple jaywalkers proceeding all at once.

There was at times a herd mentality. If one of the jaywalkers went for it, the others were sure to follow. Now, this actually often made sense, since the first one likely found an opening to jaywalk and the others also perceived the same opening.

There were some “leaders” of the pack that weren’t thinking at all about the rest of the herd. Therefore, they were not trying to find a big enough opening to get a dozen people across the street all at once. They were focusing on just themselves.

In that case, sometimes the first mover made it across, but the others did not, and they had somehow assumed that if the first mover could do it, the rest of them could.

This reminds me too of a common refrain that my New York colleagues would use on me.

They would say that any jaywalker is making their own decisions and if they get hit, well, that’s their own doing. Why should the government tell them what they can and cannot do. It’s up to the individual to choose to jaywalk or not, and it is on the head of that jaywalker as to whether they risk their life and limb or not.

I don’t buy into this claim.

Here’s another angle for you.

What about the children?

Children Learning About Jaywalking

I’ve seen jaywalkers holding the hand of a child or a group of children and trying to make a jaywalking attempt with them.

If they do jaywalking with an adult, it is a slippery slope that can readily assume they can do jaywalking on their own. Indeed, some children will happily go jaywalking to showcase to their parent that they are now sentient and their own agent and no longer need to have an adult aid them in the jaywalking.

It is a kind of rites of passage.

The counter argument from some adults is that if they don’t show the child how to “properly” jaywalk, the odds are that the child is going to do jaywalking anyway at some point, and without having done so with a “responsible” adult, the child is going to be more prone to getting hurt when trying to go jaywalking based on no prior instruction. Some would say that having a head-in-the-sand viewpoint of being a parent that pretends jaywalking will never happen, will merely make the child more vulnerable than if you instead do a parent-child jaywalking effort with the child.

Right Of Way As Jaywalking Rules

Who has the proper right of way?

In some countries, both the jaywalker and the driver are considered equals in terms of right-of-way.

Here’s what the Department of Motor Vehicles (DMV) rulebook in California states about the act of jaywalking:

“(a) Every pedestrian upon a roadway at any point other than within a marked crosswalk or within an unmarked crosswalk at an intersection shall yield the right-of-way to all vehicles upon the roadway so near as to constitute an immediate hazard.

(b) The provisions of this section shall not relieve the driver of a vehicle from the duty to exercise due care for the safety of any pedestrian upon a roadway.”

AI Autonomous Cars And Jaywalkers

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One crucial aspect involves the AI being able to contend with jaywalkers.

There are some that say there is no need for an AI self-driving car to do anything at all about a jaywalker.

Jaywalkers are acting illegally.

They get whatever they deserve.

There is no requirement that the AI of the self-driving car needs to do anything at all about a jaywalker.

It might seem astonishing to you that someone would think this way. It is a phenomenon that I refer to as the “egocentric” developer viewpoint. The world needs to conform to their view of the world, rather than the developer facing the reality of the real-world. I quickly point out to such a person that the DMV code clearly states that the car driver must exercise a duty of care, even if the jaywalker is doing something utterly wrong and illegal.

This often surprises the AI developer. They had laid all the responsibility onto the shoulders of the wayward pedestrian. In one sense, this is similar to the jaywalkers that insist they are doing a “victimless” act and that it is up to the jaywalker to choose whether to personally risk going jaywalking or not. Only after I point out the other “victims” that can get dragged into the “victimless” effort do they (hopefully) see the larger picture.

Needing To Contend With Jaywalkers

Let’s all assume that indeed the AI of the self-driving car does need to contend with jaywalkers.

It cannot ignore them.

It cannot pretend that the burden of safety is solely on the backs of the jaywalkers.

The AI must have provisions for dealing with jaywalkers.

I’d say that’s a prudent and societally expected assumption about AI self-driving cars.

This moves us then into the next kind of quirk that some AI developers offer.

There are some AI developers that will concede the notion of doing something about jaywalkers, but then argue that a jaywalker is nothing special and that the “normal” driving aspects of an AI self-driving car should suffice when dealing with jaywalkers.

In this case, the AI developer is suggesting that if the AI self-driving car is already prepared to cope with objects that might appear in the roadway, the job of having the AI be prepared for jaywalkers is already completed. No need to do anything else.

This implies that a jaywalker is no different from say a tumbleweed. If the AI is able to detect a tumbleweed in the roadway, it amounts to the same thing as detecting a human in the roadway. At least that’s the kind of thinking involved by this kind of AI developer.

Here’s an easy question for you, I think, namely do you consider it viable to ram a jaywalker, using the same logic about ramming a tumbleweed?

Thus, I claim that if an AI system is only detecting “objects” and not trying to also figure out what kind of object is involved, it is insufficient in terms of what we would all hope a true AI self-driving car is going to be able to do.

From a systems perspective, please realize that I realize that when the cameras, radar, LIDAR, and other sensors first do their detection, they are only indeed detecting “objects” and thus there is a crucial role of object detection involved. What I am saying is that after the raw sensory detection of an object, it is imperative that the AI system tries to discern what kind of object the object is, such as whether it is a tumbleweed or a human.

Overreacting To Jaywalkers

Auto makers and tech firms that are making AI self-driving cars are often dealing with just getting an AI self-driving car to deal with the rudiments of driving, and they would say that the best bet is to have the AI always assume the worst-case scenario. This means that a tumbleweed that might be a human is going to be assumed to be a human, which is considered a safer bet than not making that kind of assumption.

They would also tend toward having the AI take the “super-cautious” approach. I remember being invited to watch an AI self-driving car as it drove down an empty street, and the automaker and tech firm had a stuntman walk out into the street, acting like a jaywalker. The AI was able to detect the jaywalker and came to a nearly immediate halt. Success!

Well, not exactly.

The AI self-driving car came to a halt at about one quarter into the block, and the jaywalker was at the other quarter’s end of the block.

Sure, the AI self-driving car detected the jaywalker at a sizable distance and came to a prompt halt at a sizable distance. It was maybe just somewhat less than a half a football field away from the human when it halted.

Great, no chance of hitting that person.

Does this make sense in the real-world?

There are already reports of people opting to “prank” today’s AI self-driving cars. If you know that an AI self-driving car will stop or maybe turn when you take some kind of action as a pedestrian or maybe when driving in your own car, it is human nature that we would all likely exploit these behaviors of the AI self-driving cars. Want to get ahead and not be behind one of those slower moving AI self-driving cars, easy enough to arrange by tricking the AI self-driving car into slowing down or halting.

Use of Machine Learning And Deep Learning

A true AI self-driving car has to be embodied with the kinds of driving skills that humans use, and particularly so with regard to contending with jaywalkers.

Some are hoping that the use of Machine Learning (ML) and Deep Learning (DL) will come to the aid of trying to cope with jaywalkers.

In one sense, yes, it can be helpful to use ML and DL, and by collecting large sets of jaywalking circumstances begin to find patterns to suggest how jaywalkers behave, and therefore then have ready-made solutions in-hand by the AI.

I assure you though that today’s kind of ML and DL is not going to be the silver bullet or magic wand that provides the jaywalking kind of driving aptitude needed for a true AI self-driving car. The jaywalker aspects are far too complex. It is not the same as merely analyzing an image to ferret out whether there is a human in the scene or not. This has to do with behaviors and complex ones.


Most seasoned drivers tend to take jaywalkers in stride (pun!), meaning that we human drivers can detect jaywalkers, we can anticipate what they might do, we can adjust our driving aspects accordingly, and most of the time the dance leads to the jaywalker getting safely across the street and the car safety proceeding down the street.

This is nearly an effortless act by a seasoned human driver.

AI self-driving cars are not yet as prepared for handling jaywalkers.

The sad incident of the jaywalker in Arizona being run down by an Uber self-driving car is but one example of how limited today’s AI self-driving cars are in terms of coping with jaywalkers. We need to focus greater attention on the AI capability to specifically deal with jaywalkers and not allow the assumed everyday capabilities of the AI to be able to contend properly with jaywalking.

Why did the jaywalker cross the road?

Answer: To safely get to the other side.

In any case, let’s hope that AI developers keep attune to advancing self-driving cars toward ensuring that jaywalkers are well-detected and avoided, which is a crucial matter and not merely an “edge” problem by any means.

For free podcast of this story, visit:

The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.

More info about AI self-driving cars, see:

To follow Lance Eliot on Twitter:

For his blog, see:

For his AI Trends blog, see:

For his Medium blog, see:

For Dr. Eliot’s books, see:

Copyright © 2020 Dr. Lance B. Eliot

Written by

Dr. Lance B. Eliot is a renowned global expert on AI, Stanford Fellow at Stanford University, was a professor at USC, headed an AI Lab, top exec at a major VC.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store