Self-Driving Cars Get Confused Around Animal Drawn Vehicles

Dr. Lance Eliot, AI Insider

Image for post
Image for post

[Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column:]

If you’ve ever been to New York City (NYC), you’ve likely seen the horse drawn carriages near Central Park.

Tourists delight in going for a ride in the carriages and having a romantic trip.

There is an ongoing controversy about the horse drawn carriages and there are some that believe it is inhumane or improper to have the horses do this task, especially in the NYC environment.

According to published reports and a catalog of incidents kept by PETA, earlier this year there was a case of a horse and carriage that took off into the streets of NYC without the carriage driver, and the horse ran into two parked cars, injuring the horse and the cars.

Even worse was an incident in which a horse was being raced to the front of the hack line by a carriage driver (the hack line is where the carriages wait to find someone willing to pay for a ride, like waiting for a cab), and the entire carriage overturned and injured the horse.

A few years ago, a horse named Goldie broke free of the harness and galloped for eleven blocks in the streets of NYC. I know that New Yorkers are used to seeing unusual and at times strange antics on their streets, though I’d bet that a horse galloping on its own would be something that even the hardened New Yorkers would stop for a moment to see.

Amish And Workhorses

There are some people that rely upon a horse and buggy as their primary form of transportation.

In Lancaster County, the Amish routinely use a multitude of horse and buggy wagons to get around. There is a false assumption that the Amish are prevented by their religious beliefs in riding in a car, but this is not the case. They aren’t supposed to own a car. They can ride in a car, if the situation arises. Generally, they use a horse and buggy to get around in their locale. This is sufficient for their living purposes.

Those of us in the United States might not realize there are many places in the world that still rely upon horses, mules, donkeys, and other draft animals as a crucial part of their living.

Other Draft Animals

I’d perhaps be remiss if I left out camels.

They too are a beast of burden.

They are used in places in the world that can have them come in contact with cars by having the camels walk in car traffic, frequently so.

Dogs can be beasts of burden too.

Think about dog sleds.

A dog sled can be driven onto roads that also have car traffic.

One concern about having these animals in car traffic situations involves the rather obvious point that the animal can get injured or killed by cars.

Likewise, such an animal could ram into or run over by a car, potentially injuring occupants in the car. We need to also consider the health hazard overall to a draft animal that when in traffic might be breathing in the harmful exhaust and fumes of the cars. This can add-up over time, the more that the draft animal goes into traffic.

It can be tricky and frightening to a draft animal to be among moving cars.

This causes emotional stress for the animal.

AI Autonomous Cars And Draft Animals

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One aspect involves the AI system being able to cope with driving when nearby animal drawn vehicles.

This is generally considered an edge or corner case, rightfully so due to its rarity in the everyday city or suburb driving situations, but nonetheless it is a legal requirement that human drivers must be aware of and obey the rules and in our view so should an AI self-driving car.

You can’t just wave your hands as an AI developer and complain that encountering an animal drawn vehicle is an obscure use case.

Obscure or not, the AI had better drive properly, otherwise the animal(s) might be endangered, any human riders or drivers of an animal drawn carriage might be endangered by a car, and even the car itself and its occupants could be endangered (plus nearby pedestrians or other humans that could be involved in a potential car crash or incident sparked by the interaction of the car and the draft animal).

Are Draft Animals A Special Case Or Not

Returning to the matter of driving a car when nearby to animal drawn vehicles, let’s consider the kinds of driving tactics and strategies that an AI system for a self-driving car should be imbued with in order to safely deal with such situations.

I’m sure there are some AI developers that would immediately claim that the AI does not need any special capability for driving when nearby animal drawn vehicles.

They would contend that if an AI system can navigate and drive a car on everyday streets, it would presumably be able to drive when nearby animal drawn vehicles. Indeed, they would assert that there is nothing unusual or special to be done, and the AI can consider an animal drawn vehicle to be no different than any other moving vehicle on the road.


This kind of thinking is going to get AI self-driving cars into some bad predicaments that could have otherwise been more safely handled.

If we just play the game that an animal drawn vehicle is no different than any other moving vehicle, the odds are that eventually and inevitably something will go awry. For those pundits that are strong advocates of AI self-driving cars, I assure you that the day that an AI self-driving car gets entangled in an adverse situation with an animal driven vehicle, and especially if there is any injury to humans or animals, it will become a rallying cry for those that say AI self-driving cars are not ready for being on our roads.

An AI developer that shrugs off the possibility is missing the bigger picture.

An AI self-driving car pundit that is vocally supportive of AI self-driving cars will find that even one such adverse incident can undermine months or possibly even years of potential public-trust that might have been built up for AI self-driving cars. AI self-driving car maims horse. AI self-driving car crashes into horse drawn carriage. Horses scared by AI self-driving car and sprint away, fearing for their lives. These are all headlines that can catch like wildfire and then damage the image of AI self-driving cars, which will be very hard to undo or overcome.

AI Coping With Nearby Draft Animals

Let’s consider what the AI should be doing regarding animal drawn vehicles.

First, the detection of an animal drawn vehicle is key to being able to undertake any potential actions about it. The sensors of the AI self-driving car need to try and ascertain whether an animal drawn vehicle is nearby. This might be harder to do than you think.

For humans, we readily can look around the surroundings of a car and be able to discern that there is say a horse over there and it is pulling a carriage. Easy to do. For most AI systems, this is a much harder task. Generally, the AI will make use of cameras to visually examine the surroundings of the car. Pictures or video are then analyzed by software routines that are often trained to find everyday objects in a street scene, such as other cars, buses, bicyclists, pedestrians, and the like.

Using Machine Learning (ML) and Deep Learning (DL), the visual images are assessed and interpreted by Artificial Neural Networks (ANN’s or sometimes referred to as simply NN’s).

These ML/DL elements were likely trained via thousands upon thousands of images of everyday street scenes, of which, there were visual aspects or clues that could lend toward identifying what is in the image. As a result of this training, the AI is supposed to be able to discern what kinds of objects are nearby and also associate those objects with potential behaviors.

One issue about these ANN’s is that the training set of data needs to encompass a wide enough range of images that the training will provide sufficient examples for dealing with the real world. Suppose we fed thousands of street scene images into a ML/DL and purposely did not include scenes that had light posts and nor fire hydrants. The ANN doesn’t somehow magically realize that the images don’t contain something. Instead, it trains based on what is presented to it.

Without any training on the recognizing of light posts and fire hydrants, this means that in the real-world the AI self-driving car is not going to be informed when a light post or fire hydrant is nearby. I’d wager that most of us notice fire hydrants when we are trying to park our cars, being wary of parking near to one for concern of getting a ticket. Imagine if you did not even know what a fire hydrant looked like, which therefore you would likely park next to one, innocently, unknowingly, since you would just assume the thing sitting there was some kind of inconsequential object, a blob as it were.

I mention all of this because the same notion can be applied to an animal drawn vehicle. If the image processing portion of the AI system has not be explicitly trained or programmed to recognize animal drawn vehicles, there will be no specific means for the AI to recognize that one is nearby. Instead, there will be an unknown blob. The AI will at least potentially realize that something is there, and in addition to the visual images there will likely be radar, ultrasonic, and LIDAR that will detect the presence of the thing.

Unfortunately, detecting the presence of something is not enough, since you also would be better off knowing what the thing actually is.

Categorized as a blob, the AI really cannot do much in terms of anticipating what the blob might do.

Is it stationary and unlikely to move?

If a horse drawn carriage is sitting at the curb and waiting to take on passengers, it is momentarily still, but soon enough it will start to move into traffic. Not knowing that the unknown object is a horse drawn carriage, the AI might assume that the stationary object is always unmoving and perhaps permanently stationed in the spot that it resides.

Domestic Versus International Readiness

Detection then is crucial for the AI to be able to deal with animal drawn vehicles. There must be ML/DL that was undertaken specifically to include animal drawn vehicles. The nature and types of animal drawn vehicles would need to have been varied enough to allow for the real-world variety of circumstances that the AI self-driving car might itself in.

This brings up another aspect about AI self-driving cars, namely their being internationally ready.

Much of the work on AI self-driving car is taking place currently in the United States. There is an inherent bias on the part of the AI developers to concentrate on the kinds of roads and roadway situations encountered in the United States. That’s fine and makes sense for aiming at AI self-driving cars that will work well in the United States, but it also provides the potential downside that the AI self-driving cars won’t do well in other countries, being unable to contend with the international differences.

If you are an auto maker or tech firm and mainly care about the U.S. market, you are doing just fine to have a U.S.-only mindset. On the other hand, if you want to ultimately have your AI self-driving cars be able to work properly in Europe, and in Asia, and in all other parts of the world, you need to be considering how to establish the groundwork in your AI system to be able to deal with those other locales. If you’ve ever developed software for a global market, you know that you need to build into your software a multilingual capability and also that it needs to be culturally appropriate to the cultures that will use the software.

In the case of an AI self-driving car, the nature of the differences between the U.S. and other countries can be quite dramatic in terms of the driving task and the driving scenes. One such difference is going to be the animal drawn vehicles aspects. As mentioned earlier, it is likely rare in the U.S. for an AI self-driving car to come upon animal drawn vehicles, while in certain other countries it would be considered commonplace.

Humans And Self-Driving Car Interaction

Besides the AI self-driving car being able to detect directly whether or not an animal drawn vehicle is nearby, the AI might also be tipped to the possibility by a passenger in the AI self-driving car.

People that are riding in an AI self-driving car are likely going to want to carry on a dialogue with the AI system. Some AI developers only believe that the passengers will give a direction such as take me to the ballgame, and no other conversation will occur. This is narrow thinking and the reality is that people will want to interact with the AI, in the same fashion that they might interact with a human cab driver or a chauffeur.

As such, a human occupant might try to tell the AI that there is a horse drawn carriage over to the right of the road and the carriage waiting to proceed into traffic.

Perhaps the AI self-driving car is in New York City.

The human inside the self-driving car is concerned that the AI of the self-driving might not have detected the horse carriage and be worried that the AI is not going to take appropriate precautions. The concerned passenger of the self-driving car might then bring up the matter to alert the AI. Hopefully, the AI would have already detected the presence of the horse and carriage and reassure the human occupant that the AI realizes the matter and will be taking appropriate precautions.

Another means for the AI to potentially be aware of the presence of the animal drawn vehicle might be due to V2V (vehicle-to-vehicle) electronic communications. Suppose that another AI self-driving car is a block ahead of your AI self-driving car. The AI of that self-driving car detects the presence of a horse drawn carriage. The AI might then broadcast out via V2V to other nearby AI self-driving cars that there is an animal drawn vehicle on the street and thus forewarn those other AI self-driving cars.

This same kind of electronic heads-up could come via V2I (vehicle-to-infrastructure) and possibly via V2P (vehicle-to-pedestrian) communications.

Suppose that NYC decides to put up computers at various street corners to be able to aid in monitoring traffic, and perhaps one aspect involves those devices being aware of horse drawn carriages. Imagine that the traffic signals are controlled by those computing devices and purposely go quickly to a green light whenever a horse drawn carriage approaches the intersection. This might be a safety technique implemented to reduce the amount of time that the horse drawn carriages have to sit idle while waiting for a green light (I suppose tourists riding in the carriage would like the green lights too).

The detection of the animal drawn vehicle is the first part of the driving task for the AI system. The next step involves updating the virtual world model being used by the AI system. This is an internal kind of model that indicates where various objects are, as related to the position of the self-driving car, and also indicates the movement and direction of those objects. You might think of this as a kind of air traffic control capability, trying to monitor traffic and where it is and where it might be headed.

Animals Can Be Wild

The really difficult part comes once the AI action planning component analyzes the virtual world model and determine what kinds of next driving actions are appropriate.

Animals can do wild things.

Of course, yes, I realize that humans can do wild things too. A human driving a car can wildly opt to veer across lanes or go head-on towards other cars. It happens. Generally, we live in a world whereby our usual expectation is that other human drivers are going to do rational things and drive relatively safely. If that were not the case, I assure you that we would have many more car accidents and traffic related deaths than we do today. It is a kind of miracle that each day we have millions upon millions of cars being driven on our roadways and we are not inundated in nonstop chaos and destruction. A miracle, I say!

Anyway, without offending any animal lovers, I hope we can agree that there is a chance that animals can do wild things too. The AI needs to be anticipating what the animal drawn vehicle might be doing and where and how it will be going next.

The animal drawn vehicle is most likely being driven by a human. In theory, the human driver is directing the animal. The animal is merely pulling the vehicle and not deciding where to go, nor deciding when to stop, when to turn, etc.

Consider that we might have a situation whereby the carriage driver is fighting with the animal drawing the carriage. It might be a horse that refuses to proceed ahead. Maybe the horse is reluctant to make a turn at the corner. Perhaps the horse has come to a halt in the middle of the street, either because it fears getting hit by a car or maybe just because it wants to come to a stop.

The point being that an animal might overtake the efforts of the driver of the animal drawn vehicle.

The AI cannot assume that the human driver of the animal drawn vehicle will do the right thing in terms of directing the vehicle, and also that the human driver might want to do the right thing but be prevented or inhibited by the animal drawing the vehicle.

There is also the case of the animal drawn vehicle that has no human driver involved. This is rare and presumably only in cases wherein the human driver has perhaps failed to get into the carriage and let it loose, or maybe fallen out of the carriage, or fainted while in the carriage, etc.

The AI needs to be wary of the animal drawn carriage. It would be safest to provide as much leeway as possible, having the AI action plan targeting a sufficiently wide berth to the animal drawn carriage. Predicting the path of where the animal drawn carriage might go can be tricky. There should be multiple potential paths plotted, under the assumption that the animal might opt to go in some otherwise unsuspecting direction or the carriage driver might do so.

The AI should try to avoid any sudden moves of the car.

A quick acceleration or a sudden change of lanes can be alarming to the animal (and likely to the driver of the animal drawn vehicle).

Using the horn of the self-driving car would be generally ill-advised when around an animal drawn carriage.


For most of us, an animal drawn vehicle is something quaint. We see them at tourist areas such as in NYC near Central Park. We might see them at parades. They might be seen at farms or ranches. In some parts of the world, animal drawn vehicles are a customary practice. They are continually in and around car traffic. That’s a fact of life in those areas.

An AI self-driving car needs to be prepared to cope with situations involving animal drawn vehicles. Treating an animal drawn vehicle as merely some kind of large-size unidentifiable blob that seems to be on the roadway is insufficient. The odds are that there will be ugly and untoward entanglements between naive and ill-prepared AI self-driving cars and the real-world animal drawn vehicles that might be encountered. We’d all prefer savvy AI self-driving cars that are ready and able to contend with animal driven vehicles.

The horses, donkeys, mules, dogs, camels, and other beasts of burden will be thankful that the AI developers considered how to best assist and be compatible with those working animals.

Let’s all work on that!

For free podcast of this story, visit:

The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.

More info about AI self-driving cars, see:

To follow Lance Eliot on Twitter:

For his blog, see:

For his AI Trends blog, see:

For his Medium blog, see:

For Dr. Eliot’s books, see:

Copyright © 2019 Dr. Lance B. Eliot

Written by

Dr. Lance B. Eliot is a renowned global expert on AI, Stanford Fellow at Stanford University, was a professor at USC, headed an AI Lab, top exec at a major VC.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store