Don’t Starve The AI: System Load Balancing Crucial For Driverless Cars
Dr. Lance Eliot, AI Insider
I recall an occasion when my children had decided to cook a meal in our kitchen and went whole hog into the matter (so to speak). One aspect that caught my attention was the use of our stove top. The stove top has four burner positions. On an everyday cooking process, I believe that four heating positions is sufficient. I could see that with the extravagant dinner that was being put together, the fact that there were only four available was a constraint. Indeed, seemingly a quite difficult constraint.
During the cooking process, there were quite a number of pots and pans containing food that needed to be heated-up. I’d wager that at one point there were at least a dozen of such pots and pans in the midst of containing food and requiring some amount of heating. Towards the start of the cooking, it was somewhat manageable because they only were using three of the available heating spots. By using just three, it allowed them to then allocate one spot, the fourth one, as an “extra” for round robin needs. For this fourth spot, they were using it to do quick warm-ups and meanwhile the other three spots were for truly doing a thorough cooking job that required a substantive amount of dedicated cooking time.
Pots and pans were sliding on and off that fourth spot like a hockey puck on ice. The other three spots had large pots that were gradually each coming to a bubbling and high-heat condition. When one of the three pots had cooked well enough, the enterprising cooks took it off the burner almost immediately and placed it onto a countertop waiting area they had established for super-heated pots and pans that could simmer for a bit.
As a computer scientist at heart, I was delighted to see them performing a delicate dance of system load balancing.
System Load Balancing Is Unheralded But Crucial
You’ve probably had situations involving multiple processors or maybe multiple web sites wherein you had to do a load balance across them. This is usually due to having a limited number of resources and wanting to try and ensure that they are able to be used effectively and efficiently.
Let’s consider again the meal cooking example.
If the meal had been one involving much less preparation, such as if they had only three items to be cooked, they would have readily been able to use the stove top without any of the shenanigans of having to float around the pots and pans. They could have just put on the three pots and then waited until the food was cooked. But, since they had more needs for cooking then just the available heating spots, they needed to devise a means to make use of the constrained resources in a manner that would still allow for the cooking process to proceed properly.
This is what load balancing is all about.
There are situations wherein there are a limited available supply of resources, and the number of requests to utilize those resources might exceed the supply. The load balancer is a means or technique or algorithm or automation that can try to balance out the load.
Another valuable aspect of a load balancer is that it can try to even out the workload, which might help in various other ways. Suppose that one of the stove tops was known to sometimes get a bit cantankerous when it is on high-heat for a long time. One approach of a load balance might be to try and keep that resource from peaking and so purposely adjust to use some other resource for a while.
We can also consider the aspect of resiliency.
You might have a situation wherein one of the resources might unexpectedly go bad or otherwise not be usable. Suppose that one of the burners broke down during the cooking process. A load balance would try to ascertain that a resource is no longer functioning, and then see if it might possible to shift the request or consumption over to another resource instead.
Load Balancing Difficulties And Challenges
Being a load balancer can be a tricky task.
Suppose the kids had decided that they would keep one of stove top burners in reserve and not use it unless it was absolutely necessary. In that case, they might have opted to use the three other burners in a manner of allocating two for the deep heating and one for the warming up. All during this time, the other fourth burner would remain unused, being held in reserve. Is that a good idea?
It depends. I’d bet that the cooking with just the three burners would have stretched out the time required to cook the dinner. I can imagine that someone waiting to eat the dinner might become disturbed if they saw that there was a fourth burner that could be used for cooking, and yet it was not, and the implication being that the hungry person had to wait longer to eat the dinner. This person might go ballistic that a resource sat unused for that entire time. What a waste of a resource, it would seem to that person.
These are the kinds of considerations that go into establishing an appropriate load balancer. You need to try and decide what the rules are for the load balancer. Different circumstances will dictate different aspects of how you want the load balancer to do its thing. Furthermore, you might not just setup the load balancer entirely in-advance, such that it is acting in a static manner during the load balancing, but instead might have the load balancer figuring out what action to take dynamically, in real-time.
When using load balancing for resiliency or redundancy purposes, there is a standard nomenclature of referring to the number of resources as N, and then appending a plus sign along with an integer value that ranges from 0 to some number M. If I say that my system is setup as N+0, I’m saying that there are zero or no redundancy devices. If I say it is N+1, then that implies there is 1 and only 1 such redundancy device. And so on.
You might be thinking that I should always have a plentiful set of redundancy devices, since that would seem the safest bet. But, there’s a cost associated with the redundancy. Why was my stove top limited to just four burners? Because I wasn’t willing to shell out the bigger bucks to get the model that had eight. I had assumed that for my cooking needs, the four sized stove was sufficient, and actually ample.
For computer systems, the same kind of consideration needs to come to play.
In computing, you can consider load balancing for just about anything. It might be the CPU processors that underlie your system. It could be the GPUs. It could be the servers. You can load balance on an actual hardware basis, and you can also do load balancing on a virtualized system. The target resource is often referred to as an endpoint, or perhaps a replica, or a device, or some other such wording.
Those in computing that don’t explicitly consider the matter of load balancing are either unaware of the significance of it or are unsure of what it can achieve.
For many AI software developers, they figure that it’s really a hardware issue or maybe an operating system issue, and thus they don’t put much of their own attention toward the topic. Instead, they hope or assume that those OS specialists or hardware experts have done whatever is required to figure out any needed load balancing.
AI Autonomous Cars And Load Balancing The On-Board Systems
What does this have to do with AI self-driving driverless autonomous cars?
At the Cybernetic AI Self-Driving Car Institute, we are developing AI systems for self-driving cars. One key aspect of an AI system for a self-driving car is its ability to perform responsively in real-time.
On-board of the self-driving car you have numerous processors that are intended to run the AI software. This can also include various GPUs and other specialized devices.
You’ve got software that needs to run in real-time and direct the activities of a car. The car will at times be in motion. There will be circumstances wherein the AI is relatively at ease and there’s not much happening, and there will be situations whereby the AI is having to work at a rip-roaring pace. Imagine going on a freeway at 75 miles per hour, and there’s lots of other nearby traffic, along with foul weather, the road itself has potholes, there’s debris on the roadway, and so on. A lot of things, all happening at once.
The AI holds in its automation the key to whether the self-driving car safely navigates and avoids getting into a car accident. This is not just a real-time system, it is a real-time system that can spell life and death.
To achieve this, the on-board hardware generally has lots of computing power and lots of redundancy.
Is it enough? That’s the zillion dollar question. Similar to my choice of a four burner stove, when the automotive engineers for the auto maker or tech firm decide to outfit the self-driving car with whatever number and type of processors and other such devices, they are making some hard choices about what the performance capability of that self-driving car will be. If the AI cannot run fast enough to make sound choices, it’s a bad situation all around.
Dangers Of Silos Among Autonomous Car Components
Some auto makers and tech firms find themselves confronting the classic silo mentality of the software side and the hardware side of their development groups. The software side developing the AI is not so concerned about the details of the hardware and just expect that their AI will run in proper time. The hardware side puts in place as much computing power as it seems can be suitably provided, depending on cost considerations, physical space considerations, etc.
If there is little or no load balancing that comes to play, in terms of making sure that both the software and hardware teams come together on how to load balance, it’s a recipe for disaster.
Some might say that all they need to know is how much raw speed is needed, whether it is MIPS (millions of instructions per second), FLOPS (floating point operations per second), TPU’s (tensor processing units), or other such metrics. This though doesn’t fully answer the performance question. The AI software side often doesn’t really know what kind of performance resources they’ll need per se.
You can try to simulate the AI software to gauge how much performance it will require. You can create benchmarks. There are all sorts of “lab” kinds of ways to gauge usage. Once you’ve got AI self-driving cars in the field for trials, you should also be pulling stats about performance. Indeed, it’s quite important that their be on-board monitoring to see how the AI and the hardware are performing.
With proper load balancing on-board the self-driving car, the load balancer is trying to keep the AI from getting starved, it is trying to ensure that the AI runs undisrupted by whatever might be happening at the hardware level. The load balance is monitoring the devices involved. When saturation approaches, this can be potentially handled via static or dynamic balancing, and thus the load balancer needs to come to play.
If an on-board device goes sour, the load balancer hopefully has a means to deal with the loss. Whether it’s redundancy or whether it is shifting over to have another device now do double-duty, you’ve got to have a load balancer on-board to deal with those moments. And do so in real-time. While the self-driving car is possibly in motion, on a crowded freeway, etc.
Conclusion
For those auto makers and tech firms that are giving short shrift right now to the importance of load balancing, I hope that this might be a wake-up call.
It’s not going to do anyone any good, neither the public and nor the makers of AI self-driving cars, if it turns out that the AI is unable to get the performance it needs out of the on-board devices.
A load balancer is not a silver bullet, but it at least provides the kind of added layer of protection that you’d expect for any solidly devised real-time system. Presumably, there aren’t any auto makers or tech firms that opted to go with the four burner stove when an eight burner stove was needed.
For free podcast of this story, visit: http://ai-selfdriving-cars.libsyn.com/website
The podcasts are also available on Spotify, iTunes, iHeartRadio, etc.
More info about AI self-driving cars, see: www.ai-selfdriving-cars.guru
To follow Lance Eliot on Twitter: @LanceEliot
For my Forbes.com blog, see: https://forbes.com/sites/lanceeliot/
For my Medium blog, see: https://medium.com/@lance.eliot
Copyright © 2019 Dr. Lance B. Eliot
