Artificial intelligence is set to be the stepping stone between driver assistance systems and truly autonomous vehicles. Chris Pickering reports

You’d be forgiven for thinking that fully autonomous cars were just around the corner. In some respects, of course, they are. Partial automation – along the lines of Tesla’s much-publicised Autopilot – is set to become commonplace on premium cars over the next few years. Even when it comes to higher levels of autonomy, much of the required hardware is already available.
It’s all so tantalisingly close. And yet there is a huge amount of work – not to mention a good deal of legal and administrative wrangling – to be done before we can safely switch our cars over to autonomous mode and go to sleep.
To cross that threshold, autonomous cars have to truly comprehend their environment. They need to be able to identify potential hazards, anticipate the actions of others and make decisions of their own. The key to this ability is artificial intelligence, with systems such as neural networks promising to take us into a brave new world of machines that think for themselves.
Most of the sensor technology is here already. For long range use, radar is the default choice. It’s already widely used in adaptive cruise control and developers are aiming for up to 400 metres’ range. The same technology can be used to provide mid-range detection, along with lidar and stereo video cameras. For close proximity work, ultrasonic sensors and short-range cameras are the preferred solutions.
“All these sensors have strengths and weaknesses,” explained Charles Degutis, director of product management for highly automated driving at Bosch. “Radar is very powerful, but it can bounce off tunnels and bridges, and it can struggle to differentiate small closely-spaced objects. Video provides lots of detail, but it can be blinded by things like glare. And lidar gives you a 3D picture, but being light-based it can degrade in high moisture situations.”
Bosch believes the best way forward is to combine all three sensor types, giving a more comprehensive picture, plus a degree of redundancy. Some, however, claim that video on its own could be sufficient, and potentially cheaper, given the right processing. Either way, the sensor technology is unlikely to be an obstacle.

The final piece of the jigsaw is high-resolution mapping. In urban environments, autonomous cars will be able to pinpoint their location down to an inch or so by referencing sensor data to highly accurate 3D maps. These will be generated by radar surveys and kept up to date using data from fleets of vehicles connected to the cloud.
AI inside
Increasingly, the challenge facing autonomous vehicles is not so much capturing the world around them, but making sense of it.
The process of identifying and classifying objects from sensor data is known as semantic segmentation. For human adults – trained to recognise patterns from birth – this is a slightly abstract concept; we see an image of a car and instinctively know what it is, even if it’s not a specific type we’ve encountered before. For computers, however, this poses a significant challenge. The system has to recognise that, say, a small two-seater convertible is fundamentally the same type of object as a seven-seat SUV. Likewise, pedestrians and roadside objects all come in a bewildering array of sizes and forms.
In order to decipher these complex situations, autonomous vehicle developers are turning to artificial neural networks. As the name implies, these computer systems are inspired by the vast clusters of neurons found in the brain, and they ‘learn’ in a very similar way.

In place of traditional programming, the network is given a set of inputs and a target output (in this case, the inputs being image data and the output being a particular class of object). Essentially, what it does is feed the data into the mass of interconnected neurons – each of which can have tens of thousands of connections to the others – and then compare the observed output to the target. Over successive iterations the network refines itself, changing the strength of certain connections until the input exactly matches the desired output.
In the right conditions, neural networks can already exceed the capacity of humans in discerning specific patterns
Eventually, the network can learn to spot the tell-tale features that identify a particular class of object. It doesn’t follow any preset rules for identifying them, though. For want of a better description it simply ‘knows’. It’s this ability to think outside the box that makes neural networks such a powerful tool for semantic segmentation.
“In the right conditions, neural networks can already exceed the capacity of humans in discerning specific patterns,” said Christoph Peylo, global head of the Bosch Centre for Artificial Intelligence (BCAI). “What sets them apart is that they are capable of digesting highly-dimensional data. Other processes, such as decision trees, can work well for some applications, but they can’t cope with too many attributes. If you think about the range of inputs on an autonomous car, you might have data from the camera, radar, lidar, the road conditions, the humidity…perhaps 10 highly-dimensional sources. With so many attributes a neural network would make sense.”
The process of training a neural network for semantic segmentation involves feeding it numerous sets of training data with labels to identify key elements, such as cars or pedestrians. This data can be generated from simulations (providing they’re accurate enough) or captured from real-world footage.

The engineers at BCAI use a combination of the two, explained Peylo: “The system learns through specific examples, so you have to ensure that everything that’s potentially relevant can be trained. You can drive for perhaps millions of miles and not encounter a specific hazard, so you have to add those cases [artificially].”
Accurately identifying objects is a major step towards predicting their behaviour. A car, for example, generally follows a different set of rules to a pedestrian. But in order to make decisions, the car also needs to be able to cope with situations and behaviours that are outside of the normal rules. What should it do if there is a broken down vehicle blocking the carriageway, for instance, or how would it merge into another stream if there were no clear road markings?
A neural network could be taught to recognise that a ball bouncing into the road could be followed by a child.
In theory, this is another prime candidate for the use of neural networks. They could be used to predict behaviour based on a sequence of events. It’s not inconceivable, for instance, that a neural network could be taught to recognise that a ball bouncing into the road could be followed by a child.
“The expertise that you need to drive a car cannot be fully described in an algorithm, but you can learn by experience. Machine learning allows computers to carry out the same process,” said Peylo.
Unfortunately, at the moment there’s a snag. “Neural networks are very powerful, but they are not yet fully understood,” he explained. “We see the results, but we cannot say exactly how the machine came up with the solution. Making it understandable and explainable is a very important challenge, particularly for applications that have to be verified and certified. Understanding how the neural network functions is a prerequisite for that, and it’s one of our major research topics at BCAI.”
For the time being, Bosch prefers to use probability-based models for high-level decision making. These look at the chances of a vehicle diverging from its anticipated behaviour (i.e. failing to stop for a red light) and evaluate the potential risk. This technique is not as powerful or as flexible as a neural network, but it does have the key advantage that every decision can be tracked and understood.

Machine learning is already employed for semantic segmentation in driver assistance systems, such as autonomous emergency braking, though. It allows partially-automated cars to carry out tasks that would be virtually impossible with traditional computing techniques, helping them to comprehend the abstract and unpredictable world of driving. And in the future it could hold the key to cars that truly think for themselves.
Smart trucks
Passenger cars may grab the headlines, but it’s arguably commercial vehicles that lead the way in the adoption of driver assistance systems. Since 2015 all new trucks (over 8 tonnes) sold in the EU have had to be fitted with autonomous emergency braking (AEB). Lane keeping functions and adaptive cruise control (ACC) are also widespread, which means that commercial vehicles are already ahead of many passenger cars.
“We see a lot of potential for driver assistance systems in trucks,” said Bosch’s Emanuel Willman. “The commercial vehicle business is driven by total cost of ownership and reducing the number of accidents is a major part of that. With a highly automated truck, we think you could eliminate 90 per cent of the accidents that occur today.”
The next big thing is likely to be turn assist systems, which could dramatically reduce the number of accidents that occur between lorries and cyclists, he explains.
Looking ahead, several companies are already testing autonomous trucks and the technology required for partial autonomy on motorways is more or less production-ready. It’s closely related to that found on passenger cars, although the sensors have to be more robust (to cope with a potential million-mile lifespan). The software also has to be retrained to recognise objects from a very different angle, perched several metres further up.
Another function Bosch expects to see in the future is platooning. Here, a driver can alert others via the cloud that they are willing to lead a convoy. The other trucks can then ‘dock’ autonomously with the lead vehicle as they drive along. Willman says he foresees a system where the vehicles behind would pay a contribution to the lead driver’s costs in return for this service. It could reduce the aerodynamic drag of the convoy as a whole, as well as provide time for the other drivers to rest or carry out admin.
So goodbye to the alleged freedom of the open road. It all sounds feasible but at what cost in terms of front end costs to the vehicle. And what happens in the event of systems failure? Default to human control? Where would the liability reside in the event of a crash?
The platooning concept is a real worry. How would traffic joining a major road/motorway respond if a train of trucks is occupying the nearside lane? I could foresee this becoming a major stumbling block and cause of further difficulties.
The comment regarding the reduction of accidents involving trucks is pertinent. Trucks are involved in a proportionately higher level of traffic accidents but we seem to accept this and the killing of 3000 people and thousands of serious injuries a year in road related accidents as part of the price we are prepared to pay as a society for high levels of mobility.
I have watched, read, listened to many of the pieces about autonomous road vehicles: I have done so with little detailed knowledge of the technology in development, but of course as a driver.
[One proper? accident, when I was a young and in-experienced driver, which had as one result a crash which very slightly scared my girl-friend (who has been my wife for 52 years!)] Of course I use the many small advances (cruise control, cameras, sensors for parking) that have gradually enhanced the driving experience -and the power-trains, gears, electronics, comfort etc of modern cars are orders of magnitude better than those before. The shear number of vehicles on all roads has multiplied exponentially during my career: [I can even recall in the late 50s going out for a drive for fun and pleasure? NOT now!] The autonomous transport system -offered to all?- as opposed solely to professional drivers (plane, train, ship…) seems a step too far.
I was unaware (other than as we all surely do, noting their propensity for aggressive driving) that truck (& white van?) transport causes a higher level of accidents than normal driving. A sad reflection on those responsible. Will a non-human ‘brain’ make any difference?
The driver who do not bother to yield at entrance ramp, who shoot across the inside lane to exit a roundabout, undertake, tailgate or do not lift a finger to operate the turn signals…should definitely be moved to autonomous cars. The drivers doing it properly can remain driving. The infrastructure must keep standard up for motorway entrance ramp (currently poor care to the sizes), must keep the yield sign at entrance ramps, must mark the roundabout lanes (some drivers are neither inner or outer exit lane). From this point of view it is a good thing.
The concept of autonomous vehicles is more of something that can (with development) be done without a reasoned analysis of should it be done. There are places, times and events where autonomous vehicles can or should be used and plenty where they shouldn’t. The idea being sold is that we don’t need a driving licence, but who retains control, and in the event of the ‘driver’ being responsible how can we ensure that they are able to take control in an emergency, being woken up, gaining full situational awareness and so on? Trivial things like how do you instruct a vehicle that has off-loaded passengers or goods to go an wait out of the way, in a yard. How do you instruct specific vehicles to find their way out of that yard when others are blocking the way? How does an autonomous vehicle get controlled or directed if someone wants to park under that specific tree or with that view, which will not be defined by a geotag? Controlling a vehicle is relatively simple providing it is operating within a constrained, known and mapped environment, but outwith that it is many more orders of magnitude of difficulty. Human beings are fallible, but if we design a truly AI system then that will also be subject to the same fallibilties of incorrect or inadequate decision making as the biological brain, for the same reasons. So what is to be gained, and when the claims companies start to sue the ‘driver’ are the system manufacturers going to be held responsible when whatever the cause or outcome it will be their code that created or caused the incident? Thetre is a simple solution every single one of us has an AI system inside our skulls, which only needs training and experience, so why create another that nobody will have effective control over? The concept of AVs is more a solution looking for a problem!
My BT home router can sometimes drop out and reconnect 20 time a day. Imagine what will happen when the wifi signal gives up or someone manages to take control of the wifi connected trucks hurtling along the motorway.
Two things that could be addressed now that would help a deal are restricting White Vans away from the outside lane of motorways and 3 lane highways, and forcing all cyclists to have rear number plates (the square type obviously) and insurance. The first of these is because anyone driving in this country looks not at the vehicle immediately in front but also the next 3-4 vehicles ahead, to allow adequate warning and braking time. Being unable to see beyond the back door of a van is a major hazard. In the USA drivers only look one car ahead and as a result have many more collisions. The second thought is about identification. Too many cyclists think the rules of the road are not for them Some won’t even use cycle lanes because they are too restrictive. Being able to identify them takes a step nearer to parity.
As for the AI – it will come in due time I am sure, but hopefully long after I am no longer on the roads.
Just because we can do all of this (possibly) does not mean we should!
As an older driver I am happy that my driving days will be over before this chaos really arrives in earnest. Jobs (and glory) for the academics and perceived riches for the manufacturers seems to be driving all of this. The politicians, as always, do not have a clue .
My computer crashed this week and my phone went into “partial non-responsive mode” and required turned-off-and-on-again to restore normal function. Complex software crashes, locks up and does silly things regularly. I work in the embedded environment, when our software crashes (when not if) the watchdog resets the product and it tries to pick up from where it left off (sometimes vital data is lost). Having a car doing this at speed on a busy road is a terrifying thought!
While it isn’t necessary for AVs to have access to broadband or wifi, they will need GPS, which is not always available, and they will have to use more accurate and wider coverage mapping than is available on current satnavs, otherwise they will be limited to tarmac roads only. Decision making has to be refined with at least as much flexibility as a biological brain. To allow driving in areas that are not covered by mapped infrastructure vehicle controls will need to be retained for human use or intervention. So instead of reducing mechanical complexity it will be even more than at present, coupled with even more complex electrical and electronic systems. And how will they work in an electrically denied environment?
I’ve been listening to these arguments most of my adult life, from the, much maligned (at the time) seat belt and crash helmet laws (and more recently, airbags) where I’m sure, as was being argued at the time, over the years there have been many deaths and life changing events that are wholly attributable to the fact that the victim was wearing a seat belt or a crash helmet. The fact remains though, in both cases, as soon as these laws were introduced there was an immediate and conspicuous reduction in vehicle related fatalities and injuries.
I for one am convinced that on the day that A.V’s. become mandatory, there will be a similar, dramatic reduction in vehicle related injuries and deaths.
I’m sure, anyone that’s been involved in a collision above 30 mph knows, the argument about a computer choosing between hitting a dog or a child in an emergency isn’t really that relevant as most people, particularly those over 45, don’t posses sufficient reaction time to make such decisions’. In most cases you’re a just a passenger.
I also believe that a good deal of he reluctance to accept this new technology is based, not around safety, but more (disingenuously perhaps) around the loss of the excitement of driving, the thrill of speed, the testing of one’s abilities to judge the limits of the vehicle and the feeling of freedom to do what you want, when you want (as long as a copper isn’t watching).
Remember, the most dangerous part of a car is; the nut that holds the steering wheel!
It would seem that most motorists already use some form of artificial intelligence because there doesn’t seem to be too much evidence of natural intelligence north of the neck.
It would help the debate if proponents of this wonderful technology start thinking about the flaws, which are substantial, and not gloss over the wider implications which have massive ramifications. People’s reaction time for example is usually sufficient and the processing involved in a computer making the same value judgements are still bound by the same laws of physics, momentum, mechanical responsiveness, etc. While trundling up a motorway or going from car park to car park in a built up area using a personal vehicle may well be suitable for autonomy, the vast majority of general driving is not likely to be for a very long time in the future. We would be better off spending those research budgets on enhancing human capacity instead of replacing it. No computer is likely to be as adaptable and resourceful as a human brain, which can easily cope with many novel experiences and situations, by inventing new solutions.
I’m sure, anyone that’s been involved in a collision above 30 mph knows, the argument about a computer choosing between hitting a dog or a child in an emergency isn’t really that relevant as most people, particularly those over 45, don’t posses sufficient reaction time to make such decisions’. In most cases you’re a just a passenger.
Well, that’s written off a very large percentage of the population-odd indeed that older drivers usually get cheaper insurance than you ?quick-witted young, careful drivers? Never noticed a preponderence of oldies overtaking me on the road? I had to take an advanced driving course for my career-whether that was to improve my driving or just so that my employers could say that I should have known better if an accident had occurred I am not certain, but it certainly makes you think deeper over the matter of driving fast in traffic . I also realised that I then drove more carefully in my own car, possibly because of the thought that I had done so and so had satisfied a hard judge that I was a capable driver. Ref the making a difficult decisions about which of the two too hit/avoid–instinct would make its own decision, surely? Or does Mr Kerry find that making such a “difficult” decision is solely a gift of the younger driver? And what’s that about “usually a passenger….”? Wrapped in warm blankets and hot water bottle as well, perhaps.
Stupid statement
My comments weren’t intended to criticise anyone’s driving capabilities, nor were they directed at the over 45’s (or younger drivers, come to that).
I was merely pointing out that as one gets older, reaction times become slower (and as a male, with no advanced driving skills certificates, who is fast approaching 60 years of age, it’s something that I’m very conscious of).
There are many arguments that postulate about the choice an AV would make when faced with a morality based decision (child vs dog is but one). My point here is that these arguments are usually emotionally charged and deflect from the most important fact, which is: AV’s have the potential to save large numbers of lives.
If this ever happens I have no doubt that there will be a small proportion of deaths and injuries that are completely attributable to the AV.
If you heard “he’d be alive today if he hadn’t been wearing a seat belt”, how many of you would vote to repeal seat belt laws?
I’m sorry, I didn’t understand your point about passengers, blankets and hot water bottles.
Kerry
The original poster makes a very good point. Don’t get unnecessarily huffy about the “over 45” line. Our reactions do slow with age. Younger drivers are higher risk not because their reactions are poor but because they’re less experienced and in some cases more likely to be going faster than is wise for the conditions.
I had a crash many years ago. I was, to all intents and purposes, a passenger for the last few yards. Sure, you want to steer the car through the gap between the tree and the wall but physics may not give you the option.
Ok, so we have a new vehicle, all-singing, all-dancing, but what happens when it starts getting old and the crappy (cheap) connectors used in its assembly start to fail? There isn’t anyone to adequately service and fault-find on the vehicles we already have, let alone when the complexity is quadrupled by Lidar/Radar/ etc etc. I can foresee a time when an expensive new-ish vehicle will be taken off of the road by a faulty sensor, not by sensor failure but by simple connector failure and poor diagnostic maintenance.
Has there been a discussion regarding the humans who are destructive, non-co-operative, resentful, etc., who will constructively target autonomous vehicle with a view to causing havoc? They will do it. Even a small boy who realises that he can step out in front of a robot which will react to him will have great fun, until… We should be considering recording interactions with a view to holding them responsible in court.