A German automated driving project led by Bosch has produced a prototype of a virtual companion to help lorry drivers stay awake and alert.

In addition to Bosch, which acted as consortium leader, the project’s other participants were Volkswagen, Man Truck & Bus, University of Stuttgart, Hochschule der Medien, Spiegel Institut and CanControls.

Called Tango, for “Technologie für automatisiertes Fahren, die nutzergerecht optimiert wird” (technology for automated driving, optimised to the benefit of the user), the project was funded with a grant of €5m from the German Federal Ministry for Economic Affairs & Energy.

The project focused on partially and conditionally automated driving at SAE levels two and three. It produced a prototype for an attention and activity assistant, a virtual companion that keeps the driver alert, counteracts fatigue and makes driving as pleasant as possible.

Lorries are the mainstay of logistics. In Germany, three-quarters of all goods are transported by road. Time pressure, long working hours and monotonous convoys are part and parcel of the lorry driver’s everyday routine. If the driver is distracted, overtired or reacts too slowly, the consequences can be disastrous.

Automated driving functions are key to avoiding accidents and dangerous situations. They alert drivers and support them in critical situations, and are increasingly in a position to assume monotonous and fatiguing tasks. At the same time, people remain irreplaceable even at subsequent levels of automation: they must be able to take control if necessary. What is the ideal way to coordinate the interaction between vehicle and driver?

This was the focus of the Tango project over the past three and a half years.

“In the future, manual and automated driving phases will alternate: as in a relay race, responsibility for steering will pass from vehicle to driver and back,” said Bosch project manager Michael Schulz. “In partially automated driving as per SAE level two, the driver must be prepared to intervene at any time; in conditionally automated driving as per SAE level three, only when necessary. The objective is to keep the demands placed on the driver at the optimum level so that they are always capable of taking control.”

Here is where the attention and activity assistant comes into play. Depending on assistance level, the driver’s condition, and what the actual driving situation will permit, the assistant can provide the driver with various secondary activities: listening to music or audiobooks, reading emails, writing text messages, watching movies, scheduling the day’s appointments, planning the route, and doing fitness exercises. The system has not only to determine reliably the specific driving situation, but also detect and properly interpret the driver’s condition at any given moment, for example, if the driver is tired or distracted.

In addition, the research project installed sensors to monitor the interior and combined them with artificial intelligence methods. Cameras detect if the driver’s eyes are closing, if they are blinking frequently, are losing sight of the road, or even if their head is drooping to the side with fatigue. Intelligent algorithms evaluate the images, interpret them and launch countermeasures. These might be a warning, an offer to perform a secondary activity or an active intervention such as braking.

To get to the root causes of fatigue and distraction and to determine precisely the requirements for the attention and activity assistant, the researchers accompanied lorry drivers on their routes, interviewed them about their experiences, assessed online daily logs, and repeatedly compared the intermediate development stages against test subjects so as to make revisions.

This process involved driving simulators, drives in test vehicles, and even a little magic: in a right-hand-drive truck, the passenger side of the cab was also equipped with a steering wheel, brakes, accelerator, and all the necessary control and display elements, with the two halves separated by a privacy screen. This approach, dubbed the Wizard of Oz, enabled the researchers to simulate the automated function and observe user behaviour under realistic conditions.

“We deliberately focused our work on the user and user requirements,” Schulz said.

The researchers started from the premise that lorry drivers are often wary of automated driving because they associate the technology with being stripped of responsibility and worry that their job will be eliminated. As a result, they don’t always use even today’s assistance systems, sometimes deliberately turning them off. The project determined that the strongest argument in favour of more automation was the increase in safety, though comfort as well as more relaxed driving and arrivals play a role.

Against this backdrop, there was particular emphasis on the human-machine interface (HMI).

“In the future, a vehicle will have to interact with the driver and serve as a partner,” Schulz said. “Perfect interaction between vehicle and driver requires operation to be simple, intuitive and inviting.”

The prototype contains a control panel consisting of several displays, which combines visual, acoustic, and haptic elements and depicts the attention and activity assistant as an avatar. Findings from the research project will flow into further development in areas such as interior monitoring, automated driving and entertainment systems. These findings can apply not only to lorries but also to cars, where they also stand to increase safety in road traffic.