Research and/or engineering questions/objective: Adaptive cruise control (ACC) aims to reduce driver effort and increase passenger safety. The rise in the number of cars on the road, as well as the implementation of stricter emissions regulations, requires a solution that can enhance traffic flow, vehicle stability, passenger comfort, and fuel economy. We propose a multi-objective cooperative adaptive cruise control (CACC) using deep reinforcement learning (DRL) for a hybrid electric vehicle. The CACC control actions and energy management system (EMS) power-splitting decisions are interdependent, so we propose a multi-agent DRL (MADRL-CACC) solution for the CACC and EMS of HEVs. Methodology: The framework consists of two layers: the upper layer has a DRL agent, which produces throttle and brake pedal commands considering headway, vehicle stability, passenger comfort, and fuel economy; the lower layer consists of an energy management system, which calculates the power splitting between the internal combustion engine (ICE) and electric machine (EM). The equivalent consumption minimization strategy (ECMS) handles the energy management between them. The fuel and electrical energy equivalence is obtained by parameters called equivalence factors and is typically tuned by a trial-and-error approach. Its selection is essential to obtain an optimal power split between the ICE and EM, which in turn impacts fuel efficiency. A separate DRL agent has been designated to select the equivalent factor in order to reduce the mass flow rate of the ICE and EM. Moreover, the DRL agent ensures charge-sustaining behavior, i.e., it avoids deep discharge of the battery. Results: Simulations have been formulated to analyze car-following behavior in urban and highway environments. The performance of the MADRL-CACC is evaluated against two different control strategies: PID-ECMS-ACC, and SADRL-ECMS-CACC. Further, the lead vehicle acceleration information is absent (present) in the PID-ECMS-ACC (SADRL-ECMS-CACC and MADRL-CACC) strategies. The gains in PID-ECMS-ACC are fine-tuned via a genetic algorithm to minimize the RMSE of headway and fuel consumption. The SADRL-ECMS-CACC use a single-agent DRL approach for throttle/brake command and ECMS for power splitting between ICE and EM. The equivalence factor of ECMS is initialized and kept constant for PID-ECMS-ACC, and SADRL-ECMS-CACC, whereas the second DRL agent determines the equivalence factor in the MADRL-CACC setup, providing fuel-saving benefits of over 6%. The MADRL-CACC obtains better trade-off among the key performance indexes, providing safe inter-vehicle distance, improved road usage efficiency, satisfactory comfort to the passengers, and fuel economy. Limitations of this study: DRL is often used to solve hard problems, despite the fact that it needs a lot of data and processing power. In the real world, the uncertainty in the estimation of the vehicle states’ may have an effect on the performance of the proposed algorithm. In the multi-objective problem, the weights given to each reward component affect the DRL learning process; hence, dynamic weights should be assigned to achieve a better trade-off based on the current context. What does the paper offer that is new in the field in comparison to other works of the author? The control strategies for ACC and EMS can be categorized into rule-based, model-based, and data-driven. The data-driven approaches can solve multi-objective problems even without extensive knowledge about the non-linear environment, overcoming the limitations of rule- and model-based methods. In literature, ACC is based on single or multiple objectives, such as minimizing headway tracking error, jerk, wheel slip, comfort, and fuel consumption. We overlooked fuel economy in our prior works and focused on the other aforementioned objectives. In this work, we propose a data-driven method for CACC and EMS in HEV to maximize powertrain efficiency, along with our previous work objectives. Conclusion: The MADRL-CACC framework was suggested to accomplish CACC functionalities and energy management strategies for the HEV. The performance of MADRL-CACC was evaluated in urban and highway driving scenarios. The controller aims to improve headway tracking, passenger safety, vehicle stability, and fuel economy. The MADRL-CACC outperforms different control strategies in terms of traffic efficiency, wheel slip, jerk, and fuel consumption. In future work, we will focus on the real-time implementation capability of the proposed strategy using a rapid control prototyping platform. The framework can be further evaluated in real-world experiments since environmental conditions may influence performance.

Mr. Shailesh Hegde, Ph.D. Student, Politecnico di Torino