Did you know that cells in a battery pack can become imbalanced over time? Imagine 3 water containers, each meant to hold the same amount of water but having different levels of water considered to be imbalanced. In the same way, cells with different State of Charge (SOC) levels at a given time “t” are termed as imbalanced. Cells get imbalanced due to manufacturing mismatches, cell degradation over lifetime or different heat exposures.
When cells become imbalanced, some may charge and discharge faster than others. This can cause problems like:
1. Reduced overall battery efficiency.
2. Shortened battery lifespan.
3. Risk of overheating or damage.
Handling imbalances in the cells of a battery pack is an important feature of the Battery Management System (BMS). Making the SOC of each cell identical to each other is termed as cell balancing. Due to cell balancing, the overall efficiency of the battery pack increases in both charging and discharging cycles.
Source : Bacancy
Types of Cell Balancing:
1. Passive Balancing: The excess charge is dissipated as heat through the bleed resistors connected to each cell in the battery. As it is easy to implement and cost-effective, this method is mostly used in the balancing part of the BMS. But the disadvantage of the method is the wear and tear of the hardware, like resistors, and a lot of energy is wasted as heat, which also contributes to the rise in temperature of the board.
2. Active Balancing: The excess charge is transferred from higher SOC to lower SOC among the cells within the battery pack. Elements like inductors and capacitors are used to transfer the charge. Active balancing is fast when compared to passive, but it is very costly and hard to implement.
Rule-based passive balancing:
In Passive Balancing, when a cell's voltage exceeds the threshold, the excess energy is dissipated as heat through a resistor. Each cell is continuously monitored for its voltage (), predefined threshold voltage ( ) is set. At any timestep if , balancing is activated for that cell by turning on the shunt resistor connected to it.
When balancing is triggered, the excess energy from the high-voltage cell is dissipated through a resistor. The amount of power dissipated can be calculated using Ohm’s law for the resistor:
$$ P_i = \frac{V_i^2}{R} $$
Where:
- \( P_i \) : Power dissipated in the resistor (in watts)
- \( V_i \) : Voltage of the cell (in volts)
- \( R \) : Resistance of the balancing resistor (in ohms)The total energy dissipated over a period (in seconds) can be calculated as the power multiplied by the time:
$$ E_i = P_i \times \Delta t = \frac{V_i^2}{R} \times \Delta t $$Where:
- \(E_i\) is the energy dissipated in the form of heat (in joules),
- \(V_i\) is the voltage of the cell,
- \(R\) is the balancing resistor's resistance,
- \(\Delta t\) is the time for which balancing is active.Limitation of rule-based balancing:
The traditional balancing methods rely on predefined fixed rules. They don’t adapt or change according to the dynamics of the battery pack. As the battery pack exhibits more uncertainty, there should be a method that can be used to optimize the energy loss and do balancing that will align with the dynamics of the cells in the battery pack. This is where Reinforcement Learning comes into the picture. In this blog we will explore the overview of passive balancing with reinforcement learning.
What Is Reinforcement Learning?
Reinforcement Learning is a type of machine learning where an agent learns to make decisions and optimal policy by interacting with the environment in which it is operating. Instead of being explicitly programmed, the agent uses a trial-and-error method in which it tries to maximize the cumulative reward. An RL agent in a certain state takes an action that affects the environment. The environment in return provides a reward and a new state. The terminology used in RL :
1. Agent: A robot or a computer program.
2. Environment: Physical or modeled world in which the agent operates.
3. State: Current situation of the agent.
4. Reward: Feedback from the environment.
5. Policy: Method to map agent’s state to action.
6. Value: Future reward that an agent would receive by taking an action in a particular state.
Deep RL algorithms such as Q-learning, Deep Q-Network (DQN), Trust Region Policy Optimization (TPRO), Proximal Policy Optimization (PPO), and so on are used to learn optimal policy. These algorithms do not require prior knowledge of the environment, so they are called model-free RL algorithms. In RL, training data is obtained via the direct interactions of the agent with the environment. Reinforcement learning is very flexible and can be combined with machine learning and deep learning techniques to improve performance and optimize the process.
Passive balancing with RL:
Let’s consider 3 cells having a voltage range between 3.3v – 4.1v for each individual cell. These cells are connected in series to form a battery pack. 3 switched shunt resistors are connected to each cell, which are turned ON/OFF to dissipate the excess charge in cells as heat. The variance of SOC between the cells is monitored so that the balancing operation is continued or stopped by comparing it with the threshold that is set already. This is how passive balancing is performed using a rule-based approach.
Passive balancing with RL starts by defining and modeling the complex battery pack, which is the environment with which the agent will interact. How can RL fit passive balancing? This question can be answered by explaining the RL terminology, fitting them with passive balancing.
1. Agent: Balancing part of the BMS (Balancing algorithm).
2. Environment: The rest of BMS and the battery pack, which can be modeled using a library called Gym from OpenAI.
3. State: Current state of the environment in terms of SOC, SOH, status of the switches (ON/OFF), and so on at time ‘t’.
4. Action: Turning ON/OFF the switched shunt resistors. Here we considered 3 cells, so we have 23 i.e., 8 actions possible. They are (0,0,0), (0,0,1), (0,1,0) …... (1,1,1). Here 0 represents the OFF state and 1 represents the ON state of the resistor.
5. Reward: The reward function is designed to minimize the variance of the SoC, improve battery life, and add penalties for excess switching of the resistors.
6. Policy: Deep RL algorithms such as DQN, TRPO, etc. can be used to learn optimal policy.
In RL an action is randomly selected in the beginning, i.e., switching of resistors from action space, which leads to a change of state in the environment, i.e., change in state of switches, decrement in SOC based on energy dissipated from the resistors. Based on the action taken, the agent receives a reward; here, in our case, the agent receives a reward for minimizing the SOC difference across the cells, making less switching of the resistors, and increasing overall battery capacity.
def step(self, action):
switch_states = action
soc = self.state[:self.num_cells]
voltage = self.state[6:9]
# Simulate SoC changes based on switches
for i in range(self.num_cells):
if switch_states[i] == 1:
# Discharge through the shunt resistor
I = voltage[i] / self.shunt_resistance # Current in Amperes (A)
E = I**2 * self.shunt_resistance * self.delta_t # Energy dissipated (Joules)
delta_soc = (E / (3400 * 3600)) * 100 # Assuming C_nom = 3400 mAh
soc[i] -= delta_soc # Reduce the SoC based on energy dissipated
soc[i] = np.clip(soc[i], 0.1, 0.9) # Clamp SoC to range [0.1, 0.9]$$
\text{Reward} = -\alpha (\text{SoC Variance}) - \beta (\text{Switching count}) - \gamma (\text{Battery pack capacity})
$$
Where α, β, γ are weight factors for different priorities. According to the defined reward function, the model performance is best when the rewards are close to zero.variance = np.var(soc)
num_switch_changes = np.sum(np.abs(switch_states - self.state[self.num_cells:2 * self.num_cells]))
working_indicator = 1 if all(0.1 <= soc[i] <= 0.9 for i in range(self.num_cells)) else 0
reward = -math.log(variance + 1e-6)**2 - 0.1 * num_switch_changes + 0.5 * working_indicator
RL algorithms are used to make the agent learn the optimal policy. Let us consider PPO as an RL algorithm to explain the process. The above process is repeated for n number of time steps. At some point in time, the agent ends up in either a balanced state or a termination state where the agent doesn’t have any further state to move; this will end one episode in the training process. We can put a patience level, which will end the episode after repeating the process for those number of steps.
# Define and wrap the environment
env = PassiveBalancingEnv(render_mode='human')
env = make_vec_env(lambda: env, n_envs=1)
# Define the RL model (PPO)
model = PPO("MlpPolicy", env, learning_rate=0.001, gamma=0.99, verbose=1)
# Instantiate the callback
reward_logger = RewardLoggerCallback()
# Train the model with the callback
model.learn(total_timesteps=50000, callback=reward_logger)
# Save the trained model
model.save("passive_balancing_ppo")When an episode is completed, the algorithm by default resets the environment, so that the process will repeat again by making the agent learn by interacting with the environment.
def reset(self):
# Reset state with random initial SoC and switches OFF for 3 cells
soc = np.random.uniform(0.1, 0.9, self.num_cells)
switches = np.zeros(self.num_cells)
voltage = 3.3 + (soc * (4.1 - 3.3)) # Voltage based on initial SoC
# Combine SoC, switch states, and voltage into the state
self.state = np.concatenate([soc, switches, voltage])
self.current_step = 0
return self.stateIn this way the agent would interact with the dynamic conditions of the battery and make an optimal policy to choose at a given situation so that it will improve the performance of the battery. Why is RL an optimal method compared to rule-based balancing? The following will address this question.
1. RL has the capability of learning and adapting to its environment.
2. It is an optimization-based approach that tries to maximize a reward function by finding strategies that minimize energy loss, excess switching of resistors, and improve the battery capacity.
3. By using a trial-and-error approach, RL will explore different strategies and learn the best course of action for a situation.
4. As said, batteries show more uncertainty. RL is designed to make decisions in certain conditions.
The following graph is an example of the rewards gained by the PPO algorithm for balancing the SoC in the cells for 50 episodes. The model can be optimized by training it for more episodes.
Reinforcement Learning (RL) proves to be a transformative approach in the field of passive cell balancing. By leveraging adaptive decision-making, RL achieves a remarkable 16.8% improvement in overall battery pack capacity, a 69.4% reduction in state-of-charge (SOC) variance among cells, and a 40.4% decrease in the number of switching operations. These advancements highlight RL's ability to optimize balancing strategies dynamically, reduce energy loss, and extend battery life, making it a superior alternative to traditional rule-based methods.
NOTE: The current environment and logic are simplified for demonstration; precise battery simulations will follow in future blogs.
Frequently Asked Questions:
1. Why is cell balancing important?
Balancing keeps all cells at the same charge level. Without it, some cells overwork, which reduces battery life, lowers efficiency, and may even cause overheating or damage.
2. What is passive cell balancing?
In passive balancing, extra energy from a highly charged cell is burned off as heat through resistors. It’s simple and cheap but wastes energy.
3. What is active cell balancing?
In active balancing, extra energy from one cell is moved to other cells. It saves energy and is faster but costs more and is harder to build.
4. What is the problem with rule-based balancing?
Rule-based balancing always follows fixed instructions. It doesn’t adapt to changing battery conditions, which can cause wasted energy and less effective balancing.
5. How does Reinforcement Learning (RL) improve balancing?
RL is like “learning by trial and error.” It teaches the battery system how to make smarter balancing decisions over time, reducing energy waste, switching less often, and keeping cells healthier.