Reward Function

A reward is calculated and returned each time citylearn.citylearn.CityLearnEnv.step() is called. The reward time series is also accessible through the citylearn.citylearn.CityLearnEnv.rewards property.

CityLearn provides custom reward functions for centralized agent and/or decentralized control architectures:

Class

Equation

Centralized

Decentralized

citylearn.reward_function.RewardFunction

\[min(-e, 0)\]

✔️

✔️

citylearn.reward_function.MARL

\[\textrm{sign}(-e) \times 0.01(e^2) \times \textrm{max}(0, E)\]

✔️

citylearn.reward_function.IndependentSACReward

\[min(-e^3, 0)\]

✔️

✔️

Where \(e\) is a building’s net electricity consumption while \(E\) is the district’s net electricity consumption. For rewards that work with bothe centralized and decentralized agents, \(e\) is interchangeable with \(E\) in their equation depending on the value of citylearn.citylearn.CityLearnEnv.central_agent.

How to Point to the Reward Function

The reward function to use in a simulation is defined in the reward_function key-value of the schema:

{
   ...,
   "reward_function": {
      "type": "citylearn.reward_function.RewardFunction",
      ...
   },
   ...
}

How to Define a Custom Reward Function

CityLearn also allows for custom reward functions by inheriting the base citylearn.reward_function.RewardFunction:

from typing import List
from citylearn.reward_function import RewardFunction

class CustomReward(RewardFunction):
    def __init__(self, agent_count: int, electricity_consumption: List[float], carbon_emission: List[float], electricity_price: List[float]):
        super().__init__(agent_count, electricity_consumption=electricity_consumption, carbon_emission=carbon_emission, electricity_price=electricity_price)

    def calculate(self) -> List[float]:
        """Calculates custom user-defined multi-agent reward.

        Reward is the `carbon_emission` for each building.
        """

        return list(self.carbon_emission)

The schema must then be updated to reference the custom reward function:

{
   ...,
   "reward_function": {
      "type": "custom_module.CustomReward",
      ...
   },
   ...
}