Controller examples

For a first complete v3 simulation and operational BAU, start with the quickstart. This notebook retains the flat-interface controller, RL-wrapper and dataset-generation examples. Training runs need more time and dependencies than the first simulation.

Install CityLearn in the notebook environment:

[ ]:

%pip install citylearn

CityLearn Control Agents

Passive no-control agent

This passive agent disables active actions; it is not the operational BusinessAsUsualAgent. Construct a fresh environment before changing controller type.

Run the following to simulate an environment where the storage systems and heat pumps are not controlled (baseline). The storage actions prescribed will be 0.0 and the heat pump will have no action, i.e. None, causing it to deliver the ideal load in the building time series files:

[ ]:

from citylearn.agents.base import BaselineAgent as Agent
from citylearn.citylearn import CityLearnEnv

# initialize
env = CityLearnEnv('citylearn_challenge_2023_phase_2_local_evaluation', central_agent=True)
model = Agent(env)

# step through environment and apply agent actions
observations, _ = env.reset()

while not env.terminated:
    actions = model.predict(observations)
    observations, reward, terminated, truncated, info = env.step(actions)

# test
kpis = model.env.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Centralized RBC

Run the following to simulate an environment controlled by centralized RBC agent for a single episode:

[ ]:

from citylearn.agents.rbc import BasicRBC as Agent
from citylearn.citylearn import CityLearnEnv

# initialize
env = CityLearnEnv('citylearn_challenge_2023_phase_2_local_evaluation', central_agent=True)
model = Agent(env)

# step through environment and apply agent actions
observations, _ = env.reset()

while not env.terminated:
    actions = model.predict(observations)
    observations, reward, terminated, truncated, info = env.step(actions)

# test
kpis = model.env.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Decentralized-Independent SAC

Run the following to simulate an environment controlled by decentralized-independent SAC agents for two episodes:

[ ]:

from citylearn.agents.sac import SAC as Agent
from citylearn.citylearn import CityLearnEnv

# initialize
env = CityLearnEnv('citylearn_challenge_2023_phase_2_local_evaluation', central_agent=False)
model = Agent(env)

# train
model.learn(episodes=2, deterministic_finish=True)

# test
kpis = model.env.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Decentralized-Cooperative MARLISA

Run the following to simulate an environment controlled by decentralized-cooperative MARLISA agents for two episodes:

[ ]:

from citylearn.agents.marlisa import MARLISA as Agent
from citylearn.citylearn import CityLearnEnv

# initialize
env = CityLearnEnv('citylearn_challenge_2023_phase_2_local_evaluation', central_agent=False)
model = Agent(env)

# train
model.learn(episodes=2, deterministic_finish=True)

# test
kpis = model.env.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Other Standard Reinforcement Learning Libraries

Stable Baselines3 Reinforcement Learning Algorithms

Install the Stable Baselines3 version used by the current CityLearn wrapper examples:

[ ]:

%pip install "stable-baselines3==2.3.2"

Before the environment is ready for use in Stable Baselines3, it needs to be wrapped. Firstly, wrap the environment using the NormalizedObservationWrapper (see docs) to ensure that observations served to the agent are min-max normalized between [0, 1] and cyclical observations e.g. hour, are encoded using the cosine transformation.

Next, we wrap with the StableBaselines3Wrapper (see docs) that ensures observations, actions and rewards are served in manner that is compatible with Stable Baselines3 interface.

⚠️ NOTE: central_agent in the env must be True when using Stable Baselines3 as it does not support multi-agents.

[ ]:

from stable_baselines3.sac import SAC as Agent
from citylearn.citylearn import CityLearnEnv
from citylearn.wrappers import NormalizedObservationWrapper, StableBaselines3Wrapper

# initialize
env = CityLearnEnv('citylearn_challenge_2023_phase_2_local_evaluation', central_agent=True)
env = NormalizedObservationWrapper(env)
env = StableBaselines3Wrapper(env)
model = Agent('MlpPolicy', env)

# train
episodes = 2
model.learn(total_timesteps=env.unwrapped.time_steps*episodes)

# test
observations, _ = env.reset()

while not env.unwrapped.terminated:
    actions, _ = model.predict(observations, deterministic=True)
    observations, _, _, _, _ = env.step(actions)

kpis = env.unwrapped.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

RLlib

Install the RLlib version used by these wrapper examples:

[ ]:

%pip install "ray[rllib]==2.10.0" "pyarrow<21"

We advise that you include the ClippedObservationWrapper (see docs) wrapper when working with RLlib so that observations are always clipped within the observation space before sending to the agent if not, out-of-bound observations will raise a ValueError and terminate the training.

We also wrap the environment with NormalizedObservationWrapper (see docs) to ensure that observations served to the agent are min-max normalized between [0, 1] and cyclical observations e.g. hour, are encoded using the cosine transformation.

RLlib supports both single-agent and multi-agent algorithms. See below for an example for either case.

Single Agent

The single-agent interface for RLlib is the RLlibSingleAgentWrapper wrapper.

[ ]:

import warnings
from citylearn.wrappers import ClippedObservationWrapper, NormalizedObservationWrapper, RLlibSingleAgentWrapper
from ray.rllib.algorithms.sac import SACConfig as Config

warnings.filterwarnings('ignore', category=DeprecationWarning)

# initialize
env_config = {
    'env_kwargs': {
        'schema': 'citylearn_challenge_2023_phase_2_local_evaluation',
    },
    'wrappers': [
        NormalizedObservationWrapper,
        ClippedObservationWrapper
    ]
}
config = (
    Config()
    .environment(RLlibSingleAgentWrapper, env_config=env_config)
)
model = config.build()

# train
for i in range(2):
    _ = model.train()

# test
env = RLlibSingleAgentWrapper(env_config)
observations, _ = env.reset()

while not env.unwrapped.terminated:
    actions = model.compute_single_action(observations, explore=False)
    observations, _, _, _, _ = env.step(actions)

kpis = env.unwrapped.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Multi-agent

The multi-agent interface for RLlib is the RLlibMultiAgentEnv wrapper.

[ ]:

import warnings
from citylearn.wrappers import ClippedObservationWrapper, NormalizedObservationWrapper, RLlibMultiAgentEnv
from ray.rllib.algorithms.sac import SACConfig as Config
from ray.rllib.policy.policy import PolicySpec

warnings.filterwarnings('ignore', category=DeprecationWarning)

# initialize
env_config = {
    'env_kwargs': {
        'schema': 'citylearn_challenge_2023_phase_2_local_evaluation',
    },
    'wrappers': [
        NormalizedObservationWrapper,
        ClippedObservationWrapper
    ]
}
config = (
    Config()
    .environment(RLlibMultiAgentEnv, env_config=env_config)
    .multi_agent(
        policies={a: PolicySpec() for a in RLlibMultiAgentEnv(env_config)._agent_ids},
        policy_mapping_fn=lambda agent_id, episode, worker, **kwargs: agent_id,
    )
)
model = config.build()

# train
for i in range(2):
    _ = model.train()

# test
env = RLlibMultiAgentEnv(env_config)
observations, _ = env.reset()

while not env.terminated:
    actions = {p: model.compute_single_action(o, policy_id=p, explore=False) for p, o in observations.items()}
    observations, _, _, _, _ = env.step(actions)

kpis = env.unwrapped.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)

Neighborhood Dataset Generation

Aside the provided datasets that come with the CityLearn installation, custom single-family residential datasets can be generated in CityLearn by taking advantage of the End-Use Load Profiles for the U.S. Building Stock dataset. The citylearn.end_use_load_profiles.neighborhood.Neighborhood class makes this possible.

To learn more about the methodology used in this feature, refer to the CityLearn v2 paper.

Note that to make use of this feature, EnergyPlus 9.6.0 must be installed. Other EnergyPlus versions are not yet supported.

An example of a generating a dataset and using it in simulation is:

[ ]:

from citylearn.agents.rbc import BasicRBC as Agent
from citylearn.citylearn import CityLearnEnv
from citylearn.end_use_load_profiles.neighborhood import Neighborhood, SampleMethod

# path to version EnergyPlus 9.6.0 IDD
idd_filepath = '/Applications/EnergyPlus-9-6-0/PreProcess/IDFVersionUpdater/V9-6-0-Energy+.idd'

# build a neighborhood with n buildings through random sampling of single-family residential buildings in EULP dataset.
# Sampling population is filtered to include specific county and building vintage.
# train their LSTM thermal dynamics models and generate a CityLearn schema for the two buildings
neighborhood = Neighborhood()
n = 2
neighborhood_build = neighborhood.build(
    idd_filepath=idd_filepath,
    delete_energyplus_simulation_output=True,
    sample_buildings_kwargs=dict(
        sample_method=SampleMethod.RANDOM,
        sample_count=n,
        filters={
            'in.resstock_county_id': ['TX, Travis County'],
            'in.vintage': ['2000s']
        },
    ),
)

# simulate neighborhood in CityLearn
env = CityLearnEnv(neighborhood_build.schema_filepath, central_agent=True)
model = Agent(env)
observations, _ = env.reset()

while not env.terminated:
    actions = model.predict(observations)
    observations, reward, terminated, truncated, info = env.step(actions)

kpis = model.env.evaluate_v2()
kpis = kpis.pivot(index='cost_function', columns='name', values='value').round(3)
kpis = kpis.dropna(how='all')
display(kpis)