This documentation is built from the development branch, master. It includes the changes listed under Unreleased, which the latest release, CityLearn 3.6.0, does not have. To read the documentation of the version you installed (pip show citylearn), open the docs folder at its tag on GitHub, for example v3.6.0.

citylearn.agents.sac module

class citylearn.agents.sac.SAC(env: CityLearnEnv, **kwargs: Any)[source]

Bases: RLC

get_encoded_observations(index: int, observations: List[float]) → ndarray[tuple[Any, ...], dtype[float64]][source]
get_exploration_prediction(observations: List[List[float]]) → List[List[float]][source]

Return randomly sampled actions from action_space multiplied by action_scaling_coefficient.

static get_normalization_scale(std: float | ndarray[tuple[Any, ...], dtype[float64]]) → float | ndarray[tuple[Any, ...], dtype[float64]][source]

Divisor used to standardize observations and rewards: std + 1e-5, or 1 where std <= 1e-5.

A feature (or reward) that does not vary in the replay buffer when the statistics are computed, e.g. month in a warm-up that stays within one month, has no usable scale. Dividing by ~1e-5 would turn any later change into a z-score of ~1e5 and saturate the policy, so such features are centred but not rescaled.

get_normalized_observations(index: int, observations: List[float]) → ndarray[tuple[Any, ...], dtype[float64]][source]
get_normalized_reward(index: int, reward: float) → float[source]
get_post_exploration_prediction(observations: List[List[float]], deterministic: bool) → List[List[float]][source]

Action sampling using policy, post-exploration time step

predict(observations: List[List[float]], deterministic: bool = None)[source]

Provide actions for current time step.

Will return randomly sampled actions from action_space if end_exploration_time_step <= time_step else will use policy to sample actions.

Parameters:
  • observations (List[List[float]]) – Environment observations

  • deterministic (bool, default: False) – Wether to return purely exploitatative deterministic actions.

Returns:

actions – Action values

Return type:

List[float]

set_encoders() → List[List[Encoder]][source]

Get observation value transformers/encoders for use in agent algorithm.

The encoder classes are defined in the preprocessing.py module and include PeriodicNormalization for cyclic observations, OnehotEncoding for categorical obeservations, RemoveFeature for non-applicable observations given available storage systems and devices and Normalize for observations with known minimum and maximum boundaries.

Returns:

encoders – Encoder classes for observations ordered with respect to active_observations.

Return type:

List[List[Encoder]]

set_networks(internal_observation_count: int = None, observation_dimensions: List[int] = None)[source]
set_training_schedule(batch_size: int = None, standardize_start_time_step: int = None, end_exploration_time_step: int = None)[source]

Adapt the default training schedule to the episode, then validate it.

The policy acts from time step end_exploration_time_step + 1 and needs the standardization statistics, which update() computes at the first time step >= standardize_start_time_step at which the replay buffer holds batch_size transitions (and, for MARLISA, at least as many as its PCA has components). Only if the schedule cannot meet this are defaults adapted, in order: the default standardize_start_time_step waits for the minimum number of transitions, the default batch_size shrinks to the transitions collected by standardize_start_time_step, and the default end_exploration_time_step moves to the time step before standardization. Schedules that already meet it are left unchanged.

Parameters:
  • batch_size (int, optional) – User-given batch size, None if it is the default.

  • standardize_start_time_step (int, optional) – User-given earliest standardization time step, None if it is the default.

  • end_exploration_time_step (int, optional) – User-given end of exploration, None if it is the default.

Raises:

ValueError – If the schedule cannot be met without changing given values.

update(observations: List[List[float]], actions: List[List[float]], reward: List[float], next_observations: List[List[float]], terminated: bool, truncated: bool)[source]

Update replay buffer.

Parameters:
  • observations (List[List[float]]) – Previous time step observations.

  • actions (List[List[float]]) – Previous time step actions.

  • reward (List[float]) – Current time step reward.

  • next_observations (List[List[float]]) – Current time step observations.

  • terminated (bool) – Indication that episode has ended.

  • truncated (bool) – If episode truncates due to a time limit or a reason that is not defined as part of the task MDP.

class citylearn.agents.sac.SACRBC(env: CityLearnEnv, rbc: Agent | type | str = None, **kwargs: Any)[source]

Bases: SAC

Uses a rule-based controller to select actions during exploration before using citylearn.agents.sac.SAC.

Parameters:
  • env (CityLearnEnv) – CityLearn environment.

  • rbc (Union[Agent, type, str], optional) – Controller used to select actions during exploration: an citylearn.agents.base.Agent instance built on the same env, an agent class, or the import path of an agent class, e.g. ‘citylearn.agents.baseline.RBCBasicPolicy’. Any agent qualifies, including citylearn.agents.rbc.RBC and its subclasses and the rule baselines of citylearn.agents.baseline. Defaults to citylearn.agents.rbc.RBC.

  • **kwargs (Any) – Other keyword arguments used to initialize super class. When rbc is a class or an import path, the keyword arguments that the learner’s own constructors do not declare, such as action_map for citylearn.agents.rbc.HourRBC or the hyperparameters of a rule baseline, are also passed to the rbc constructor. The learner’s hyperparameters (e.g. lr, batch_size, hidden_dimension) are not.

get_exploration_prediction(observations: List[float]) → List[float][source]

Return the actions of rbc.

property rbc: Agent

Controller used to select actions during exploration.