Baselines

OpenAI Baselines is a set of high-quality implementations of reinforcement learning algorithms.

These algorithms will make it easier for the research community to replicate, refine, and identify new ideas, and will create good baselines to build research on top of. Our DQN implementation and its variants are roughly on par with the scores in published papers. We expect they will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones.

Prerequisites

Baselines requires python3 (>=3.5) with the development headers. You'll also need system packages CMake, OpenMPI and zlib. Those can be installed as follows

Ubuntu

sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev

Mac OS X

Installation of system packages on Mac requires Homebrew. With Homebrew installed, run the follwing:

brew install cmake openmpi

Virtual environment

From the general python package sanity perspective, it is a good idea to use virtual environments (virtualenvs) to make sure packages from different projects do not interfere with each other. You can install virtualenv (which is itself a pip package) via

pip install virtualenv

Virtualenvs are essentially folders that have copies of python executable and all python packages. To create a virtualenv called venv with python3, one runs

virtualenv /path/to/venv --python=python3

To activate a virtualenv:

. /path/to/venv/bin/activate

More thorough tutorial on virtualenvs and options can be found here

Installation

Clone the repo and cd into it:

git clone https://github.com/hluecking1/baselines.git
cd baselines

If using virtualenv, create a new virtualenv and activate it

    virtualenv env --python=python3
    . env/bin/activate

Install baselines package

pip install -e .

Testing the installation

All unit tests in baselines can be run using pytest runner:

pip install pytest
pytest

Training Models

To train a PPO model just run run_mujoco.py with the following arguments:

• --num-timesteps Number of timesteps to train. Default: 1e6
• --save_model path to where the model should be saved
• --env The environment to train the model in (for a full list of environments see: https://gym.openai.com/envs/#mujoco)

Loading an existing model and replay

To load and replay an existing model run play_model with the following arguments:

• --model_path The model to load as a path
• --env The environment to train the model in
• --seed The random seed. Default is 0

Tensorboard

To access Tensorboard graphs and summaries just run: tensorboard --logdir="your_path_to/baselines/saved_models

Subpackages

• A2C
• ACER
• ACKTR
• DDPG
• DQN
• GAIL
• HER
• PPO1 (Multi-CPU using MPI)
• PPO2 (Optimized for GPU)
• TRPO

To cite this repository in publications:

@misc{baselines,
  author = {Dhariwal, Prafulla and Hesse, Christopher and Klimov, Oleg and Nichol, Alex and Plappert, Matthias and Radford, Alec and Schulman, John and Sidor, Szymon and Wu, Yuhuai},
  title = {OpenAI Baselines},
  year = {2017},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/openai/baselines}},
}