You're Invited:Meet the Socket Team at BlackHat and DEF CON in Las Vegas, Aug 4-6.RSVP
Socket
Book a DemoInstallSign in
Socket

gbrl

Package Overview
Dependencies
Maintainers
1
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

gbrl

Gradient Boosted Trees for RL

1.1.0
pipPyPI
Maintainers
1

Gradient Boosting Reinforcement Learning (GBRL)

GBRL is a Python-based Gradient Boosting Trees (GBT) library, similar to popular packages such as XGBoost, CatBoost, but specifically designed and optimized for reinforcement learning (RL). GBRL is implemented in C++/CUDA aimed to seamlessly integrate within popular RL libraries.

License PyPI version

Overview

GBRL adapts the power of Gradient Boosting Trees to the unique challenges of RL environments, including non-stationarity and the absence of predefined targets. The following diagram illustrates how GBRL uses gradient boosting trees in RL:

GBRL Diagram

GBRL features a shared tree-based structure for policy and value functions, significantly reducing memory and computational overhead, enabling it to tackle complex, high-dimensional RL problems.

Key Features:

  • GBT Tailored for RL: GBRL adapts the power of Gradient Boosting Trees to the unique challenges of RL environments, including non-stationarity and the absence of predefined targets.
  • Optimized Actor-Critic Architecture: GBRL features a shared tree-based structure for policy and value functions. This significantly reduces memory and computational overhead, enabling it to tackle complex, high-dimensional RL problems.
  • Hardware Acceleration: GBRL leverages CUDA for hardware-accelerated computation, ensuring efficiency and speed.
  • Seamless Integration: GBRL is designed for easy integration with popular RL libraries. We implemented GBT-based actor-critic algorithm implementations (A2C, PPO, and AWR) in stable_baselines3 GBRL_SB3.

Performance

The following results, obtained using the GBRL_SB3 repository, demonstrate the performance of PPO with GBRL compared to neural-networks across various scenarios and environments:

PPO GBRL results in stable_baselines3

Getting started

Dependencies

  • Python 3.9 or higher

Installation

GBRL provides pre-compiled binaries for easy installation. Choose one of the following options:

CPU-only installation (default):
pip install gbrl

GPU-enabled installation (requires CUDA 12 runtime libraries):
pip install gbrl-gpu

For further installation details and dependencies see the documentation.

Usage Example

For a detailed usage example, see tutorial.ipynb

Current Supported Features

Tree Fitting

  • Greedy (Depth-wise) tree building - (CPU/GPU)
  • Oblivious (Symmetric) tree building - (CPU/GPU)
  • L2 split score - (CPU/GPU)
  • Cosine split score - (CPU/GPU)
  • Uniform based candidate generation - (CPU/GPU)
  • Quantile based candidate generation - (CPU/GPU)
  • Supervised learning fitting / Multi-iteration fitting - (CPU/GPU)
    • MultiRMSE loss (only)
  • Categorical inputs
  • Input feature weights - (CPU/GPU)

GBT Inference

  • SGD optimizer - (CPU/GPU)
  • ADAM optimizer - (CPU only)
  • Control Variates (gradient variance reduction technique) - (CPU only)
  • Shared Tree for policy and value function - (CPU/GPU)
  • Linear and constant learning rate scheduler - (CPU/GPU only constant)
  • Support for up to two different optimizers (e.g, policy/value) - **(CPU/GPU if both are SGD)
  • SHAP value calculation

Documentation

For comprehensive documentation, visit the GBRL documentation.

Citation

@article{gbrl,
  title={Gradient Boosting Reinforcement Learning},
  author={Benjamin Fuhrer, Chen Tessler, Gal Dalal},
  year={2024},
  eprint={2407.08250},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2407.08250}, 
}

Licenses

Copyright © 2024, NVIDIA Corporation. All rights reserved.

This work is made available under the NVIDIA Source Code License-NC. Click here. to view a copy of this license.

FAQs

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts