User:Yangyj21/sandbox

General Optimal control Problems Solver (GOPS) is an open-source reinforcement learning (RL) package that aims to build real-time and high-performance controllers in industrial fields. Developed by iDLab (Intelligent Driving Laboratory), GOPS is built with a highly modular structure and user-friendly framework, enabling the creation of real-time, high-performance controllers for diverse industrial control tasks.

Overview
Solving optimal control problems serves as basic demands of industrial control tasks. Existing methods like model predictive control often suffer from heavy online computational burdens. RL has shown great promise in computer and board games but has yet to be widely adopted in industrial applications due to lacking accessible and high-accuracy solvers. Therefore, iDLab at Tsinghua University has developed General Optimal control Problems Solver (GOPS), an easy-to-use RL solver package that aims to build real-time and high-performance controllers in industrial fields. GOPS is built with a highly modular structure that retains a flexible framework for secondary development. Considering the diversity of industrial control tasks, GOPS also includes a conversion tool that allows for the use of Matlab/Simulink to support environment construction, controller design, and performance validation. To handle large-scale control problems, GOPS can automatically create various serial and parallel trainers by flexibly combining embedded buffers and samplers. It offers a variety of common approximate functions for policy and value functions, including polynomial, multilayer perceptron, convolutional neural network, etc.

Features
GOPS boasts several key features tailored to industrial control applications, which are summarized as follows:


 * GOPS adopts a highly modular configuration that allows for easy secondary development of environments and algorithms, making it accessible for users without professional RL knowledge or programming skills.
 * GOPS supports multiple training modes for handling complex and large-scale problems, including serial and parallel modes for on-policy and off-policy, model-free and model-based, and direct and indirect algorithms. It can handle special requirements from industrial control, such as explicit policies, state constraints, and model uncertainties.
 * Considering the widespread use of Matlab/Simulink in industry control, GOPS offers a convenient conversion tool to support high-performance controller design for Simulink models. This tool enables the transformation of existing Simulink models into GOPS-compatible environments and allows for performance validation and controller deployment by sending the learned policy back to Simulink.

Applications
GOPS has been applied in various industrial control scenarios as evidenced by its adoption in research papers. Examples include: coupled velocity and energy management optimization, travel pattern analysis and demand prediction , design of reward functions in vehicle velocity control , improving freeway merging efficiency , accelerating model predictive path integral , drill boom hole-seeking control , origin-destination ride-hailing demand prediction , etc.

Documentation and Usage
The GOPS package is available on GitHub at Intelligent-Driving-Laboratory/GOPS, where users can access the source code and contribute to its development. Detailed documentation, including installation instructions, usage guidelines, and examples, is provided in the GOPS documentation.