Reinforcement Learning for Spacecraft Planning and Scheduling

Herrmann, Adam Paul

Graduate Thesis Or Dissertation

Reinforcement Learning for Spacecraft Planning and Scheduling Public Deposited

Analytics

Citations

Citeable URL: https://scholar.colorado.edu/concern/graduate_thesis_or_dissertations/xs55md76n

Abstract

The coming decades of space exploration will require a massive increase in spacecraft autonomy due to an explosion in the number of Earth-orbiting satellites, which will tax current operations infrastructure and capabilities, and the over subscription of deep space network services for deep space and cislunar missions.

This dissertation investigates the use of reinforcement learning (RL) for spacecraft planning and scheduling, which is the process by which the sequence of tasks a spacecraft must execute to achieve its objectives is computed. RL is a machine learning technique that allows for autonomous decision-making agents to learn an optimal policy that maps situations to actions to maximize a numerical reward function. RL offers closed-loop decision-making, fast execution times after training, and few constraints on problem representation.

This dissertation first investigates the application of RL to the single satellite Earth-observing scheduling problem, taking into consideration various spacecraft resources, data downlink, and agile targeting of surface targets. RL-based formulations and methods are shown to meet or exceed the performance of genetic algorithms and generalize over the state space. Then, scalable Earth-observing constellation operations utilizing single-agent RL policies are investigated. Various communication assumptions and target distribution methods are explored. A novel fully decentralized RL-based architecture that can automatically adjust to a new constellation design or new distribution of surface targets is developed and shown to be more performant than a centralized architecture that relies on an integer program for target distribution. Finally, RL is applied to the problem of small body science operations, demonstrating that RL is capable of autonomously managing maneuvers, navigation updates, resources, and science operations to accomplish a mission.

Creator