Autonomous systems have long held the promise of replacing or assisting humans in a wide range of applications. Today autonomous systems operate effectively in challenging domains such as self-diving cars and trucks, search and rescue, and sidewalk navigation and delivery. However the classic perceive, plan, execute control loop limits the utility and intelligence of systems by requiring robots to make complete observations of the area they are planning over. As opposed to humans, who can make common-sense inference of geometry without direct observation, robots employ a suite of sensors or pre-loaded maps to generate complete observations of their operational environments. Without access to full map information operation can be slow and unintuitive. In this paper we aim to supplement existing robot sensing capabilities by leveraging recent advances diffusion models and generative AI to produce realistic predictions of unobserved space in running occupancy maps.

In this paper we present a transformer-based model for generative occupancy prediction and some of the restrictions of similar models for occupancy mapping. Then we adopt diffusion models for a means of realistic occupancy prediction and show that our model SceneSense generates better representations of local occupancy than just the running occupancy map built from sensor measurements. After making key modifications to the original model, we deploy SceneSense onboard a real-world robotic platform and show that SceneSense can be a “drop-in” improvement for existing planning and exploration stacks. We show that the SceneSense enhanced map increased both the rate of exploration and the consistency of exploration when compared to the same planning and explore stack informed by just sensor measurements.