Abstract

Advances in deep reinforcement learning have allowed us to develop artificial agents that master intimidating problems such as Go, Chess, and Starcraft. Yet, we struggle to create agents that display general intelligence. We need new algorithms, but it will be difficult to invent these in a vacuum. We need the right data streams to develop the next generation of algorithms. Progress in game playing was driven by tackling the end-to-end problem and training on the objective of interest. Since much of what we call general intelligence is bound up in human problems and judgements, this suggests we should train agents through grounded interaction with humans in open-ended settings. To lay groundwork for large-scale reinforcement learning with humans-in-the-loop, we trained agents to imitate the interactions of hundreds of human participants in a simulated environment. Aided by self-supervised objectives, our agents learn to respond to humans in natural language, following instructions and answering simple questions. While imperfect, these agents offer a compelling starting point for iterative interactive improvement via reinforcement learning on human feedback.