Reinforcement Learning provides a general formalism to address control problems, but has yet to be widely used in the real-world. We have been investigating using reinforcement learning to optimize drinking water treatment, directly on the physical system, and have naturally run into many technical challenges. In this talk, I will discuss one such important question for real-world deployment: selecting hyperparameters offline, for a continual learning agent. Before a learning agent can be deployed, its hyperparameters need to be specified. I will outline one potential solution, under the setting where an offline batch of data is available from a previous controller in the system, and discuss key open challenges.