The increasing frequency and impacts of natural disasters appear to be surprising infrastructure systems and causing unexpected damages. This is problematic given the importance of critical services like electricity, water, mobility, food, and communications to both civil society and military operations. In this work, we study decision-making under uncertainty and whether standard reinforcement learning techniques are capable of handling surprise. First, we provide a formal definition of surprise based on the cognitive and emotional response of people embedded in infrastructure operations and management. Then, we overview a simple approach to model this decision context with a network flow model representing system operations and stochastic failures driven by infrastructure condition. We implement q-learning techniques to an infrastructure repair and replace game to determine how well a trained neural network (NN) performs alongside simple heuristics for infrastructure failure response. Finally, we discuss how these results provide insight into (1) what the NN might be learning about network state and operations; and (2) how this may inform training and games for real-world systems.