The source address selection was random, and sometimes we picked a source address that the container didn't have a route to. Adding a default route fixes that!