How do evolutionary strategies find a policy?
The evolutionary strategies algorithm can be applied to search future simulation trajectories to find the best set of policy parameters needed to achieve some discounted future reward.
This algorithm relies on sorting the sampled simulation trajectories according to their discounted future rewards and then using the top fraction of these to update the best known policy parameters (and the variance around them) after each timestep.
Sampled simulation trajectories
Optimises best fraction of policy parameters (blue lines) towards the target. Grey lines are the others.
Best discounted future reward
References and further reading
Hansen, Nikolaus, and Andreas Ostermeier. 2001. “Completely Derandomized Self-Adaptation in Evolution Strategies.” Evolutionary Computation 9 (2): 159–95. https://doi.org/10.1162/106365601750190398.
Hardwick, Robert J. 2026a. “How do I automate taking actions with simulations?” umbralcalculations, September. https://umbralcalc.github.io/posts/how_do_i_automate_taking_actions_with_simulations.html.
Hardwick, Robert J. 2026b. “What is discounted future reward?” umbralcalculations, November. https://umbralcalc.github.io/posts/what_is_discounted_future_reward.html.
Salimans, Tim, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv preprint. https://arxiv.org/abs/1703.03864.
Wikipedia contributors. n.d. “Evolution Strategy.” Wikipedia. https://en.wikipedia.org/wiki/Evolution_strategy.
Questions whose answers cite this one