Learning optimal policies with the simulation itself
Learning using the simulation itself (evolutionary strategies, monte carlo tree search and touch on how these are different to Deep RL but still in the same family of sample-driven - which depend on simulations existing)
If we want to learn parameters which correspond to optimal simulation trajectories, we first need to specify what ‘optimal’ means.
We do this by defining an objective whose maximum/minimum possible value will be achieved when our goal is met.
For instance, we may define some logic in a state partition iteration of the simulation which replicates taking ‘actions’ in the real world. This logic can depend on the simulation parameters so that the latter encodes the behaviour quantitatively.
Given this setup, a very common goal of interest is then in finding the best actions to take; which is analogous to optimising the parameters of the action-taking state partition iteration. We will refer to these parameters as ‘policy parameters’.
But what should be use as an objective?
The ‘discounted future reward’ is a quantity we can specify that a simulation trajectory will have accumulated into the future, accounting for increasing distance into the future by ‘discounting’ it gradually with a weighting.
We are using this concept of discounted future reward in the same way that it is used in reinforcement learning.
The idea is that, as you go further into the future, the importance of the reward you have accumulated by then is increasingly irrelevant to actions you might take at the present moment.
Example: Optimising with evolutionary strategies
The evolutionary strategies algorithm can be applied to search future simulation trajectories to find the best set of policy parameters needed to achieve some discounted future reward.
This algorithm relies on sorting the sampled simulation trajectories according to their discounted future rewards and then using the top fraction of these to update the best known policy parameters (and the variance around them) after each timestep.
