Preparing a simulation for action-taking

The two kinds of sensitivity

Data Scientists will often ask the following question: if I change this model or its parameters, how will my prediction change?

We discussed the same idea for simulations in this post. So let’s give a name to this idea, and call it ‘prediction sensitivity’.

But the question we were asking at the end of the last post was more aligned with: if I change this model (simulation) or its parameters, how will my taken action change?

Let’s call this one ‘action-taking sensitivity’.

When we change the parameters of a simulation, its iteration at the same point in time may produce different outcomes, depending on the system state and its history (which may also have changed due to previous iterations in time being different).

Similarly, when we change the simulation itself, its iteration at the same point in time may also produce different outcomes.

In both cases, the simulated outcomes help us to determine which action to take, but in order to do this we need to define what outcomes we are targeting in the first place.

Explaining why the simulation predicts what it does

Cover these topics: - past-looking simulation graph analysis - forward-looking counterfactual analysis: simulation differencing and why SHAP is better - forward-looking counterfactual analysis: simulation SHAP analysis

Inferring which simulation to use

Cover these topics: - Bayesian model selection applied to action-taking is the same as doing causal inference