How do I choose between simulation structures?
By comparing how many parameters the data actually constrains in each, not just how well each fits.
There is usually more than one simulation structure that can be a viable model for the real world system of interest, and these often have different numbers of parameters.
Choosing between different simulation structures should, in principle, be as easy as selecting the one with the best overall value for the objective. But it is actually more complicated than this because you can have complex or simple models which fit the data equally well.
Let’s say we have two different simulation structures which we are comparing by fitting them to the same dataset. Let’s also say that one of these simulations has lots of parameters and the other has much fewer.
We might interpret the concept of sloppiness to mean that the simulation which has more parameters is always the better choice. This would be wrong without a deeper analysis.
Remember that simulation sloppiness refers to parameters which are not constrained by the data.
What we need to find out is how many parameters are constrained (or learned) by the data in both simulations. In other words; how many parameters does each simulation have which the objective value is sensitive to changing?
The general wisdom is to prefer simulations and models with fewer constrained parameters.
Why? Think about the parameter sensitivity of the quality of fit: if there are more parameters being constrained by the data, this means that the objective value is sensitive to more parameters. So there are more ‘moving parts’ to the simulation which have a greater chance of being misaligned with some new data we haven’t seen yet.
We can think of simulations with fewer constrained parameters (but equal quality of fit) to be more likely to generalise across different datasets from a systems of the same, or similar, type in the real world. This also indicates more fundamental mechanisms are being learned from the data that are potentially transferrable to other systems.
Example: Counting the parameters which are constrained
Let’s imagine two different simulation structures being fit to the same dataset. The simple structure has 3 parameters, while the complex one has 7.
Let’s also imagine that the complex structure always fits the observed data at least as well as the simple one; so the objective value on its own can never tell you which structure to choose.
Switching between the two structures, we can compare how many of their parameters the data actually constrains, and then use this as a way to choose between them.