Negotiating a Future that is not like the Past

ABSTRACT Agent-based models combine data and theory during both development and use of the model. As models have become increasingly data driven, it is easy to start thinking of agent-based modelling as an empirical method, akin to statistical modelling, and reduce the role of theory. We argue that both types of information are important where the past is not a reliable blueprint for the future, which occurs when modelling dynamic complex systems or to explore the implications of change. By balancing theory and data, agent-based modelling is a tool to describe plausible futures, that we call ‘justified stories’. We conclude that this balance must be maintained if agent-based models are to serve as a useful decision support tool for policymakers.


Introduction
We have to make decisions, small ones on a daily basis (e.g. whether to take an umbrella on a walk) and big ones with far-reaching consequences (e.g. decide on actions to curb a pandemic or mitigate climate change). The more we know about the world, the better the decisions we can make. What we would ideally like is full knowledge of what will happen in the future, but we must make do with predictions and forecasting based on partial information extrapolated into the future. This is particularly difficult in a complex system, where outcomes arise from the micro-interactions and small changes can contribute to radically different system trajectories (Johnson et al., 2017).
That partial information comes in two broad types. Theory reflects some agreed understanding of the causal relationships, regular patterns or mechanisms that link aspects of the world in meaningful ways. Data are evidence about the current or past states of relevant aspects of the world. Useful claims about the future combine the two types of information. For example, a short term weather forecast might consist of the pattern that 'tomorrow's temperature is usually pretty similar to today's' and some observations like 'today's temperature is cool' leading to a forecast of cool weather for tomorrow. For most days of the year, this inference will be a pretty good predictor, but it would fail to predict a storm, a sudden heat wave or an extreme cold spell. However, breaks in the pattern are when an accurate prediction would be most useful. We thus employ weather simulations to foresee possible changes in weather with rules that are derived from far more sophisticated theory. This paper argues that both theory and data must be included in models of complex, dynamic worlds. Data-driven models are able to effectively predict in a stable world. But theory must also be embedded in the model if it is to represent how particular phenomena come about. We argue that theory comprises causal assumptions and that we have to continuously refine our understanding of the world to elicit causal mechanisms. Only by embedding such causal mechanisms in our predictions are we able to understand the ways in which the world may change, allowing us to better negotiate the future. We argue further that ABM is a good method for this as it combines causal assumptions with data in an open, transparent, and testable way, allowing us to increase our understanding of the system by modelling the effect of mechanisms to explore different potential futures. This exploration allows the model user to increase their understanding of possible change without expecting a particular future to occur, in contrast to the practice of prediction and forecasting.

Combining data and theory in agent-based modelling
Agent-based modelling (ABM) is a computational method that simulates individual entities making decisions and taking actions according to programmable rules that incorporate the influence of the characteristics of the entity and its social and physical environment (Gilbert, 2019). In the balance between theory and data, ABM starts from the theory. Model rules explicitly represent the theoretical influences and causal connections between individuals, situations, and actions. More simply, the model captures theoretical knowledge about a system and organises the explanation of how that system behaves in a coherent and accessible way.
However, unless the model is a pure thought experiment, the model also represents data from the system being modelled. The first use of data is abductive; the theory that is implemented in model rules is not necessarily a true explanation of the system but is instead the 'best explanation' based on the data (Psillos, 2005). Data is also used inductively in calibration, to fine-tune those causal pathways so that the model outputs are plausible given the assumptions of the model. That is, patterns of data about the real world are reproduced in the model. This may be achieved by constraining parameters so that patterns that are not directly modelled are able to be generated by the complex interactions that are represented (Grimm et al., 1996).
Importantly, model development and calibration highlights where system understanding is missing or is inconsistent with the evidence. While there are many different uses of models (Edmonds et al., 2019), the common thread is generating scenarios from theoretical assumptions. In essence, the model is a tool for thinking through the consequences of the modeller's understanding of the world.
This integration of theory and data is not unique to ABM, it occurs for many modelling methods including other simulation techniques (such as system dynamics) and mathematical models of physical systems. But each method represents the system being modelled in a different way. ABM is a natural choice when modelling complex systems with interacting heterogeneous individuals because the models are composed of interacting individuals that can have different attributes and behaviours (Chattoe-Brown, 2013).

Data-driven modelling
In contrast, statistical modelling starts from the dataset. The model uses a mathematical expression to describe the relationship between variables using an appropriate functional form. The coefficients for that expression are then calculated so as to best match data for those variables (Fielding & Gilbert, 2006). In linear regression, for example, the expression asserts that each unit change in one variable is associated with a constant change in another variable and the coefficient is the value of that constant change. The modeller's task is then to find the values of the coefficients so that the fully specified mathematical expression best summarises the dataset.
After fitting, the model and the dataset are compared to (1) test whether the assumptions of the functional form are adequately satisfied, and (2) determine whether all the elements of the model are needed to effectively summarise the dataset as compared to random variation. Statistical practice tightly interweaves fitting the model and using it to test hypotheses about the world from which the data are drawn. To the extent that the pre-determined mathematical expression represents some theoretical relationship between variables, a statistical model represents theory. However, the objective is to accurately summarise the dataset rather than accurately represent theoretical relationships. If a statistical model adequately captures variation in the dataset, then it is a good model.
However, this focus on the dataset limits the potential for the model to describe a world that is different from the one that generated the dataset. The difficulty arises because there is no suitable dataset to assess the fit of the model. Statisticians are therefore very cautious about extrapolating a model to a different world where it is not known whether the explanatory variables are sufficient to express the relationships of interest.
Machine learning relies even more on the data. Instead of simply fitting coefficients to predetermined relationships between variables, machine learning algorithms also extract structure and patterns to develop rules about those relationships. As the derived relationships are not constrained by theoretical assumptions about what they 'should' look like, the practice in machine learning is to hold back a portion of the dataset and measure the extent to which the relationships found in the training data also exist in the remainder of the dataset (Hastie et al., 2009).
Some rules are imposed through the algorithms that are used to detect these patterns. For example, the k-nearest neighbours algorithm assigns a class to a new data point based on the known classes of the nearest points that are already in the dataset. This requires a rule about how to measure distance and also a choice of how many nearest points to count (Burkov, 2019). However, these rules do not represent expected causal relationships in the system being modelled and therefore do not embed any theory in the model.
Model accuracy can only be assessed by keeping aside part of the original data as the model is not able to distinguish between meaningful and spurious associations found by the machine learning algorithm. Similarly, there is no capacity for a generated model to make claims about a different world because there is no way to identify which associations would be valid in the different world.
Distinguishing meaningful associations is even more difficult in a complex world because what is observable is the effect of micro-interactions, which are unobservable directly or at least involve a large number of dependents. The associations between variables in the observable system behaviour easily break down through small changes in the micro-interactions. Statistical methods thus analyse the system at the wrong level of aggregation, limiting their usability within complexity social science and policy research.

Theory and data in policy and planning
The different contributions of theory and data also play out in debates about how to make policy decisions. In this domain, data oriented approaches are located in the sprawling field of 'evidence based policy'. The idea behind evidence-based policy is to evaluate and monitor policy interventions closely to understand 'what works', i.e. which interventions lead to which outcomes (Davies & Nutley, 2000). Evidence-based policy approaches are focussed on reducing ideological bias of policy, with methods and approaches according to strict protocols, allowing for transparency, comparability, and measurement. Particularly elevated approaches are statistical and experimental methods that try to reduce bias by limiting researcher involvement through highly transparent, standardised ways of executing protocols (e.g. statistical modelling, randomised controlled trials (RCT) and systematic reviews).
A randomised controlled trial defines an intervention group and a control group and properly randomising the members of these groups are key to the validity of an RCT. In the attempt to isolate effects of an intervention, the groups should be as similar as possible, and the intervention applied in a tightly controlled manner, to ensure that it is the intervention, rather than some other exogenous effect, bringing about the outcome. The goal is to repeatedly generate a particular outcome and thereby establish a particular causal connection through induction. Theory enters through the design of the intervention to be trialled, but further analysis is focussed on whether it works rather than why it did or did not work. An additional method employed to get to the core of 'what works' is the systematic review, a desk-based method that reviews existing research studies, trials, and interventions to distil findings even further (Higgins et al., 2021).
RCT are difficult in the social world and many known challenges exist, such as ethical problems, practical problems like separating the population into groups and spillover effects. Beyond these implementation issues, however, there is increasing doubt about whether findings from RCT can be generalised over new contexts and situations (Cartwright & Hardie, 2012). As a result, these methods have been heavily criticised in their applicability to real-world decision-making (Pawson, 2006).
Cartwright and Hardie (2012) produce a cutting critique of RCT. Their focus is on the complexity of the social world, the potential for small changes to produce large effects and the importance of context. They argue that RCT let us know that a policy intervention had a particular effect in the trial situation, but the inferences from 'worked' to 'work' to 'will work' -are not just a matter of grammatical detail. To move from one to the other 'requires hard intellectual and practical effort.' (p. ix). Cartwright and Hardie (2012) then develop the argument that we cannot get causes out of a method without making causal assumptions to start with -or 'no causes in, no causes out' (p. 38), discussing different methods, notably econometric modelling, process tracing and Bayes nets, for the assessment of causal connections in the real world. These methods let us understand how things work rather than just what. Pawson (2006) is another vocal critic of existing evidence-based policy research. His critique focuses on systematic reviews, stating that extracting comparable evidence from a large number of studies strips interventions of all context specificity and, in Pawson's mind, makes the evidence useless. In his own development of realist evaluation (Pawson & Tilley, 1997) the question of what works becomes 'what works for whom and in what circumstances', a configuration of context, mechanism and outcomes. Only by keeping the contextual aspects of an intervention and then trying to find out the underlying mechanisms can we hope to understand the consequences of future policy interventions. Like Cartwright and Hardie (2012), Pawson highlights the importance of context. Cartwright and Hardie locate an intervention within a context, recognising that elements of the context may contribute to the outcome of the invention. Pawson goes further and explains the influence of context through a constructivist lens, developing the idea that the outcomes of policy interventions result from people's perceptions, interpretations, and (re)-actions to the interventions.
A range of approaches and methods have been developed to go beyond strictly experimental methods, such as theory-based approaches (Chen, 1990;Weiss, 1997), realist approaches (Pawson & Tilley, 1997), comparative approaches (Rihoux & Ragin, 2008) and simulation approaches (Gilbert & Troitzsch, 2005). There is also recognition that policy evaluation requires complexity-aware methods that are able to deal with continuous change and adaptation (Bicket et al., 2020). Whilst very different in their methods and techniques, what these approaches have in common is that they explicitly acknowledge the need to partner data with theory to incorporate context into the analysis.

Exploring (Rather than Predicting) the Future
Greenhalgh (2020) develops an argument of how poor our existing methods are in dealing with the Covid pandemic. Questions she raises are for example, 'Were care home deaths avoidable?' or 'What role does health system resilience play in controlling the pandemic?'. She draws out the need for overcoming methodological rigidity as we are trying to make rapid decisions in a complex world in which economic, health, and social aspects are intimately intertwined.
Whilst not tackling those exact questions, ABM has been used during the Covid pandemic to model consequences of particular policy interventions (Ghorbani et al., 2020;Giabbanelli et al., 2020;Lorig et al., 2021). One example of this approach is JuSt-Social (Badham et al., 2021), an agent-based model developed to assist planners in North East England to understand the potential impact of COVID-19 on health and social service demand, particularly hospital bed requirements.
JuSt-Social has two intertwined layers, one for epidemic spread and one for policy interventions. The epidemic layer has simulated people randomly moving in an abstract space and potential transmission if an infectious person is close to a susceptible person. Theory is represented by the available epidemic states and the transmission from infectious people to susceptible people. Once exposed, the simulated person passes through various epidemic states (potentially including hospitalisation) until they either recover or die. The main contribution of data to this layer is to provide the probability distributions for sequence of states and the duration in each state.
The policy intervention layer allows the JuSt-Social user to construct scenarios that represent policy options such as social distancing by manipulating the activity levels of the simulated people through time. A key advantage of ABM over population level epidemic models is that the activity level is a characteristic of the individual. For example, some proportion of the simulated people are able to be shielded in JuSt-Social by reducing their activity to near-zero, while continuing higher activity levels for the majority of the people. The operation of the policy layer is entirely theory driven, with JuSt-Social rules making the same assumptions as those in the policy debate -that reducing interactions will reduce transmission. Data contribute to this layer by adjusting settings that represent changing government restrictions until the JuSt-Social output is similar to the pattern of hospital admissions. These constructed scenarios are then continued to a point ahead in time to estimate future service demand.
In creating the scenarios and calibrating them to historical data, the JuSt-Social user learns how the parameter settings relate to potential real-world policy options. Thus, JuSt-Social not only projects existing data forward but also allows for potential changes to existing trends to be explored through simulation experiments.
As demonstrated through this extended example, ABM is therefore able to deal with a changing world to the extent that the modeller's knowledge contains the potential implications of those changes. The model is a tool for synthesising and extrapolating that knowledge. By representing the causal connections in a computational model, the assumptions are open, transparent, and surveyable. But having gone through the modelling process, the implausible causal pathways have been eliminated and the representation of the world is of higher fidelity than it would have been without the model.
ABM are sometimes tasked with prediction just like other simulation models (e.g. weather forecasts). But high fidelity is not sufficient to allow ABM to be used for prediction. Prediction carries the expectation of the world ending up in a particular future state. However, given the problem of equifinality (many paths to the same outcome) in complex systems, high fidelity does not translate into having identified the correct causal path when the model is extrapolated.
What ABM allows us to do is explore potential futures (via different paths) by telling 'justified stories' . The term 'justified stories' intentionally combines the roles of theory and data. Theory allows the model to describe internally coherent futures, or stories. Those stories are justified through empirical connection between model output and empirical data. The model output therefore represents futures that are plausible given the information available but are not necessarily expected to occur. By generating justified stories, models help us to understand and explore likely and unlikely developments of the system. As these justified stories are generated within the model based on the theoretical understanding of interactions and heterogeneity, they are able to represent the dynamics that arise in complex systems such as path dependency and nonlinearity. It is through the experience of complexity in-silico that models can support decision makers to understand the potential implications of their policy decisions, given their understanding of the real-world mechanisms and consistency with available evidence.

Conclusion
We need to understand the world in order to make good decisions. We have become ever more focused on gathering evidence to inform our decisions. In particular, trends that extrapolate past facts into likely future developments are used as a basis of decisions. As long as the future is 'like the past' it is reasonable to base decisions on this evidence. However, such extrapolation is not reliable when faced with either complexity or change (or both); evidence that is currently judged as 'best' is less useful as it ignores context and process. It is in these situations that we need other methods to try to understand the future, methods that will help us understand the possible futures in the face of change, possibly radical change. We are left somewhere between the desire to predict and the demand to stand back and not make assumptions. We cannot have it both ways. This is what Cartwright and Hardie (2012, ix) call the 'hard intellectual and practical effort' needed to extrapolate from the what is or what worked to the what might be the case or might work in future.
An ABM can contribute to this hard intellectual effort, producing a transparent and sound representation of theoretical assumptions calibrated with available data, it can help us to understand change and its possible and plausible consequences. It is, however, important to cut the tie between the model doing forecasting and the model being used to improve forecasting. ABM are not there to forecast, we do not expect that future to occur. If we want to forecast, we are much better off with methods that extrapolate trends from past to future. ABM are however good for improving forecasting -to help us understand how different trends might interact, how changes in one part of the system might influence currently stable domains and how feedback loops might bring about radical change. Justified stories generated by an ABM combine the internal coherence arising from mechanisms and consistency with external evidence. All of these help us to understand the plausible effects of the causal undercurrents that bring about change and will help to improve our decisionmaking, not through forecasting but rather through increased understanding of the workings of the system. By exploring plausible futures, ABM becomes a decision support tool that can help us to make better decisions in a complex and dynamic world.

Disclosure statement
No potential conflict of interest was reported by the author(s).

Funding
This work was supported by the MRC/CSO Social and Public Health Sciences Unit core grant (MRC grants MR/ S037578/1 and MC_UU_00022/5, and Scotland Chief Scientist Office Grant SPHSU 20) and the UK Prevention Research Partnership (MR/S037578/1). The research was also supported by the ESRC Project Centre for Evaluation of Complexity Across the Nexus (ES/S007024/).

Notes on contributors
Corinna Elsenbroich is a Reader in Computational Modelling at Glasgow University (MRC/CSO Social & Public Health Sciences Unit). Corinna is particularly interested in methodological and epistemological aspects of novel methods, in particular computational methods such as agent-based modelling and social simulation, and has published on aspects of ontology, explanatory power and context validity in modelling.

Jennifer Badham is Assistant Professor in Social Data
Science at Durham University. Originally trained as a mathematician, she became interested in complex systems methods while working for government in the health sector. Much of her research concerns how the transmission of ideas, disease or behaviour is shaped by social structure.