If validation is limited to these latter activities, one can be misled by agreement with preconceived notions or with models that are based on a set of commonly held and unverified assumptions. In addition: there will be little or no support for important feedback loops to help indicate areas in the model in need of improvement, there will be little indication of the quality of the model for predicting various outputs of interest, and it will be impossible to construct hypothesis tests that indicate whether discrepancies with the results of other models or with field results are due to natural variation or to real differences in the model s.
Therefore, face validity must be augmented with more rigorous forms of model validation. In the last years a number of statistical advances have been made that are relevant to the practice of model validation; we have not seen evidence of their use in the validation of constructive simulation models used for defense testing.
We note five of the advances that should be considered:. Morgan and Henrion , McKay , and others have identified techniques for carrying out an uncertainty analysis 4 in order to assist external validation. One-variable-at-a-time sensitivity analysis assumes a linear response surface, is inefficient in its use of model runs, and is too narrowly focused on a central scenario.
These deficiencies can be addressed by using inputs produced by Latin Hypercube sampling McKay et al. To help analyze the resulting paired input-output data set, McKay's ANOVA decomposition of outputs from models run on inputs selected by Latin Hypercube sampling is useful for identifying which outputs are sensitive to which inputs.
The response surface of these input-output vector pairs can be summarized using various non-parametric regression techniques, such as multivariate adaptive regression splines Friedman, For a single output, there are techniques that can identify a small collection of important inputs from a larger collection of candidate inputs.
These techniques can be very helpful in simplifying an uncertainty analysis, a sensitivity analysis, and even an external validation, since it indicates which variables are the crucial ones to vary see Cook, ; Morris, ; McKay, ; and others. One formal approach to the design and analysis of computer experiments has been developed by Sacks et al. Identifying outliers to patterns shown by the great majority of inputs to a constructive simulation can be used to better understand regions of the input output space in which the behavior of the simulation changes qualitatively see, e.
The developers of a constructive simulation often make fairly arbitrary choices about model form. Citro and Hanushek , Chatfield , Draper , and others offer ways of addressing this problem of misspecification of the relationship between inputs and outputs in computer and statistical models, wherein the simulation is less precise than one measures by using the usual "analysis of variance" techniques.
Some of the above ideas may not turn out to be directly applicable to defense models, but the broad collection of techniques being developed to analyze non-military simulations are likely to be relevant.
Given the importance of operational testing, testing personnel should be familiar with this literature to determine its value in the validation of constructive simulations. As noted above, a sensitivity analysis is the study of the impact of changes on model outputs from changes in model inputs and assumptions. An uncertainty analysis is the attempt to measure the total variation in model outputs due to quantified uncertainty in model inputs and assumptions and the assessment of which inputs contribute more or less to total uncertainty.
In addition to model validation, a careful analysis of the assumptions used in developing constructive simulation models is a necessary condition for determining the value of the simulation.
Beyond the usual documentation, which for complicated models can be fairly extensive, an "executive summary" of key assumptions used in the simulation model should be provided to experts to help them determine their reasonableness and therefore the utility of the simulation.
A full history of model development, especially any modification of model parameters and their justification, should also be made available to those with the responsibility for accrediting a model for use in operational testing. In model-test-model, a model is developed, a number of operational test runs are carried out, and the model is modified by adjusting parameters so that it is more in agreement with the operational test results.
Such external validation on the basis of operational use is extremely important in informing simulation models used to augment operational testing. However, there is an important difference one we suspect is not always well understood by the test community between comparing simulation outputs with test results and using test results to adjust a simulation.
Many complex simulations involve a large number of "free" parameters—those that can be set to different values by the analyst running the simulation. In model-test-model some of these parameters can be adjusted to improve the correspondence of simulation outputs with the particular operational test results with which they are being compared.
When the number of free parameters is large in relation to the amount of available operational test data, close correspondence between a "tuned'' simulation and operational results does not necessarily imply that the simulation would be a good predictor in any scenarios differing from those used to tune it.
A large literature is devoted to this problem, known as overfitting. An alternative that would have real advantage would be "model-test-model-test," in which the final test step, using scenarios outside of the "fitted" ones, would provide validation of the version of the model produced after tuning and would therefore be a guard against overfitting.
If there was interest in the model being finalized before any operational testing was performed, this would be an additional reason for developmental testing to incorporate various operationally realistic aspects.
Overfitting is said to occur for a model and data set combination when a simple version of the model selected from a model hierarchy, formed by setting some parameters to fixed values is superior in predictive performance to a more complicated version of the model formed by estimating these parameters from the data set.
For some types of statistical models, there are commonly accepted measures of the degree of overfitting. An example is the Cp statistic for multiple regression models: a model with high Cp could be defined as being overfit. Recommendation 9. The panel reviewed several documents that describe the process used to decide whether to use a simulation model to augment an operational test.
There are differences across the services, but the general approach is referred to as verification, validation, and accreditation. Verification is "the process of determining that model implementation accurately represents the developer's conceptual description and specifications" U. Department of Defense, a. For constructive simulations, verification means that the computer code is a proper representation of what the software developer intended; the related software testing issues are discussed in Chapter 8.
Validation is "the process of determining a the manner and degree to which a model is an accurate representation of the real-world from the perspective of the intended uses of the model, and b the confidence that should be placed on this assessment" U. Accreditation is ''the official certification that a model or simulation is acceptable for use for a specific purpose" U.
The panel supports the general goals of verification, validation, and accreditation and the emphasis on verification and validation and the need for formal approval, that is, accreditation, of a simulation model for use in operational testing.
Given the crucial importance of model validation in deciding the utility of a simulation for use in operational test, it is surprising that the constituent parts of a comprehensive validation are not provided in the directives concerning verification, validation, and accreditation. A statistical perspective is almost entirely absent in these directives. For example, there is no discussion of what it means to demonstrate that the output from a simulation is "close" to results from an operational test.
It is not clear what guidelines model developers or testers use to decide how to validate their simulations for this purpose and how accrediters decide that a validation is sufficiently complete and that the results support use of the simulation. Model validation cannot be algorithmically described, which may be one reason for the lack of specific instruction in the directives.
A test manager would greatly benefit from examples, advice on what has worked in the past, what pitfalls to avoid, and most importantly, specific requirements as to what constitutes a comprehensive validation. This situation is similar to that described in Chapter 1 , regarding the statistical training of those in charge of test planning and evaluation.
Model validation has an extensive literature, in a variety of disciplines, including statistics and operations research, much of it quite technical, on how to demonstrate that a computer model is an acceptable representation of the system of interest for a. Operational test managers need to become familiar with the general techniques represented in this literature, and have access to experts as needed.
We suggest, then, a set of four activities that can jointly form a comprehensive process of validation: 1 justification of model form, 2 an external validation, 3 an uncertainty analysis including the contribution from model misspecification or alternative specifications, and 4 a thorough sensitivity analysis. All important assumptions should be explicitly communicated to those in a position to evaluate their merit. This could be done in the "executive summary" described above.
A model's outputs should be compared with operational experience. The scenarios chosen for external validation of a model must be selected so that the model is tested under extreme as well as typical conditions. The need to compare the simulation with operational experience raises a serious problem for simulations used in operational test design, but it can be overcome by using operationally relevant developmental test results.
Although external validation can be expensive, the number of replications should be decided based on a cost-benefit analysis see the discussion in Chapter 5 on "how much testing is enough".
External validation is a uniquely valuable method for obtaining information about a simulation model's validity for use in operational testing, and is vital for accreditation. An indication of the uncertainty in model outputs as a function of uncertainty in model inputs, including uncertainty due to model form, should be produced. This activity can be extremely complicated, and what is feasible today may be somewhat crude, but DoD experience at this will improve as it is attempted for more models.
In addition, exploration of alternative model forms will have benefits in providing further understanding of the advantages and limitations of the current model and in suggesting modifications of its current form. An analysis of which inputs importantly affect which outputs, and the direction of the effect, should be carried out and evaluated by those with knowledge of the system being developed.
The literature cited above suggests a number of methods for carrying out a comprehensive sensitivity analysis. It will often be necessary to carry out these steps on the basis of a reduced set of "important" inputs: whatever process is used to focus the analysis on a smaller number of inputs should be described.
There are tutorials that are provided at conferences, and other settings, and excellent reports in the DoD community e. A description of any methods used to reduce the number of inputs under analysis should be included in each of the steps.
Models and simulations used for operational testing and evaluation must be archived and fully documented, including the objective of the use of the simulation and the results of the validation.
The purpose of a simulation is a crucial factor in validation. For some purposes, the simulation only needs to be weakly predictive, such as being able to rank scenarios by their stress on a system, rather than to predict actual performance. For other purposes, a simulation needs to be strongly predictive. Experience should help indicate, over time, which purposes require what degree and what type of predictive accuracy. Models and simulations are often written in a general form so that they will have wide applicability for a variety of related systems.
An example is a missile fly-out model, which might be used for a variety of missile systems. A model that has been used previously is often referred to as a legacy model. In an effort to reduce the costs of simulation, legacy models are sometimes used to represent new systems, based on a complete validation for a similar system.
Done to avoid costly development of a de novo simulation, this use of a legacy model presents validation challenges. In particular, new systems by definition have new features. Balcan et al. A first layer, where the population and mobility are implemented, allows for the partitioning of the world into geographical regions—i.
This partition defines a second layer, the sub-population network, where the inter-connection represents the fluxes of individuals via transportation infrastructures and general mobility patterns— KS 2 implements M 2. Finally, and superimposed onto this layer, is the epidemic layer, that defines the disease dynamic inside each sub-population— KS 3 implements M 3 — Balcan et al.
In the study conducted by Ajelli et al. The sub-population network uses geographic census data— IM 1 —and the mobility layers obtain data from different databases, including the International Air Transport Association database consisting in a list of airports worldwide connected by direct flights i.
The example of the two computer simulations give us a good sense of the interconnectivity between different M i , their implementation as KS i , and the final integration into a fully functional SM via IM k. Let us now summarize our findings. Each KS i , in turn, implements a multiplicity of M i in such a way that is computationally tractable.
Since most of the structures found in an M i can be reconstructed in terms of a given programming language, it is reasonable to take that KS i is an accurate implementation of such M i. The process of information hiding discussed earlier warrants the implementation. The previous sections made an effort to present and discuss the architecture of simulation models, and to show how this structure better accounts for the current practice of simulation modeling.
I argued for a rich and complex structure that builds from different sources ranging from mathematical models to sizable external databases. I now argue how, by means of taking SM as units of analysis in their own right, we could give content to the idea that computer simulations introduce new challenges to the philosophy of science.
To make this visible, I briefly revisit two recent discussions on scientific explanation for computer simulations and show why the whole explanatory enterprise fails when taking simulations as the simple implementation of mathematical models. The first attempts to elaborate on the logic of scientific explanation for computer simulations is found in the works of Krohs and Weirich On occasion, the M includes equations that are too complex for humans to solve analytically, and therefore the representation of RW p cannot be guaranteed.
This means that an explanation of RW p cannot take effect. Computer simulations amend this by finding the set of solutions to M , and thus, claim Krohs and Weirich, reestablish the representational relation to RW p. Thus understood, computer simulations play the instrumental role of solving M , but not the explanatory role of accounting for RW p , which ultimately is left to M.
It is in the sense of being a mere instrument that computer simulations do not pass the mark of what constitute a philosophical issue. Opposing this view is my own work , where I argue that the logic of explanation for computer simulations must include the SM into the explanatory relation. To this end, it must be first shown that the SM is in a better position to explain than any M i.
This is accomplished precisely by noticing that a given SM is a collection of models, relations, and external modules that exceed any one M i. Furthermore, I call attention to the fact that researchers are interested in explaining the results of their simulations with the purpose of understanding real-world phenomena, but not to explain the latter directly.
Decoupling explanation from understanding allows me to fully circle the logic of explanation for computer simulations. A simple example will help to contrast the two approaches. Consider a simulation of a satellite orbiting a planet under tidal stress, as presented by Woolfson and Pert b.
The visualization of the results shows a series of spikes that represent the tidal stress that the satellite undergoes during orbiting. Researchers are naturally inclined to first explain the behavior of the simulated satellite, such as the spikes i.
In the context of this example, I call attention to the fact that the visualization shows the spikes having a steadily downwards trend, an effect that is the result of a truncation error during the computation of SM. I conclude that, if an M exogenous to the simulation were to be used for explanatory purposes v.
I then show how, by reconstructing the SM as the explanans , researchers are able to account for this trend and thus understand the overall behavior of the satellite.
Motivated by these results, I offer an in-depth account of explanation for computer simulations that, as a matter of fact, includes the SM in the explanatory relation. The fundamental difference between my approach, on the one hand, and Krohs and Weirich, on the other, is that we understand computer simulations in radically different ways. I build my claims around the idea that the SM is a rich and complex model, with methodological and epistemological value in and by itself.
Krohs and Weirich, echoing authors such as Hartmann , Frigg and Reiss , Parker , and others, mingle the analysis of simulations with the analysis of other forms of mathematical models. Now, if these claims are correct, we find ourselves with a handful of genuine philosophical issues attached to the logic of scientific explanation for computer simulations.
On the one hand, and taking my approach seriously, further arguments need to be provided on how and to what extent researchers are able to access and reconstruct the SM for the purpose of explaining. Although I offer a detailed reconstruction of the SM as the explanans for the simulation of the satellite, I also notice that cognitively accessing every function, procedure, and data-type in the SM is challenging. In this respect, the right levels of description for the explanans is a novel issue that needs to be discussed in detail if this account is to be successful.
On the other hand, a central philosophical question that seeks an answer is how, by explaining the results of a computer simulation, researchers understand RW p. These issues draw on problems related to representation, realism, and the notion of understanding in the context of computer simulations. Unfortunately, these topics have also received little attention in the specialized literature except for Bueno See footnote 9.
The bottom line is that we have on our hands a computationally-based explanation that differs in important aspects from a model-based explanation, and thus offers significant philosophical value.
The philosophical analysis of computer simulations must recognize the rich structure and methodology of SM , and refrain from reducing the discussion to the mere implementation of an M onto a physical computer.
The latter way of looking at computer simulations impoverishes their nature and importance in scientific practice, ultimately assimilating simulations as tools with mere instrumental value. Over the past years, much has been discussed concerning the heterogeneity of models and the myriad of methods by which they can be constructed, applied, and evaluated in their own right.
The work presented here is an attempt to support the claims behind these and other philosophers who share similar viewpoints.
I depart from them, however, in that I focus on a new kind of scientific model, that is, the simulation model. I am, of course, not alone in this enterprise. As mentioned, Humphreys already attempted to distinguish mathematical models from computational methods, and authors sensitive to the methodology of computer simulations have followed a similar path Winsberg ; Morrison ; Varenne ; Symons and Alvarado My approach departs from theirs on several accounts, including the use of studies in computer science and software engineering as the most suitable body of knowledge for understanding simulation models.
With this firmly in mind, I showed that recasting is a significant methodological move that brings new insight into studies on simulation models and computer simulations. Moreover, recasting makes sense of the fact that simulation models handle many parameter values and thus hold great representational capability and model expressiveness, arguably the most celebrated characteristics of computer simulations along with their computing power. Admittedly, with some discussions, I have only scratched the surface of what is now deemed a pressing issue.
This is most visible in my treatment of the novelty of simulation models for explanations. Humphreys also works on the philosophical novelty of computer simulations in his book Humphreys Because Humphreys is interested in computational science in general, he calls these models computational models. Because I am interested in computer simulations in particular, I use the more accurate term simulation models. Whereas it can be shown that the architecture of simulation models fits different computer simulations of varying complexity, the philosophical implications discussed in Sect.
This explains why all five functions of simulations described by Hartmann heavily resemble finding solutions to the underlying model Hartmann , 84— This article supports the claim that our general understanding of computer simulations can be significantly advanced if we shift away from the study of simulations as branches of mathematics and physics, and focus more on computer science and software engineering as basic disciplines.
We are not interested in discussing the nature of algorithms. Important insights can be found in Blass et al. To the best of my knowledge, only Bueno has offered a theory of representation for computer simulations.
Let us note that the interpretation offered by Humphreys , , Winsberg , are special cases to my architecture where the two exceptions apply. Not to be confused with other forms of structuralism in the literature, such as structuralism about the natural numbers Benacerraf ; Halbach and Horsten and structural realism Ladyman , among others.
Let it be noted that I group under the same umbrella a series of disparate computer processes: a database and a daemon are different in fundamental respects. Nevertheless, and for the purposes of this article, I see no objection in referring to all of them as integration modules IM k.
Ajelli, M. Comparing Large-scale computational approaches to epidemic modeling: agent-based versus structured metapopulation models.
Google Scholar. Altman, A. Causal models. Bailer-Jones, D. Scientific Models in Philosophy of Science. Pittsburgh: University of Pittsburgh Press. Balcan, D. Multiscale mobility networks and the spatial spreading of infectious diseases. Proceedings of the National Academy of Sciences, 51 , — Barberousse, A. Computer Simulations and empirical data.
Newcastle upon Tyne: Cambridge Scholars Publishing. Beisbart, C. How can computer simulations produce new knowledge? European Journal for Philosophy of Science, 2, — Benacerraf, P. What numbers could not be. The Philosophical Review, 74 1 , MathSciNet Google Scholar. Blass, A. When are two algorithms the same? The Bulletin of Symbolic Logic, , — Boge, F.
Why computer simulations are not inferences, and in what sense they are experiments. European Journal for Philosophy of Science, 9 1 , Boyer-Kassem, T. Layers of models in computer simulations. International Studies in the Philosophy of Science, 28 4 , — Bueno, O.
Computer simulation: An inferential conception. The Monist, 97 3 , — Chalmers, D. On implementing a computation. Minds Mach, 4 4 , — Chirimuuta, M. Minimal models and canonical neural computations: The distinctness of computational explanation in neuroscience, explanation and explain, explanation and explain and explanatory, explanation and explain and explanatory and simulation.
Synthese, 2 , — Colburn, T. Software, abstraction, and ontology. The Monist, 82 1 , 3— Abstraction in computer science. Minds and Machines, 17 2 , — Copeland, J.
What is computation? Synthese, 3 , — DeAngelis, D. Individual-based models in ecology after four decades. FPrime Reports, 6 39 , 1—6. Dresner, E. Measurement-theoretic representation and computation-theoretic realization. Journal of Philosophy, 6 , — Computer simulations and the changing face of scientific experimentation.
Arnold Eds. Varieties of simulations: From the analogue to the digital. Gehring Eds. Berlin: Springer. Computer simulations in science and engineering.
A formal framework for computer simulations: Surveying the historical record and finding their philosophical roots a formal framework for computer simulations: Surveying the historical record and finding their philosophical roots. Article Google Scholar. Grounds for trust: Essential epistemic opacity and computational reliabilism. Minds and Machines, 28 4 , — Simulation is not just a fun activity where learners get to collaborate with colleagues and peers. As designers and training professionals, you must help them see how the decisions they made in the simulation and the application of new skills and practices relate to their work.
In some cases that link is obvious, such as when the decisions in the simulation mirror real life, and other times the simulation is practicing something new. Take time in the formal learning setting to discuss current state vs. If you combine all of these elements into a simulation, you will create a memorable and impactful learning activity for your formal learning while also creating a powerful tool where learners learn from each other.
Amy Happ is the senior vice president for performance consulting and business leadership at Advantexe Learning Solutions. Stay up to date on the latest articles, webinars and resources for learning and development.
Learning Application This is the learn-by-doing element. Finally, after feeding the model data, the simulation could be run and its operation seen over time, allowing refinement and analysis of the results. If the average queue size exceeded the specified limit, the number of available staff was increased and a new experiment was done.
It is possible for this to happen automatically until an optimal solution is found. Overall, multiple scenarios may be explored very quickly by varying parameters. They can be inspected and queried while in action and compared against each other.
The results of the modeling and simulation, therefore, give confidence and clarity for analysts, engineers, and managers alike. Try for free Read the white paper. Simulation is the best approach for addressing business challenges Learn why in the white paper Developing Disruptive Business Strategies with Simulation Read now.
0コメント