Testing the Agent‐Based Simulation Verification, Validation, and Accreditation Framework Study Framework: Validation Report for the Pythagoras Counter‐Insurgency (COIN) Model Technical Report T08‐ER07‐002 version 1.3 In support of Northrop Grumman sub‐contract GVVAIIWER07, Purchase Order 7500012276 The Agent‐Based Simulation Verification, Validation, and Accreditation Framework Study Phase II Prepared for Northrop Grumman Space and Mission Systems Corp. 12900 Federal Systems Park Dr. Fairfax, VA 22033 (703) 968‐3433 and The Marine Corps Combat Development Center Quantico, VA Prepared by WernerAnderson, Inc. 6609 Main Street, Gloucester, VA 23061 (804) 694‐3173 www.werneranderson.com 25 August 2008
T08‐ER07‐002 Executive Summary The validation report is in response to Phase II of the U.S. Marine Corps (USMC) Agent‐Based Simulation (ABS) Verification, Validation, and Accreditation (VV&A) Framework Study. The objective of Phase I was to create a framework for the VV&A of models under consideration for future entry into the USMC Irregular Warfare (IW) Analytic Baseline. While VV&A covers three distinct processes, the primary focus of the Framework Study effort was on validation. The objective of Phase II is to test the developed framework. This validation report results from the application of the framework to a Counter‐
Insurgency (COIN) scenario developed for Pythagoras (P‐COIN) in support of Irregular Warfare Analysis. The main purpose of this validation effort is primarily to test the framework developed in Phase I of this study. This testing effort also includes two other validation efforts. The first is a validation of the conceptual model validation of P‐COIN. The second is a validation of an obstacle clearing model implemented in Pythagoras. This report of the validation effort completed for P‐COIN not only provides the validation analysis, it also serves as a model for other reports based on this framework. The report also provides insights into framework improvements and general ABS Validation methodology. These insights and the resulting improvements to the Agent‐Based Simulation Validation Framework will be provided in the ABS VV&A Framework Study Phase II Final Report. A main goal of the ABS VV&A Framework Study was to provide a validation process that is traceable, repeatable, transparent, and communicable to allow the making of an appropriate decision on the use of the ABS. These goals mean inherently that the supporting documentation and data required to allow another analyst to trace through or report the validation effort should be available. While a validation report proper could contain the supporting documentation and detail for the conclusions reached through the validation assessment, in the interests of readability, these are provided in appendices. ES.1. Background Although, the main purpose of this validation effort was to test the ABS VV&A Framework developed in Phase I and not to assess the validity of the P‐COIN model per se, the evaluation of the framework could not occur without a reasonably complete validity assessment of P‐COIN. Further, to satisfy the goal of providing a model for future validation reports, this report needs to provide an example of the material that should be included in a validation report. Therefore, while the purpose of this validation effort was not to “validate P‐COIN,” the process of testing the ABS VV&A framework, of necessity, created a validation assessment and accompanying report. A particular challenge in completing this validation effort was that the methodology and framework were in development concurrently with the P‐COIN development and analysis. Additionally, significant insights into the framework occurred midway through the validation effort causing a significant redirection of effort and an alignment between two separate efforts: the P‐COIN analysis and development and the P‐COIN validation as a test of the ABS VV&A Framework. This concurrence and alignment as well as the mid‐course insight has led to some areas having less reporting than others as well as work completed early in the validation process not strictly utilized as part of the final validation effort. For completeness, these additional validation analyses are included as appendices with descriptions of their applicability to the overall effort. Areas with less validation assessment than others are also included with the limitations of the full analysis identified. The focus on this validation effort and the ABS VV&A Framework is on analysis. That is, the questions that the P‐COIN model can support and the degree to which this model can support those questions. In WernerAnderson, Inc. T08‐ER07‐002 the validation process, there is a heavy reliance on the documentation of the analysis by the analyst. The results of the framework and the application of the validation experimental procedures provide a method for decisions makers to know the boundaries of a model’s use and the limits the “believability” of their results. The development product is a procedure to evaluate whether a model or simulation can be used to support an analysis objective. Therefore, validation is, in part, a questioning or prodding of the analysis and cannot be decoupled from that analysis. In other words, it is an analysis of the analysis. In this validation process, the analyst assesses when answering the posed analytical question (intended use) the important elements for answering the question, the data required, and the tests (sensitivity, experiments, or excursions) needed so that the decision makers or recipients of the analysis can have their questions answered and confidence in the simulation results. That is, it is the role of the analyst in validating the use of the model for an analytical purpose to ensure the ability to assign a reason for all the elements in the model and the results. ES.2. Analysis Question While the primary intent for the development of P‐COIN was to assess whether and how Pythagoras could be used to model population dynamics, the analysis question for this validation effort is In a Disaster Relief/Humanitarian Assistance mission for the above scenario, is it better to base the MAGTF ashore or afloat? Answering this question is the intended use for the validation assessments. ES.3.
Conclusions A summary of the conclusions of the validation assessments made to P‐COIN are as follows: 1) The P‐COIN simulation fails to capture the dynamic effects intended in the conceptual model of the insurgency in Colombia provided to the P‐COIN developer. That is, P‐COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF. 2) The data supporting the P‐COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming‐up” the Markov chains supporting the data used to build the P‐COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change. 3) The P‐COIN model should not be used to evaluate long term effects on the population resulting from the influencing event of the MAGTF arrival. 4) This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs. 5) There is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure. However, item 1 implies that P‐COIN also provides little insight into the ashore or afloat question in its current implementation. This report provides the supporting documentation for these conclusions. WernerAnderson, Inc. ii T08‐ER07‐002 ES.4. Recommendations Although the results of the analysis reflect the expectations of the analysis based on the input data tables and the analysis does not recommend a change in course of action, it is difficult to trust the results without being able to trust underlying dynamics of the model. The chosen instantiation of only applying attribute changers reflective of the start state of the agent population distribution should be thoroughly evaluated with the recommendation to apply these attribute changers more robustly to reflect how the population insurgency affiliation changes over time and in response to system events. The full dynamics intended by the use of salience and the natural drift aspects of the population segments ought to be included in the P‐COIN model in order to allow for the emergence of secondary and tertiary effects of the influencing event of the MAGTF arrival. The developer should also develop robust test cases to evaluate the dynamic behavior in isolation to gain surety that the combined dynamic behavior should be trusted. Further, the following material would have been useful in the validation assessment: 1)
2)
3)
4)
5)
6)
Better documentation on the P‐COIN instantiation Time series data A descriptive walk‐thru of results charts (meaning & implications) Verification cases (isolated effects) to ensure dynamics have expected direction (first derivative) and order of magnitude with descriptions of what was believed to be correct Better explanations of expected resulting effects from data values in the referent as most had to be inferred and order of magnitude differences unknown Expected interaction effects WernerAnderson, Inc. iii T08‐ER07‐002 Table of Contents Executive Summary ........................................................................................................................................ i ES.1. Background .................................................................................................................................... i ES.2. Analysis Question .......................................................................................................................... ii ES.3. Conclusions ................................................................................................................................... ii ES.4. Recommendations ....................................................................................................................... iii 1 Introduction .......................................................................................................................................... 1 1.1 Structure of the Report ................................................................................................................. 1 1.2 Authors and Contributors ............................................................................................................. 1 2 Context of Validation ............................................................................................................................ 3 2.1 Application Area ............................................................................................................................ 3 2.2 Intended Use ................................................................................................................................. 3 2.3 Validation Criteria ......................................................................................................................... 4 2.3.1 Decision Criteria .................................................................................................................... 4 2.3.2 Analysis Measures of Effectiveness ...................................................................................... 5 2.4 Analysis Overview ......................................................................................................................... 5 3 Description of the Conceptual Model ................................................................................................... 6 3.1 Columbian Population ................................................................................................................... 7 3.2 Insurgency Susceptibility of Population Sectors ........................................................................... 7 3.3 Salience between Population Sectors ........................................................................................... 8 3.4 MAGTF Influence......................................................................................................................... 10 4 Implementation in P‐COIN .................................................................................................................. 11 4.1 Basic Agent Implementation ....................................................................................................... 11 4.2 Attribute Changers ...................................................................................................................... 11 4.2.1 Natural insurgency susceptibility ........................................................................................ 11 4.2.2 Salience ............................................................................................................................... 12 4.2.3 MAGTF Event Influence ...................................................................................................... 14 5 Conceptual Model Validation ............................................................................................................. 16 6 Data Considerations ............................................................................................................................ 17 6.1 Salience Data ............................................................................................................................... 17 6.2 Vulnerability Data ....................................................................................................................... 18 6.3 MAGTF Data ................................................................................................................................ 18 6.4 Data Precision ............................................................................................................................. 18 6.5 Data Processing ........................................................................................................................... 19 7 Results Validation ............................................................................................................................... 20 7.1 Analytical Results ........................................................................................................................ 20 7.2 Validity Assessment .................................................................................................................... 21 7.3 Accuracy Assessment .................................................................................................................. 21 8 Recommendations and Conclusions ................................................................................................... 23 8.1 Caveats and Limitations .............................................................................................................. 23 8.2 Cautions ...................................................................................................................................... 23 8.3 Validity with Respect to Intended Use ........................................................................................ 23 WernerAnderson, Inc. i T08‐ER07‐002 8.4 8.5 Summary ..................................................................................................................................... 23 Recommendations ...................................................................................................................... 24 Appendix A P‐COIN Support to the Irregular Warfare Study ............................................................... A‐1 Appendix B Irregular Warfare Project Description ............................................................................... B‐1 Appendix C Conceptual Model Data ..................................................................................................... C‐1 C.1 Scenario Population Data ........................................................................................................... C‐1 C.2 Agent Allocation ......................................................................................................................... C‐2 C.3 Salience ...................................................................................................................................... C‐2 C.4 Scenario Vulnerability Data ........................................................................................................ C‐3 C.4.1 Salience matrix ................................................................................................................... C‐5 C.5 MAGTF Influence Estimation ..................................................................................................... C‐5 Appendix D Instantiated Model ............................................................................................................ D‐1 Appendix E Descriptions of Attribute Changers .................................................................................... E‐1 Appendix F Results from the P‐COIN Analysis ...................................................................................... F‐1 Appendix G Pythagoras Usage, Validation, and Accreditation Histories .............................................. G‐1 Appendix H Pythagoras‐COIN Verification during Development ......................................................... H‐1 H.1 Underlying Mathematics ........................................................................................................... H‐1 H.2 Other Verification ..................................................................................................................... H‐1 Appendix I Validation Documentation Prior to Workshop 2 ................................................................ I‐1 Appendix J Acronyms ........................................................................................................................... J‐1 WernerAnderson, Inc. ii T08‐ER07‐002 List of Figures Figure 1. Conceptual Model of Civilian Population ...................................................................................... 6 Figure 2. Natural Drift between Population Segments ................................................................................ 8 Figure 3. Effect of Salience on Population Segment Sectors ..................................................................... 10 Figure 4. Measure of Pro‐COIN and COIN .................................................................................................. 20 Figure 5. Measure of Pro‐ FARC and FARC ................................................................................................. 21 List of Tables Table 1. Distribution of Columbian People across Population Segments .................................................... 7 Table 2. Population Distribution across Segments and Sectors................................................................... 7 Table 3. Salience Values for Colombian Scenario ........................................................................................ 9 Table 4. MAGTF Influence Estimation Data ............................................................................................... 10 Table 5. Initial Agent Allocation by Segment and Sector ........................................................................... 11 Table 6. Distribution of Columbian People across Population Segments ................................................. C‐1 Table 7. Population Distribution across Segments and Sectors................................................................ C‐1 Table 8. Number of People in Each Sector by Population Segment (Derived) ......................................... C‐2 Table 9. Initial Agent Allocation by Segment and Sector (
) ................................................................ C‐2 Table 10. Catholic Church Vulnerability Matrix ........................................................................................ C‐3 Table 11. Displaced Persons Vulnerability Matrix .................................................................................... C‐3 Table 12. Illicit Organizations Vulnerability Matrix ................................................................................... C‐3 Table 13. Military Vulnerability Matrix ..................................................................................................... C‐3 Table 14. Old Money Vulnerability Matrix ................................................................................................ C‐4 Table 15. Police Vulnerability Matrix ........................................................................................................ C‐4 Table 16. Urban Middle Class Vulnerability Matrix .................................................................................. C‐4 Table 17. Urban Poor Vulnerability Matrix ............................................................................................... C‐4 Table 18. Salience Values for Colombian Scenario ................................................................................... C‐5 Table 19. MAGTF Influence Estimation Data ............................................................................................ C‐5 WernerAnderson, Inc. iii T08‐ER07‐002 1
Introduction This report gives the results of the validation assessments for the Northrop Grumman Pythagoras Counter Insurgency (P‐COIN) model conducted as part of the Agent‐Based Simulation (ABS) Verification, Validation, and Accreditation (VV&A) Framework Study Phase II in support of Northrop Grumman sub‐
contract GVVAIIWER07, Purchase Order 7500012276. The objective of these validation assessments is to test the ABS Validation Framework developed in Phase I of the study. This report not only provides the validation analysis, it also serves as a model for other reports based on this framework. The report also provides insights into framework improvements and general ABS Validation methodology. These insights and the resulting improvements to the ABS Validation Framework will be provided in the ABS VV&A Framework Study Phase II Final Report. This report is prepared for Northrop Grumman Space and Mission Systems Corp., at 12900 Federal Systems Park Dr., Fairfax, VA 22033 and the Marine Corps Combat Development Center (MCCDC), in Quantico, VA. It is submitted to meet deliverable requirements of Task 2 in the Statement of Work. 1.1 Structure of the Report The structure of this report deviates from that suggested in the Phase I final report. It attempts to incorporate suggestions made during workshop 2 of Phase II of the ABS VV&A Framework Study. Workshop 2 suggestions were to develop a multi‐layered report with a preface containing key recommendations and conclusions, more detail in the body of the report that substantiates these conclusions, and leaving the supporting data and the detailed analyses for appendices. This report endeavors to do so. The structure of this report has gone through several iterations in determining the most effective way of communicating the logic and reasoning behind the conclusions reached in the validation assessments. The organization suggested in the Phase I final report was by topic (e.g., conceptual model, mathematical model, implemented model). This organizational structure resulted in redundancies of information and lacked a cohesive structure when relating the results of the validity assessments of the model and the analysis to which it was put. Therefore, the body of this report contains three main elements: 1) Background about the model and the analysis (sections 1, 2, and 3) 2) Validity assessments and issues (sections 4, 5, 6, and 7) 3) Conclusions and Recommendations (section 8) Appendices contain supporting documentation and elaborate on the reasoning where needed. In addition to the appendices required to support the validation discussions provided in this report, Appendix I contains much of the validation work done prior to Workshop 2, since much of that work done prior to it was rendered moot following this workshop. 1.2 Authors and Contributors The following people wrote this report: Lisa Jean Moya Eric W. Weisel, Ph.D. WernerAnderson, Inc. WernerAnderson, Inc. WernerAnderson, Inc. 1 T08‐ER07‐002 The following persons 1 (listed in alphabetical order) contributed ideas, facts, references, and topics included in this report: Edmund Bitinas Brittlea Sheldon Northrop Grumman Northrop Grumman For additional information regarding the content of this report, please contact Lisa Jean Moya at email [email protected] or telephone (804) 694‐3173. 1
Any errors or omissions in the content of this report are the responsibility of the authors, not the contributors. WernerAnderson, Inc. 2 T08‐ER07‐002 2
Context of Validation The purpose of these validation experiments is to exercise the overall validation framework developed as part of Phase I of the Agent‐Based Simulation (ABS) Verification, Validation, and Accreditation (VV&A) Framework Study. The purpose of this exercising of the framework is to improve the framework through a documented application, to clarify the framework for future use, to give an example of the use of the framework and the resulting validation report, and to identify specific validation tests and areas of interest for agent‐based simulations. This validation will address all areas of the framework in order to illustrate and illuminate the framework. However, this report does not address all areas for validation as completely as would be expected if the purpose of the validation were to validate the Pythagoras‐COIN (P‐COIN) model. The body of the report identifies these areas as they occur. In order to exercise the framework, the validation agent (WernerAnderson, Inc.) needed a context against which to test the validation experiments. This section describes that validation context. 2.1 Application Area P‐COIN was developed as part of a research effort into how analysts can construct campaign level models of irregular warfare. The first objective of the effort is “to create a Pythagoras scenario that simulates the effects on the civilian population of Marine Air‐Ground Task Force (MAGTF) operations during a specific counterinsurgency as part of a joint task force” <<Pyth_COIN_Draft Final Report.pdf>>. This effort includes the construction of software and the identification of the necessary data with collection, interpretation, and preparation methods for use into the developed models. This effort specifically includes development of a process for applying data into the existing Pythagoras modeling environment. Thus, the first question to be answered by the P‐COIN study is, “Can Pythagoras be used to model population dynamics, and if so, how is that accomplished?” Secondly, “[t]he effort will then determine if the resulting scenario can be reconfigurable to represent other regions of the world and applied to real‐world situations” <<Pyth_COIN_Draft Final Report.pdf>>, with a specific emphasis on the determination of whether the resulting model can be used to answer specific operational questions. The sample question posed in the study application is, “Should the MAGTF deploy ashore or should it remain afloat?” Appendix I of “Pythagoras COIN Application to Support the IW Study Final”, dated 9 April 2008 found as Appendix A of this report gives the scenario specifics surrounding this analytical question. In summary, in the developed P‐COIN scenario, the population modeled has eight general population segments based on ethnicity, political orientation, family history, socio‐economic status, living location and occupation: Catholic Church, Displaced Persons, Illicit Organizations, Military, Police, Old Money, Urban Middle Class, and Urban Poor. It is sub‐divided further into five insurgency sectors: FARC, Pro‐
FARC, Neutral, Pro‐GOVT, and GOVT. The population has a tendency to drift naturally between these insurgency sectors based on a natural tendency (vulnerability) and the influence that the various population segments have upon each other (salience). The arrival of the MAGTF, either ashore or afloat, is an event that further influences these population segments in insurgency orientation for the duration of its stay. 2.2 Intended Use The P‐COIN Development Team was provided with a Colombia scenario to model COIN based upon the lessons learned while modeling the Troubled Country scenario. In this scenario, the Marine Air Ground‐
Task Force (MAGTF) has been assigned to carry out humanitarian assistance and disaster relief in WernerAnderson, Inc. 3 T08‐ER07‐002 response to the tsunami in the Valle Del Cauca and Cauca. The population in this province is divided into eight general segments based upon ethnicity, political orientation, family history, socio‐economic status, living location and occupation. These eight population segments are the Urban Poor, Displaced Persons, Military, Police, Illicit Organizations, Old Money, Catholic Church, and Urban Middle Class. 2 While the primary intent for the development of P‐COIN was to assess whether and how Pythagoras could be used to model population dynamics, the analysis question for this validation effort is In a Disaster Relief/Humanitarian Assistance mission for the above scenario, is it better to base the MAGTF ashore or afloat? Answering this question is the intended use for the validation assessments. When structuring this analysis, a key element was posing the question in such a way and identifying metrics as well as specifying a methodology that allows the answering of the question in a supportable way. This process may require a restructuring of the question asked, an evaluation of the methodology and results from the decision maker’s perspective, and a clear specification of the question at hand as well as possible “answers.” These answers are part of the validation criteria. 2.3 Validation Criteria The assessment between ashore or afloat in the posed analysis question is determined by which alternative produces the fewest pro‐insurgents and insurgents during the duration of the MAGTF stay. Thus, rather than having the actual proportion of the population in each insurgency sector as the metric of interest for an accuracy assessment, the validation metric of interest is the accuracy of the relative ranking between the two alternatives. This relative ranking needs to be insensitive to any inaccuracies in the data used to populate the model and to the fidelity and level of detail in the conceptual model. The focus on this validation effort and the ABS VV&A Framework is on analysis. That is, the questions that the P‐COIN model can support and the degree to which this model can support those questions. In the validation process, there is a heavy reliance on the documentation of the analysis by the analyst. The results of the framework and the application of the validation experimental procedures provide a method for decisions makers to know the boundaries of a model’s use and the limits the “believability” of their results. The development product is a procedure to evaluate whether a model or simulation can be used to support an analysis objective. Therefore, validation is, in part, a questioning or prodding of the analysis and cannot be decoupled from that analysis. In other words, it is an analysis of the analysis. In this validation process, the analyst assesses when answering the posed analytical question (intended use) the important elements for answering the question, the data required, and the tests (sensitivity, experiments, or excursions) needed so that the decision makers or recipients of the analysis can have their questions answered and confidence in the simulation results. That is, it is the role of the analyst in validating the use of the model for an analytical purpose to ensure the ability to assign a reason for all the elements in the model and the results. 2.3.1 Decision Criteria The decision metric for this question is to minimize the negative impact of the MAGTF arrival as measured by the FARC and Pro‐FARC sentiment of the population segments within the population. This 2
This material is quoted from The Pythagoras Counterinsurgency (COIN) Application to Support the Marine Corps Irregular Warfare (IW) Study, Draft Final Report, dated 9 April 2008 WernerAnderson, Inc. 4 T08‐ER07‐002 is measured through the percent of the total population in these combined insurgency sectors as well as an evaluation of any change as a result of the MAGTF arrival within individual population segments. Two outcomes are of interest: 1. Do no harm: create no increase in insurgency activity. 2. Improve the political situation: create an improvement in GOVT and Pro‐GOVT sectors. These results need to be evaluated not only for the entire population but also for high interest or high value population segments. Since the standard course of action is to base afloat, of particular interest is whether there is a statistically significant reason to believe from the P‐COIN analysis that ashore is a better option. 2.3.2 Analysis Measures of Effectiveness The analysis team determined which option was better by evaluating box and whiskers plots of the 50 runs completed for each option comparing the insurgency orientation change for each population segment over the time of the simulation run with preference given to the option that appeared to have a lesser FARC population by percentage. 2.4 Analysis Overview Northrop Grumman did not provide written descriptions of the analysis procedure in time to develop this validation report. Descriptions of the analysis conducted were developed from discussions with the Northrop Grumman team as well as Interim Reports and Interim Report Briefings provided to the validation agent. A draft final report with the analysis included as appendix N to that report was provided to the validation agent as courtesy on 17 June 2008. Additional evaluations are ongoing, but no significant changes in the conclusions of this report are anticipated. To conduct the analysis, the scenario was built in Pythagoras (called P‐COIN). Attribute changers were used to affect the change in insurgency orientation of the population segments due to vulnerability, salience, and the influencing event of the MAGTF arrival. Since the data is imprecise, tolerances were applied to the input data to capture the available precision of that data. The MAGTF influence had a tolerance of 10%, while salience and vulnerability had a tolerance of 25%. Fifty runs were made for each of three options: no MAGTF, MAGTF‐ashore, and MAGTF‐afloat. These runs were from a uniform draw around the input data based on the tolerances applied. The data was collected to form a distribution for each of the options. The results were compared on a relative basis and any outliers were assessed. The non‐MAGTF runs formed a baseline for the comparison and provided a basis for a reasonableness comparison for the results. WernerAnderson, Inc. 5 T08‐ER07‐002 3
Description of the Conceptual Model The focus in P‐COIN is on the changing distribution of insurgency sector orientation amongst population segments. This orientation changes over time due to the natural tendency of the population to be affiliated with the insurgency sector, the degree to which each population segment wants to be like another, and the effect events have on the population segments. The Valle Del Cauca and Cauca provinces of Colombia are the population of interest developed for Pythagoras‐COIN. The population modeled has eight general population segments based on ethnicity, political orientation, family history, socio‐economic status, living location and occupation: Catholic Church, Displaced Persons, Illicit Organizations, Military, Police, Old Money, Urban Middle Class, and Urban Poor. It is sub‐divided further into five insurgency sectors along the spectrum between support for the government (GOVT) and for the insurgency, i.e., the Revolutionary Armed Forces of Colombia (FARC): FARC, Pro‐FARC, Neutral, Pro‐GOVT, and GOVT. The population has a tendency to drift naturally between these insurgency sectors based on a natural tendency (vulnerability) and the influence that the various population segments have upon each other (salience). The arrival of the MAGTF, either ashore or afloat, is an event that further influences these population segments in insurgency orientation for the duration of its stay. A Markov chain provides the based descriptive model and data for the vulnerability of the population segments toward the spectrum of insurgency. Salience factors modify this Markov chain to allow the population segments to influence each other. This allows events to affect population segments not directly targeted or affected by an event. Additional factors are used to capture the influence of events in the simulation; in the case of this model application, the on‐shore or afloat presence of the MAGTF. Figure 1 depicts this. Civilian
Population
Vulnerability
Salience
Influencing events
Population
Segments
Insurgency Behavior Orientation
FARC
Pro‐FARC
Neutral
Pro‐GoC
GoC
FARC = Revolutionary Armed Forces of Colombia
GoC = Govt of Colombia
Figure 1. Conceptual Model of Civilian Population WernerAnderson, Inc. 6 T08‐ER07‐002 3.1 Columbian Population The population distribution of the P‐COIN irregular warfare (IW) analysis at Table 1. ID 1 2 3 4 5 6 7 8 Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Total People 324 27,882 324 973 324 648 134,222 159,510 324,207 0 months 3 as shown in % Population 0.10% 8.60% 0.10% 0.30% 0.10% 0.20% 41.40% 49.20% 100.00% Table 1. Distribution of Columbian People across Population Segments Five insurgency sectors further divide this population as shown in Table 2: Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor FARC Pro‐FARC Neutral Pro‐GOVT GOVT 0.00%
6.88%
0.00%
1.61%
0.00%
0.00%
3.75%
5.80%
0.00%
36.64%
27.65%
9.68%
2.08%
5.26%
3.75%
9.17%
38.46%
44.70%
39.02%
0.00%
4.17%
0.00%
62.50%
65.75%
61.54%
10.29%
18.18%
88.71%
62.50%
94.74%
10.00%
15.12%
0.00%
1.50%
15.15%
0.00%
31.25%
0.00%
20.00%
4.16%
Table 2. Population Distribution across Segments and Sectors 3.2 Insurgency Susceptibility of Population Sectors Two factors determine the transition between insurgency sectors. The first is the natural insurgency susceptibility, the natural drift or vulnerability, of a population segment between insurgency sectors. The second is the degree to which population segments are more or less inclined to be similar to one another. The first is modeled via vulnerability matrices; the second is modeled via salience. Vulnerability matrices describe the standard state transition Matrix of a Markov chain. These susceptibility, or vulnerability, values remain constant for the duration of the simulation. Figure 2 notionally depicts this transition between insurgency sectors. 3
The Pythagoras COIN Application to Support the IW Study Interim Report #4, 11 Feb 2008, documentation provided on the model specifies a time step of weeks. Subsequent communication on 27 Feb 2008 with the Pythagoras‐COIN developers corrected the time step to months. WernerAnderson, Inc. 7 T08‐ER07‐002 FARC
Pro‐FARC
Neutral
Pro‐GOVT
GOVT
Catholic Church
Displaced Persons
Illicit Organizations
Military
Old Money
Police
Urban Middle Class
Urban Poor
Figure 2. Natural Drift between Population Segments The graphic allows all transitions between the five insurgency sectors: FARC, Pro‐FARC, Neutral, Pro‐
GOVT, and GOVT. The vulnerability matrices for each population segment specify the probability values for the transitions shown by the arcs on the graph. The data itself specifies whether this transition is possible for a specific population segments within the scenario modeled in P‐COIN. These data were derived in an analysis process documented in Appendix B. The data values for the transition matrices provided to the P‐COIN development team are given in Appendix C. 3.3 Salience between Population Sectors Salience is the degree of influence different population segments have on the beliefs of other population segments. It represents the indirect effect of actions and events occurring within the population. That is, it determines the push or pull of one population segment toward or away from the insurgency orientation of another population segment. For instance, if the influenced population is Neutral, then the FARC, Pro‐FARC, Pro‐GOVT and GOVT sectors of the influenced population would be pushed toward Neutral by the degree of salience (i.e., positive salience values would push towards Neutral, negative values would pull away from Neutral). Neutral in the influenced population would be more or less neutral depending on the degree of influence, the direction of that influence, and the delta in average insurgency. It is derived from Charles Osgood’s Semantic Differential, used widely in advertising and market research. This measurement combines three factors to create a measure of salience: evaluative (E), measured from good to bad; activity (A), measured from active to passive; and potency (P), measured from strong to weak. Table 3 gives salience values for the Colombia Pythagoras‐COIN scenario. Positive values in the table indicate a tendency to move toward the position of the population in question, while negative values indicate a tendency to move away from the position of the population in question. WernerAnderson, Inc. 8 T08‐ER07‐002 Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Catholic Church 0.000 0.706 0.207 0.750 0.474 0.000 0.816 0.202 0.000 (0.262) 0.000 (0.351) (0.486) (0.843) (0.623) (0.259) 0.000 (0.052) 0.425 (0.295) (0.713) (0.174) (0.387) 0.101 (0.078) (0.230) (0.053) 0.492 0.000 (0.039) 0.136 0.225 0.801 0.178 0.420 0.262 0.840 (0.273) (0.269) (0.033) (0.078) (0.366) (0.190) 0.685 (0.506) 0.767 (0.205) (0.134) 0.000 0.480 (0.467) 0.000 0.494 0.000 0.392 0.552 0.000 (0.208) 0.183 (0.257) (0.250) (0.218) 0.132 (0.079) Table 3. Salience Values for Colombian Scenario The conceptual meaning for the salience values 1, 1 is not given. Descriptions of the methods used to generate the data can be found in Appendix B. The following interpretation for the data in Table 3 as demonstrated by Figure 3 was used for the validation analysis. The picture shows the average insurgency orientation for each population segment. For population segments having positive salience values, sectors within that population are driven toward that population segment average insurgency orientation (shown in blue). That is, a positive salience value indicates that the population segment in question (shown in the rows of Table 3) wants to be more like the population segment of interest (shown in the column of Table 3). The opposite is true for negative salience values. For population segments having negative salience values, sectors within that population are driven away from that population segment average insurgency orientation (shown in red). The magnitude of the values in the table determines the degree of this change. Each population segment affects each other population segment including itself. This causes the influenced population segments to be more/less like the influencing population segment. The overall effect of all these population influences neither is known nor is posited. Hence, this is a source of dynamic or emergent behavior within the system; i.e., the combined effect of all these influences once the MAGTF arrives is unpredicted from a strict evaluation of the data. For instance, the Catholic Church influences the Urban Middle Class quite strongly to be more like the Catholic Church (i.e., Pro‐GOVT). In contrast, the Displaced Persons and Illicit Organizations while also being influenced by the Catholic Church to be more like the Church, both influence the Urban Middle Poor to be less like their orientation. This effect could either counteract or magnify the influence of the Church, and which result it not obvious. These effects are combinatorial across the population segments with various segments having a positive influence through one group and a negative influence through other groups. The data suggests that these indirect effects from the MAGTF arrival could influence the preferred solution through their secondary and tertiary effects. WernerAnderson, Inc. 9 T08‐ER07‐002 FARC
Pro‐FARC
Neutral
Catholic Church
3.03
Displaced Persons
2.63
Pro‐GOVT
Illicit Organizations
3.76
Military
3.89
Old Money
3.21
Police
4.23
Urban Middle Class
3.62
GOVT
3.39
Urban Poor
Figure 3. Effect of Salience on Population Segment Sectors 3.4 MAGTF Influence The arrival of the MAGTF is an influencing event in the model. Explanations for interpreting the data were not provided to the validation agent. For each option, sea based or shore based, the MAGTF influenced each population segment both toward the FARC (left) and to COIN (right). Table 4 gives the data provided and used in the model development. Sea Based Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Right 0.845 0.596 0.000 0.000 0.631 0.564 0.780 0.798 Left 0.000 0.000 0.397 0.408 0.000 0.000 0.000 0.000 Shore Based Right Left 0.401 0.000 0.721 0.117 0.000 0.447 0.000 0.408 0.631 0.000 0.564 0.000 0.184 0.210 0.722 0.211 Table 4. MAGTF Influence Estimation Data WernerAnderson, Inc. 10 T08‐ER07‐002 4
Implementation in P‐COIN This section describes the way in which P‐COIN implements the system of interest. It describes the implementation of the model in terms of the building blocks within Pythagoras 2.0.0. Appendix C contains the data used in the conceptual model development. Appendix D contains the XML implementation as P‐COIN. 4.1 Basic Agent Implementation Agents in P‐COIN represent 1% within one of the eight population segments, where at initialization the entirety of the 1% population segment has the same insurgency orientation. Thus, an agent has five attributes representing the percent of that agent’s population within one of the five insurgency sectors. P‐COIN has 800 agents; 100 for each population segment. Since the proportions of population in each sector are not integer in percentage, the modeler needs to make a judgment call on how to allocate these agents amongst each insurgency sector. The allocation chosen is shown in Table 5 and is found by rounding. Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Total FARC 0 7 0 2 0 0 4 6 19 Pro‐FARC 0 37 39 10 2 5 4 9 106 Neutral 38 45 28 0 4 0 62 66 243 Pro‐GOVT 0 10 18 88 63 95 10 15 299 GOVT 62 1 15 0 31 0 20 4 133 Table 5. Initial Agent Allocation by Segment and Sector The Pythagoras COIN Application to Support the Marine Corps IW Study Final Report, dated 9 April 2008, gives the reasoning for using one agent to represent one percent of a population segment. First, it ensures that the model accurately represents the distribution between insurgency sectors amongst the population segments. It also gives an equal weight to each population segment. That is, “a small but influential group such as the Catholic Church (324 persons) has the opportunity to reach as many people as would the Urban Poor (159,510 persons)” <<Pyth_COIN_Draft Final Report.pdf>>. 4.2 Attribute Changers As the simulation progresses through time, the agent changes the proportion of its 1% population segment within each of the five possible insurgency sectors from natural drift, salience, and MAGTF influence. These influences are captured in P‐COIN via attribute changers. The documentation for Pythagoras 2.0.0 gives the exact methodology for applying attribute changers within the simulation update cycle. 4.2.1 Natural insurgency susceptibility The IW Modeling Methods Study Team provided the insurgency vulnerability data to the P‐COIN development team in the form of Markov Chain transition matrices. Data tables can be found in C.2. WernerAnderson, Inc. 11 T08‐ER07‐002 Pythagoras does not have the capability to implement a Markov Chain directly. Therefore, the P‐COIN development team chose to implement the insurgency vulnerability with incremental attribute changers. Agent attributes (percentage in the 1% population segment represented within each insurgency sector) are normalized once all attribute changers are applied. Vulnerability attribute changers are described by the population segment‐insurgency sector (initial group) that they affect. The incremental attribute changers were applied to the agents based on their initial insurgency orientation at each time step. The P‐COIN developers utilized the percentages found in the rows of the Markov Chain matrices as the values in the attribute changers, with the value of the attribute changer for the initial start state of the agent as zero: Matrix Pythagoras Attribute 1 0.0% 0 Attribute 2
0.7%
+7
Attribute 3
98.7%
0
Attribute 4 0.5% +5 Attribute 5
0.0%
0
These vulnerability changes are incremental changes each time step. The Attribute 3 change is grayed out, because it represents the neutrals staying the same. Rather than including that change in the attribute changer, this value will be adjusted when the attributes are normalized. <<AttributeExplanation.doc>> Mathematically the attribute changers work as follows: Displaced Persons Neutral Initial Attribute 1 Attribute 2
0 0 Attribute 3
1000
Attribute 4
0
Attribute 5
0
0
5
0
1000
5
0
Vulnerability Increment Values 0 7 Intermediate Step (Initial + Increment)
0 7 Final Values (Normalization) 0 7 988
5
0
<<AttributeChangeExamples.xls>> 4.2.2 Salience P‐COIN uses its communications devices to model salience between population segments and to cause relative attribute changes. These communications devices affect a relative attribute change across the duration of the simulation causing the agents receiving the message “to become a designated percent closer to the agent sending the message” <<Pyth_COIN_Draft Final Report.pdf>>. These attribute changers on the comm devices are activated at each time step implemented through the alternate behavior selection of the agents. Table 3 gives the salience data provided to the P‐COIN development team, repeated below for convenience. The P‐COIN development team implemented salience as the effect on the agents’ insurgency affiliation. The data provided to the P‐COIN development team did not account for the specific effect on an insurgency orientation rather than on the general effect that it has on the WernerAnderson, Inc. 12 T08‐ER07‐002 Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Catholic Church population segment as a whole. Rather, this data provides the influence one population segment has on another population segment (including the dynamic effect it has on itself). Therefore, some data processing was required for the P‐COIN implementation. IW Modeling Methods Study Team developed the data processing technique in conjunction with the P‐COIN development team (described in Appendix I). 0.000 0.706 0.207 0.750 0.474 0.000 0.816 0.202 0.000 (0.262) 0.000 (0.351) (0.486) (0.843) (0.623) (0.259) 0.000 (0.052) 0.425 (0.295) (0.713) (0.174) (0.387) 0.101 (0.078) (0.230) (0.053) 0.492 0.000 (0.039) 0.136 0.225 0.801 0.178 0.420 0.262 0.840 (0.273) (0.269) (0.033) (0.078) (0.366) (0.190) 0.685 (0.506) 0.767 (0.205) (0.134) 0.000 0.480 (0.467) 0.000 0.494 0.000 0.392 0.552 0.000 (0.208) 0.183 (0.257) (0.250) (0.218) 0.132 (0.079) From the table it is clear that every population segment influences every other population segment (including itself) and that the combined effect of all these influences once the MAGTF arrives is unpredicted from a strict evaluation of the data (see section 3.3). That is, the data suggests that these indirect effects from the MAGTF arrival could influence the preferred solution through their secondary and tertiary effects. P‐COIN uses its communications devices to model salience between population segments and to cause relative attribute changes. These communications devices affect a relative attribute change across the duration of the simulation causing the agents receiving the message “to become a designated percent closer to the agent sending the message” <<Pyth_COIN_Draft Final Report.pdf>>. The alternate behavior selection in the agents capability within Pythagoras activate these attribute changers on the communication devices at each time step. Attribute changers affect agents’ attributes (i.e., the percentage within the 1% population segment with an insurgency orientation from FARC to GOVT). Therefore, while the data in the salience table is independent of insurgency affiliation, Pythagoras‐COIN requires that the data be applied by insurgency affiliation. While the IW Modeling Methods Study Team provided a spreadsheet for processing this data (see Appendix A), the P‐COIN development team did not use this data processing technique when developing the data for the P‐COIN model implementation. The following documentation was provided to the validation agent describing the use of the salience data table (Table 3): Example‐Influence of the Catholic Church Average Orientations: Catholic Church = 3.62 Displaced Persons = 2.63 WernerAnderson, Inc. 13 T08‐ER07‐002 Salience 0.706 (71 %) Neutral to Neutral Salience is Calculated: = √(71) = 8% Catholic Church FARC ProFARC Neutral ProCOIN COIN Attribute 1 1% ‐ ‐ ‐ ‐ Attribute 2 ‐
1%
‐
‐
‐
Attribute 3 ‐
‐
8%
‐
‐
Attribute 4 ‐
‐
‐
71% ‐
Attribute 5 ‐
‐
‐
‐
71%
Each of these changes represents the influence the Catholic Church has in bringing the Displaced Persons attributes closer to its own. <<AttributeExplanation.doc >> In this example, since the Catholic Church is at the simulation’s initialization more Pro‐COIN than the Displaced Persons, the P‐COIN development team chose to have the Catholic Church COIN and Pro‐COIN insurgency sectors to influence the Displaced Persons COIN and Pro‐COIN attributes as a relative attribute changer. The P‐COIN development team chose to allow other insurgency sectors within the population segment to have less influence. This lesser influence is reflective, though not identical, to the methodology provided to the P‐COIN development team. Appendix C contains the adjusted data. The update of the agent attributes once the attribute changer is applied is shown below: Catholic Church ProCOIN acting on the Displaced Persons Neutral Start influencing after first vulnerability timestep [sic] So "Initial" Values Displaced Persons Neutral "Initial"
Attribute 1 Attribute 2
0 7 Catholic Church ProCOIN "Initial" 0 0 Attribute 3
988
Attribute 4
5
Attribute 5
0
0
1000
0
‐ ‐
71%
‐
7 988
711
0
Final Values (Normalization) 0 4 579
417
0
<<AttributeChangeExamples.xls >> Catholic Church Influence ‐ Displaced Persons Change 0 4.2.3 MAGTF Event Influence The following documentation was provided to the validation agent describing the use of the MAGTF event influence data table in Table 4, repeated here for convenience. WernerAnderson, Inc. 14 T08‐ER07‐002 Sea Based Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Right 0.845 0.596 0.000 0.000 0.631 0.564 0.780 0.798 Left 0.000 0.000 0.397 0.408 0.000 0.000 0.000 0.000 Shore Based Right Left 0.401 0.000 0.721 0.117 0.000 0.447 0.000 0.408 0.631 0.000 0.564 0.000 0.184 0.210 0.722 0.211 Given Data for Shore‐Based: Right (towards government) 0.721 Left (towards FARC) 0.117 Neutral to Neutral Calculated: = √(0.721 x 0.117) = 0.290 Attribute 1 x117% Attribute 2
x117% Attribute 3
x290%
Attribute 4
Attribute 5
x721%
x721%
<<AttributeExplanation.doc >> The update of the agent attributes once the attribute changer is applied is shown below: MAGTF shore action affecting displaced persons neutral "Initial" Attribute 1 0 (used values after 1 vulnerability timestep [sic] since multiplying by zero wouldn't produce anything interesting) Attribute 2
Attribute 3
Attribute 4
Attribute 5
7 988
5
0
Multiplier Values 117% 117% 290%
721%
721%
2866
36
0
985
12
0
Intermediate Step (Initial x multiplier)
0 8 Final Values (Normalization) 0 3 The MAGTF influences act as multipliers upon the current attribute values <<AttributeChangeExamples.xls >> WernerAnderson, Inc. 15 T08‐ER07‐002 5
Conceptual Model Validation The underlying conceptual model is provided in the documentation found in Appendix A and Appendix B. This validation effort focused on the implementation of the conceptual model within P‐COIN and the implications for the analysis effort. While some sections of this report (e.g., section 6.4) discuss specific aspects of the conceptual model and the limitations those aspects might pose for the analysis, a detailed analysis of the conceptual model was not undertaken in this effort. For instance, this validation exercise does not assess the appropriateness of the assumption of using a Markov chain for the model of the population drift between insurgency orientations in the population of Colombia. Another validation exercise within the ABS VV&A Framework Study explicitly looked at the validity of the conceptual model. In general, the purpose of discussing the validity of the conceptual model is not to argue every assumption made during the model development. Rather, the purpose is to assess the assumptions, the limits these assumptions provide to the analysis, and to assess whether the mitigation of these assumptions is necessary when performing the analysis. In the conceptual model, the results in the analysis depend on the influence of the MAGTF arrival and basing on the insurgency affiliation of the population segments. The natural vulnerability drift and the salience of beliefs between populations provide a backdrop and dynamic structure for that influence. That is, these items are not core to the analysis question. Rather, they provide interesting, influencing effects. However, they do provide the core dynamics for the system. As such, they are important elements of the simulation. WernerAnderson, Inc. 16 T08‐ER07‐002 6
Data Considerations This analysis effort exists in three concurrent efforts of which this validation is a piece of one. The development of the COIN scenario within Pythagoras as P‐COIN is another. The third is an effort to understand how IW can be modeled, including the data generation processes that fed the development of the data that populated the P‐COIN model. Appendix B gives the documentation describing the data generation process. Appendix C contains the data provided to the Northrop Grumman P‐COIN development team. Since one of the purposes of the third effort described above was to determine methodologies for modeling IW, this report provides no value judgment on the methodologies used to generate the data. Rather it focuses on the use of the data within P‐COIN and any mitigation strategies necessary in the analysis resulting from that use. 6.1 Salience Data The attribute changers within P‐COIN are applied via the initial orientation of the agent (1% of the population segment, initialized with a single insurgency sector). That is, as the insurgency orientation of the population segment represented by the agent changes, the population does not feel the salience influence of the new insurgency orientations. Therefore, in the example given in section 4.2.2, if an agent is initialized with a Pro‐COIN orientation, this agent always influences other agents towards the Pro‐COIN affiliation, regardless of the current state of that agent or the average insurgency of the influencing or influenced population at that time step. Therefore, there is an embedded bias against the influence of a dynamic shift in insurgency orientation. That is, if the agent under consideration at some future time step shifts from COIN to FARC in orientation and an influenced agent has the tendency to shift toward the same insurgency affiliation, should the influenced agent have a Neutral orientation, the effect of this implementation is to have the influenced agent shift towards COIN, even though the influencing agent is now affiliated with FARC. This can be seen by making a simple change in the initial orientation of an agent using the same table provided by the P‐COIN developer to the validation agent. Catholic Church ProCOIN acting on the Displaced Persons Neutral Some future time step after the affiliation changes for the influencing group (Catholic Church) So "Initial" Values Displaced Persons Neutral Agent Attribute 1 Attribute 2
Attribute 3
Attribute 4
Attribute 5
0 7 988
5
0
Catholic Church ProCOIN "Initial" with attributes at a future timestep (currently Pro‐FARC) 0 1000 0
0
0
Catholic Church Influence ‐ ‐ ‐
71%
‐
Displaced Persons Change 0 7 988
1
0
Final Values (Normalization) 0 4 992
1
0
WernerAnderson, Inc. 17 T08‐ER07‐002 As can be seen by the calculations, due to the influence changer, rather than having the Displaced Persons agent increase its Pro‐FARC orientation to be more like the Catholic Church agent, its Pro‐FARC affiliation actually is reduced. Therefore, the dynamic influence of that the population segments have on each other as the internal population segment orientation changes from its natural vulnerability drift and the influence of the MAGTF arrival is not captured in this model and the agents’ insurgency orientations are at risk of not being pushed in the correct direction once the initial orientation changes. Thereby, this potentially reduces the secondary and tertiary effects that result in emergent behavior. This is a high risk for the use of the simulation. The simulation results need to be evaluated at time step by time step basis to determine whether this cautionary situation occurs in this particular application of the model. In discussions with Northrop Grumman, it was their view that while the implementation was not perfect, it captured sufficiently the dynamics for the intended use, since “the current implementation has the influence get weaker as the percentage of the population shifts from its initial orientation to the orientation that it is drifting toward. That means that if the original influence is getting weaker, then the influenced population is more susceptible to other influences” <<email Edd Bitinas 17 June 2008>>. 6.2 Vulnerability Data The application of the vulnerability attribute changers suffers from the same difficulties in capturing the dynamic relationships as the initial orientation of the agents change over each of the time steps as the salience attribute changers. The attribute changers applied to the agent are reflective of its initial state only; they do not reflect the dynamic change over time. Therefore, secondary and tertiary effects of the MAGTF arrival are minimized. For instance, if at some time step, the agent above (Displaced Persons Neutral) drifted in its insurgency orientation such that its attributes were 0, 484, 171, 345, 0 (the result from repeated application of the vulnerability described above), then while this 1% population segment is now nearly equally distributed between Pro‐FARC and Pro‐GOVT, the natural drift of these insurgency sectors for this population segment is not applied nor felt. This reduces the dynamism of the system and has a stabilizing effect on the result. This is counter to the intent of the natural drift/vulnerability concept. The method of applying the vulnerability attribute changers to the population segments based solely on the segment’s initial orientation for the entirety of the simulation run is a risk factor for using the simulation. Further evaluation is required to determine whether this has a detrimental effect on the use of the results based on the time scale used. 6.3 MAGTF Data The MAGTF influence changers are also applied with respect to the agents’ initial orientation. Shifts in the population dynamics have no resulting effect on the influence of the MAGTF on that population. This is a risk factor for using the simulation. Further evaluation is required to determine whether this has a detrimental effect on the use of the results based on the time scale used. 6.4 Data Precision The methodology for generating the data for the P‐COIN model has little precision. This is due both to methodological means and to the number of subject matter experts available to perform the WernerAnderson, Inc. 18 T08‐ER07‐002 assessments. However, while additional experts might reduce data variability, precision of the might still be arguable. This limits the ability of any use of that data to give a predictive response. Further, data cannot be considered current beyond some reasonable timeframe not specified in the documentation or in the presence of an event that could cause a reformulation of beliefs within the populace. Moreover, the IW Modeling Methods Study Team used a Markov Chain as its base model for the natural drift. A characteristic of stationary Markov chains containing a single class is that they will reach steady state after some number of iterations. This assumption of a Markov chain means that, without any other influences in the system, the distribution between insurgency sectors across the total population, and within each population segment, eventually will be uniform across the total population, providing all of the insurgency sectors communicate with each other in the chain. Therefore, the data used to populate the model may be perishable since the assumptions of stationarity may not hold in fact for the system of interest. That is, world events may change the underlying data. Which world or local events that could cause changes in the provided data matrices are not provided. Further, the data clearly cannot be used beyond the timeframe provided to the subject matter experts in the data generation methodology. Therefore, the data could rightly be viewed with skepticism. It is also clear that the underlying data model is not valid for long term analysis of the effect of the MAGTF arrival on the population. The analysis using P‐COIN can and did incorporate mitigation methods to account for data precision difficulties. The P‐COIN development team mitigated this lack of precision by including tolerances on the data to generate a uniform distribution around the provided data values in the attribute changers for 50 runs of the simulation. These runs provide a statistical distribution for the options for each of the population segments and insurgency sectors. The vulnerability and salience values used a 25% tolerance in the implementation, while the MAGTF influence used a 10% tolerance. The differences in the chosen tolerances were due to the implementation methods chosen by the development team rather than due to any information provided to the team by the IW Modeling Methods Study Team. The tolerance levels chosen to generate the distributions of expected response should be sufficient for the analysis problem posed. However, without the full incorporation of the dynamic influences between the population segments and insurgency sectors, it is impossible to know. The analysis results as presented indicate that for the instantiation of the model chosen by the P‐COIN developers, these tolerances are sufficient to give a relative ranking based on the distribution of Non‐FARC leaning population within the population segments for the two options: ashore or afloat. 6.5 Data Processing Missing from any of the documentation provided is a verification assessment that the data processing for the salience and the MAGTF data gives the expected directional and magnitude changes in insurgency orientation. This data verification should be completed and documented. However, preliminary results assessment indicates that any discrepancies resulting from the data processing and implementation should have little effect on the answer from the analysis. WernerAnderson, Inc. 19 T08‐ER07‐002 7
Results Validation The purpose of the P‐COIN model is to support an analysis as described in section 2 of this report. The purpose of this validation effort is to assess this analysis. That is, this validation effort is an analysis of the analysis undertaken using P‐COIN. As both of these efforts occurred concurrently, there was much feedback and discovery along the way. Since the insight of this validation effort in the application of ABS to IW analysis application of validation as “an analysis of the analysis” did not come until mid‐way through the validation effort, there are some inevitable holes in this report. These holes are identified. 7.1 Analytical Results First, Pythagoras had to be able to model population dynamics. Then, in order to evaluate the model in the best possible way, the question being asked needed to be changed so that Pythagoras could formulate a credible answer. A new question was created so that only a better answer was chosen rather than attempting to determine the best course of action. Once the question was established, the measure of the effectiveness and the method of measuring were analyzed. The two measurements utilized in the analysis were that there is no increase in insurgency activity and that the MAGTF support improves the backing of the pro‐government political orientation. This was shown in a box‐and‐whisker plot that depicted the results with no MAGTF, MAGTF ashore, and MAGTF afloat. See Figure 4 and Figure 5.. Percent Population Pro-COIN, COIN
100
Percent Population
90
80
70
60
50
Displaced Persons Illicit Organizations
Old Money
Urban Middle
Class
Afloat
Ashore
NoMAGTF
Afloat
Ashore
Afloat
Police
NoMAGTF
Ashore
NoMAGTF
Afloat
Ashore
Afloat
Military
NoMAGTF
Ashore
NoMAGTF
Afloat
Ashore
NoMAGTF
Afloat
Ashore
Afloat
Catholic Church
NoMAGTF
Ashore
NoMAGTF
40
Urban Poor
Afloat has nearly equal or more Pro-COIN, COIN
Figure 4. Measure of Pro‐COIN and COIN WernerAnderson, Inc. 20 T08‐ER07‐002 Percent Population Pro-FARC, FARC
40
35
Percent Population
30
25
20
15
10
5
Catholic Church
Displaced Persons Illicit Organizations
Military
Old Money
Urban Middle Class
Afloat
Ashore
NoMAGTF
Afloat
Ashore
Afloat
Police
NoMAGTF
Ashore
Afloat
NoMAGTF
Ashore
NoMAGTF
Afloat
Ashore
NoMAGTF
Afloat
Ashore
Afloat
NoMAGTF
Ashore
NoMAGTF
Afloat
Ashore
NoMAGTF
0
Urban Poor
Afloat has the same or fewer Pro-FARC, FARC
Figure 5. Measure of Pro‐ FARC and FARC 7.2 Validity Assessment Although trajectories were provided to the validator for early runs of the P‐COIN model, these were not provided for the final P‐COIN instantiation. Further, the data sheet provided did not include sufficient information to generate them. The only information provided was an excel spreadsheet used to generate the above charts, with consolidated data, and the charts provided. Therefore, there was no way to assess whether the results were an accurate representation of what would be expected based on the data provided. Therefore, there was no ability to evaluate whether the salience implementation moved the population segments in the expected direction or whether the order of magnitude of influence was expectedly proportional to the starting values. Neither were similar evaluations for the vulnerability or the MAGTF influence able to be completed. Although asked, the developer did not provide verification data that these assessments were made. 7.3 Accuracy Assessment Evaluating the input data tables, the result that Afloat is a better result than Ashore makes sense. MAGTF influences are move the population segments in identical directions for nearly of the population segments with Afloat having stronger values. The only concern is in the divergent directional changes between the salience and the MAGTF with no indication in the referent data or in the testing that identifies the expected secondary and tertiary effects. So while the military moves towards the FARC with the arrival of the MAGTF, it is highly influenced by the Catholic Church, which moves strongly to the government. Nothing in the analysis or results provided allowed understanding of whether these contradictory influences were accurately captured in the model. Further verification of the model is required as well as limited test cases to assess whether expected results occur (magnitude and direction) in the model as instantiated by P‐COIN. WernerAnderson, Inc. 21 T08‐ER07‐002 WernerAnderson, Inc. 22 T08‐ER07‐002 8
Recommendations and Conclusions Below are the recommendations and conclusions from this validation effort. 8.1 Caveats and Limitations The P‐COIN simulation fails to capture the dynamic effects intended in the conceptual model of the insurgency in Colombia provided to the P‐COIN developer. That is, P‐COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF. The data supporting the P‐COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming‐up” the Markov chains supporting the data used to build the P‐COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change. 8.2 Cautions The P‐COIN model should not be used to evaluate long term effects on the population resulting from the influencing event of the MAGTF arrival. This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs. 8.3 Validity with Respect to Intended Use Subject to the caveats, limitations, and cautions listed above, P‐COIN can answer question as long as it is evaluated on a relative scale rather than on an absolute or predictive scale. While there is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure, P‐COIN also provides little insight into the ashore or afloat question in its current implementation. Changes would be required to fully capture the dynamics to enable this insight. 8.4 Summary A summary of the conclusions of the validation assessments made to P‐COIN are as follows: 1) The P‐COIN simulation fails to capture the dynamic effects intended in the conceptual model of the insurgency in Colombia provided to the P‐COIN developer. That is, P‐COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF. 2) The data supporting the P‐COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming‐up” the Markov chains supporting the data used to build the P‐COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change. 3) The P‐COIN model should not be used to evaluate long term effects on the population resulting from the influencing event of the MAGTF arrival. WernerAnderson, Inc. 23 T08‐ER07‐002 4) This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs. 5) There is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure. However, item 1 implies that P‐COIN also provides little insight into the ashore or afloat question in its current implementation. 8.5 Recommendations Although the results of the analysis reflect the expectations of the analysis based on the input data tables and the analysis does not recommend a change in course of action, it is difficult to trust the results without being able to trust underlying dynamics of the model. The chosen instantiation of only applying attribute changers reflective of the start state of the agent population distribution should be thoroughly evaluated with the recommendation to apply these attribute changers more robustly to reflect how the population insurgency affiliation changes over time and in response to system events. The full dynamics intended by the use of salience and the natural drift aspects of the population segments ought to be included in the P‐COIN model in order to allow for the emergence of secondary and tertiary effects of the influencing event of the MAGTF arrival. The developer should also develop robust test cases to evaluate the dynamic behavior in isolation to gain surety that the combined dynamic behavior should be trusted. Further, the following material would have been useful in the validation assessment: 1)
2)
3)
4)
5)
6)
Better documentation on the P‐COIN instantiation Time series data A descriptive walk‐thru of results charts (meaning & implications) Verification cases (isolated effects) to ensure dynamics have expected direction (first derivative) and order of magnitude with descriptions of what was believed to be correct Better explanations of expected resulting effects from data values in the referent as most had to be inferred and order of magnitude differences unknown Expected interaction effects WernerAnderson, Inc. 24 T08‐ER07‐002 Appendix A
P‐COIN Support to the Irregular Warfare Study “The Pythagoras Counterinsurgency (COIN) Application to Support the Marine Corps Irregular Warfare (IW) Study” report by Northrop Grumman describes the P‐COIN model development and the IW analysis completed using the IW model. The latest draft of this report provided to the validation agent is dated 9 April 2008, << Pyth_COIN_Draft Final Report.pdf>>. A draft of Appendix N to this report, << Pyth_COIN_FinalFINAL_APP_N (BKS).doc >>, which contains a description of the analysis completed, was provided to the validation agent as a courtesy on 17 June 2008. This appendix contains both reports in the CD version of this report. WernerAnderson, Inc. A‐1 T08‐ER07‐002 Appendix B
Irregular Warfare Project Description Two documents give an overview of the data generation methodology used to support the P‐COIN model development: ƒ
ƒ
“The Analysis of Irregular Warfare: Using a Realistic Example to Devise a Useful Methodology,” dated 28 March 2008, <<OAD_IW_study_draft.doc>> “Irregular Warfare Project,” a briefing given during Workshop 2 during Phase II of the ABS VV&A Framework Study, <<MCCDC_OADS_IW_Leavenworth_brief.ppt>> The CD version of this report contains these documents. WernerAnderson, Inc. B‐1 T08‐ER07‐002 Appendix C
Conceptual Model Data This appendix contains model data used to support the instantiation of the P‐COIN model. This appendix duplicates some of the data tables contained in the body of this report for ease of reference. C.1
Scenario Population Data This section contains the population data for the Colombia scenario. ID 1 2 3 4 5 6 7 8 Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Total People 324 27,882 324 973 324 648 134,222 159,510 324,207 % Population 0.10% 8.60% 0.10% 0.30% 0.10% 0.20% 41.40% 49.20% 100.00% Table 6. Distribution of Columbian People across Population Segments Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor FARC 0.00%
6.88%
0.00%
1.61%
0.00%
0.00%
3.75%
5.80%
Pro‐FARC 0.00%
36.64%
27.65%
9.68%
2.08%
5.26%
3.75%
9.17%
Neutral 38.46%
44.70%
39.02%
0.00%
4.17%
0.00%
62.50%
65.75%
Pro‐GOVT 61.54%
10.29%
18.18%
88.71%
62.50%
94.74%
10.00%
15.12%
GOVT 0.00%
1.50%
15.15%
0.00%
31.25%
0.00%
20.00%
4.16%
Table 7. Population Distribution across Segments and Sectors WernerAnderson, Inc. C‐1 T08‐ER07‐002 Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Total FARC Pro‐FARC 19 30
1,919 10,215
5 31
0 51
0 90
0 14
0 0
5,982 5,982
7,924 16,412
Neutral 213
12,462
0
0
126
27
51,624
99,694
164,146
Pro‐GOVT 49
2,869
287
922
59
405
82,598
15,951
103,140
GOVT 13 417 0 0 49 203 0 31,902 32,584 Total 324
27,882
324
973
324
648
134,222
159,510
324,207
Table 8. Number of People in Each Sector by Population Segment (Derived) C.2
Agent Allocation This section contains the agent allocation used to instantiate the Pythagoras‐COIN model. Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Total FARC 0 7 0 2 0 0 4 6 19 Pro‐FARC 0 37 39 10 2 5 4 9 106 Neutral 38 45 28 0 4 0 62 66 243 Pro‐GOVT 0 10 18 88 63 95 10 15 299 Table 9. Initial Agent Allocation by Segment and Sector (
GOVT 62 1 15 0 31 0 20 4 133 ) C.3 Salience The application sponsor provided salience data in two forms. First, a salience matrix was provided. Second, a derivation method from that data was provided. Appendix C.4.1 gives the salience matrix, while Appendix Error! Reference source not found. WernerAnderson, Inc. C‐2 T08‐ER07‐002 C.4
Scenario Vulnerability Data . Notice, that due to round Below are the vulnerability matrices for each population segment, off error or other factors, some rows add up to something other than one. Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 100.000%
0.000%
0.020%
0.000%
0.000%
Pro‐FARC 0.000% 100.000% 0.000% 0.020% 0.000% Neutral 0.002%
0.000%
100.000%
0.000%
0.003%
Pro‐GOVT 0.000% 0.023% 0.000% 100.000% 0.000% GOVT 0.000% 0.000% 0.015% 0.000% 100.000% Table 10. Catholic Church Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 99.849%
0.355%
0.035%
0.000%
0.000%
Pro‐FARC 0.150% 99.470% 0.728% 0.020% 0.100% Neutral 0.001%
0.153%
98.729%
0.856%
0.002%
Pro‐GOVT 0.000% 0.023% 0.481% 99.074% 0.149% GOVT 0.000% 0.000% 0.026% 0.050% 99.749% Table 11. Displaced Persons Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 99.364%
0.000%
0.019%
0.005%
0.000%
Pro‐FARC 0.055% 95.711% 4.575% 0.501% 0.000% Neutral 0.050%
3.791%
94.165%
0.300%
0.983%
Pro‐GOVT 0.494% 0.263% 0.761% 98.363% 0.050% GOVT 0.037% 0.235% 0.480% 0.830% 98.967% Table 12. Illicit Organizations Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 99.800%
0.302%
0.020%
0.000%
0.000%
Pro‐FARC 0.200% 99.675% 0.199% 0.202% 0.000% Neutral 0.000%
0.000%
99.666%
0.000%
0.000%
Pro‐GOVT 0.000% 0.023% 0.100% 99.798% 0.000% GOVT 0.000% 0.000% 0.015% 0.000% 100.000% Table 13. Military Vulnerability Matrix WernerAnderson, Inc. C‐3 T08‐ER07‐002 Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 100.000%
0.000%
0.020%
0.000%
0.000%
Pro‐FARC 0.000% 99.198% 0.100% 0.020% 0.000% Neutral 0.000%
0.701%
99.666%
0.000%
0.000%
Pro‐GOVT 0.000% 0.100% 0.199% 98.980% 0.100% GOVT 0.000% 0.000% 0.015% 1.000% 99.900% Pro‐GOVT 0.000% 1.000% 0.000% 99.980% 0.000% GOVT 0.000% 0.000% 0.015% 0.000% 100.000% Pro‐GOVT 0.000% 0.023% 1.531% 99.472% 0.000% GOVT 0.000% 0.000% 0.015% 0.000% 99.997% Table 14. Old Money Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 100.000%
0.000%
0.020%
0.000%
0.000%
Pro‐FARC 0.000% 99.000% 0.000% 0.020% 0.000% Neutral 0.000%
0.000%
99.965%
0.000%
0.000%
Table 15. Police Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 99.403%
0.000%
0.020%
0.000%
0.000%
Pro‐FARC 0.498% 99.467% 0.000% 0.020% 0.000% Neutral 0.100%
0.510%
98.434%
0.508%
0.003%
Table 16. Urban Middle Class Vulnerability Matrix Sector FARC Pro‐FARC Neutral Pro‐GOVT GOVT FARC 99.354%
0.150%
0.020%
0.000%
0.000%
Pro‐FARC 0.348% 97.996% 0.407% 0.020% 0.000% Neutral 0.249%
1.345%
99.075%
1.200%
0.100%
Pro‐GOVT 0.050% 0.509% 0.482% 98.529% 0.100% GOVT 0.000% 0.000% 0.015% 0.250% 99.800% Table 17. Urban Poor Vulnerability Matrix WernerAnderson, Inc. C‐4 T08‐ER07‐002 Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Catholic Church C.4.1 Salience matrix Below is the salience matrix between population segments. 0.000 0.706 0.207 0.750 0.474 0.000 0.816 0.202 0.000 (0.262) 0.000 (0.351) (0.486) (0.843) (0.623) (0.259) 0.000 (0.052) 0.425 (0.295) (0.713) (0.174) (0.387) 0.101 (0.078) (0.230) (0.053) 0.492 0.000 (0.039) 0.136 0.225 0.801 0.178 0.420 0.262 0.840 (0.273) (0.269) (0.033) (0.078) (0.366) (0.190) 0.685 (0.506) 0.767 (0.205) (0.134) 0.000 0.480 (0.467) 0.000 0.494 0.000 0.392 0.552 0.000 (0.208) 0.183 (0.257) (0.250) (0.218) 0.132 (0.079) Table 18. Salience Values for Colombian Scenario C.5
MAGTF Influence Estimation Sea Based Segment Catholic Church Displaced Persons Illicit Organizations Military Old Money Police Urban Middle Class Urban Poor Right 0.845 0.596 0.000 0.000 0.631 0.564 0.780 0.798 Left 0.000 0.000 0.397 0.408 0.000 0.000 0.000 0.000 Shore Based Right Left 0.401 0.000 0.721 0.117 0.000 0.447 0.000 0.408 0.631 0.000 0.564 0.000 0.184 0.210 0.722 0.211 Table 19. MAGTF Influence Estimation Data WernerAnderson, Inc. C‐5 T08‐ER07‐002 Appendix D
Instantiated Model The P‐COIN analysis required three XML files: ƒ
ƒ
ƒ
<<NoMAGTF48timesteps.xml>> – gives the base case without the MAGTF <<SeaVsAllMAGTFleaves.xml>> – gives the sea‐based results <<ShoreVsAllMAGTFleaves.xml>> – gives the shore based results The CD version of this report contains these files. In addition, the file <<Basic Scenario Set Up (Interacting Segments).xlsx>> was used to verify an early test version of the scenario files, <<InteractingSegments.xml>>, to ensure that the attribute changers were instantiated as intended by the developer. Both these files are also on the CD version of this report. While some errors were found in the test version of the scenario file, the corrections noted in the Excel verification file were made. WernerAnderson, Inc. D‐1 T08‐ER07‐002 Appendix E
Descriptions of Attribute Changers The P‐COIN development team provided the following documentation describing the attribute changers using in P‐COIN: ƒ
ƒ
<<AttributeExplanation.doc>> – gives a description of the attribute changers <<AttributeChangeExamples.xls> – demonstrates the calculations The CD version of this report contains these documents. WernerAnderson, Inc. E‐1 T08‐ER07‐002 Appendix F
Results from the P‐COIN Analysis The CD version of this report contains the results from the P‐COIN analysis, <<BoxAndWhiskerNumber2.xls>>. WernerAnderson, Inc. F‐1 T08‐ER07‐002 Appendix G
Pythagoras Usage, Validation, and Accreditation Histories The Pythagoras modeling system can best be thought of as a collection of building blocks that represent objects with specific functions. Weapons represent objects that can kill, suppress or otherwise change another agent. Sensors detect other agents. Communication devices send information and messages to other agents. The agents themselves can carry and use these other objects and have a set of behaviors, including movement and engagement desires that govern their actions and interactions with other agents. Because these capabilities can be assembled in nearly infinite combinations, resulting in representations of complex objects that vary from poor through adequate to nearly complete, and those assemblies are application specific, general verification and validation of Pythagoras is not possible. What is possible, and has been accomplished, is the verification and validation (V&V) of the building blocks themselves. The developers have conducted this type of V&V as part of unit testing. The approach used was to state the capability with sufficient detail to allow the tester to understand the intent of the capability. Since most of these capabilities are simple and straightforward, this part is usually quite easy to accomplish. For example, a movement desire may be for one agent to follow his leader, or a more complex test may be to first have a group of agents follow a hierarchal leader, and then to follow a charismatic leader. Next, the tester develops a simple scenario that will demonstrate this capability. When the agents first follow their leaders in a hierarchy (a line from greatest leader to worst leader), and then switch to only following the single greatest leader in a mob, the test is completed and passed, validating/verifying the capability. Some of the test articles, which consist of simple scenarios, are provided along with the Pythagoras system so that new users that want to include a specific capability or want to better understand how a specific capability works. Naval Post‐graduate School (NPS) students and others have used the Pythagoras software in numerous applications. Descriptions of the NPS can be found by searching the NPS website. No usage, validation, or accreditation histories were provided to the validation agent. WernerAnderson, Inc. G‐1 T08‐ER07‐002 Appendix H
Pythagoras‐COIN Verification during Development Northrop Grumman was asked to provide a description of the verification process used to establish the methodology for applying Pythagoras in Pythagoras‐COIN. That is, the reasoning for the use of the different attribute changers, the data applied within those attribute changers as related to the data provided in the conceptual model, and the testing done to ensure that the chosen applications worked as expected. These descriptions were not provided. Informal interviews were conducted and the Interim Reports containing some of the reasoning processes were provided. In addition, the validation agent, in early phases of the ABS VV&A Framework Study Phase II, examined the underlying mathematics of P‐COIN as compared to the expected mathematics from the described conceptual model. Therefore, detailed verification is not available for this report. However, limited discussion is provided below. H.1 Underlying Mathematics The underlying mathematics expected from the descriptions of the conceptual model do not match that in the implementation of P‐COIN, especially that of a Markov chain as a model for the natural drift of the insurgency vulnerability of the population segments. The mathematical implementation and the differences between the P‐COIN implementation and the conceptual model are provided in Appendix I. This assessment is not complete due to the redirection of the effort mid‐way through the ABS VV&A Framework Study Phase II when it was determined that a thorough assessment of this type was unnecessary. Since the intent is not to model the “math” but rather to model the relationships between the factors capturing the desired trends, the differences in the underlying mathematics is not seen as a difficulty within the analysis application. Further, the mathematics does not drive result as each alternative uses the same underlying mathematics. The actual numerical result of the population distributions in each insurgency sector is not of interest, rather the trends from the influencing factors is of interest. The implementation in P‐COIN is sufficient to capture the desired effect. Finally, P‐COIN’s not directly implementing the mathematics of the Markov Chain mitigates the inherent limitations of a Markov chain given by the necessary assumptions of a Markov Chain. However, P‐COIN still cannot be used for long term without revision to the vulnerability attribute changers post‐MAGTF. H.2 Other Verification Other material supporting the modeling choices made by the P‐COIN team, such as the agent distribution, can be found in <<Pyth_COIN_IR4_Final.pdf>>. To determine how the variables for the COIN model might interact, the Study Team implemented the conceptual model using Microsoft ® Excel. This model was then recreated using Java™ code and Microsoft ® Excel spreadsheets as postprocessors. Troubled country scenario The Study Team, in concert with the Study Sponsor, decided that a two‐sided model in which both sides (the insurgents and the COIN) are controlled by decision rules and general guidelines would be appropriate. The events would include MAGTF and Insurgency actions. Messages about these events would originate in one or more population segments and may or may not be received by other population segments. The Study Team has acknowledged that physical proximity can affect message dissemination; however, physical movement of all population segments was deemed unnecessary for the modeling of WernerAnderson, Inc. H‐1 T08‐ER07‐002 this scenario. The Study Team used a spiral development process to model the scenario in Pythagoras based on the conceptual model. The Pythagoras scenario was first built with just two population segments (including their four orientations: insurgent, pro‐insurgent, indifferent and pro‐COIN) so that the Study Team could gain a strong understanding of the population segment interactions and the effects of actions on the population segments. This was then extended to include an additional two population segments. WernerAnderson, Inc. H‐2 T08‐ER07‐002 Appendix I
Validation Documentation Prior to Workshop 2 Workshop 2 of the ABS VV&A Framework Study Phase II provided significant insight into the project and resulted in significant redirection of the work done in support of the validation analysis conducted against P‐COIN. Prior to the workshop, the validation agent was emphasizing the mathematical representational accuracy between the conceptual model documentation and the implementation of that conceptual model with P‐COIN. The validation agent was evaluating the degree of the match between the two representations, and the implications of the differences in the mathematical representations. Further, the validation agent sought to discover whether there was a range of data input for which the P‐COIN could validly serve as a simulation tool. That is, was there a class of problems similar to that described by P‐COIN for which P‐COIN could be used. The workshop, however, brought to light that the application of interest was not a class of problems similar to the Colombia scenario with the representation of vulnerability, salience, and an influencing event. Rather, the application of interest was the specific question at hand and its accompanying analysis: ashore or afloat. With this redirection, much of the preliminary work was rendered moot. The preliminary draft of this work is included in this appendix for completeness to illustrate the redirection of thought in the process as well as to serve as a guide, albeit in draft form, for a potential future validation analysis of the type originally intended by the validation agent. This document, <<T08‐ER07‐002 v 1.0 _Pythagoras‐
COIN Validation Report_ 20080309.pdf>>, is on the CD version of this report. WernerAnderson, Inc. I‐1 T08‐ER07‐002 Appendix J
Acronyms ABS Agent‐Based Simulation AOA Amphibious Operations Area AUC Autodefensas Unidas de Colombia (United Self‐Defense Forces of Colombia) CB Caribbean Breeze COIN Counter‐Insurgency FARC Revolutionary Armed Forces of Colombia GOVT Government HA/DR Humanitarian Assistance/Disaster Relief IW Irregular Warfare MAGTF Marine Air‐Ground Task Force MCCDC Marine Corps Combat Development Center MEB Marine Expeditionary Brigade MEU Marine Expeditionary Unit USAID United States Agency for International Development USMC United States Marine Corps V&V Verification and Validation VV&A Verification, Validation, and Accreditation WernerAnderson, Inc. J‐1