Developing & Maintaining
Dynamic Micro-Simulation
Models at DWP
Sally Edwards
Simon Gault
ESRC – 2nd April 2009
1
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
2
Model Development Unit
Strategy Directorate
Introductions
Simon Gault
 Statistician, with a strong background in Economics
 Extensive experience in developing models
 Manages the Forecasting Simulation models team
Sally Edwards
 Systems / Business Analyst
 Predominantly worked in the private sector on large scale IT systems
 Manage the Genesis Simulation Model
Members of a the Model Development Unit (MDU) which has
responsibility for Micro-Simulation Models at DWP
MDU consists of 20+ people
3
Model Development Unit
Strategy Directorate
Micro-Simulation Models
 Dynamic Micro-Simulation Models, based on our standard
architecture, Genesis:
 Pensim2 private & state pension income
 Inform Integrated Forecasting model for working age
benefit claimants
 Individual Benefit Forecasting for RP, IB, DLA/AA
 Employment Model, currently being validated
 Static Micro-Simulation Models:
 Policy Simulation Model (PSM)
 Employment and Hours Model (EHM)
4
Model Development Unit
Strategy Directorate
Key factors taken into Consideration during
Development
Key factors considered when we designed our first dynamic micro-simulation
model, Pensim2 - also valid for our other models:
 Simplify the complexity of the model as much as possible, using a modular
approach and allowing a simple prototype model to be developed quickly
 Several developers work on each model at the same time
 Large staff turnover
 Models are intended for long term use (e.g. Pensim2 anticipated to have
15-20 year life) so most of the people using/maintaining the models are
not involved in their development
 Clear documentation, processes, training and user guides are essential
 Excel and SAS are the preferred tools, as both are well understood and
extensively used among the Analysts at DWP
5
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives for Pensim2 (first DMS model)
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
6
Model Development Unit
Strategy Directorate
Original Objectives for Pensim2
Pensim2 Feasibility Study document (2002) – Objectives
1. Flexibility
2. Robustness
3. Transparency
4. User-friendliness
5. Speed
6. Reliable Output
7. Availability
(i.e. desktop model)
8. Efficiency (in use of memory)
9. Timeliness
10. Done with own expertise
11. Done with own resources
12. Ease of handover
13. Ease of maintenance
14. Independence of base data
7
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives:
8
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives:
Excel based front end – Pensim2
 User friendly, transparent code in a standard format
 Easy to understand and flexible, enabling analysts to change
parameters & structure of model without understanding the
underlying code.
 Easy to maintain and handover
 Allowed independence of base data
9
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives: :
Excel based front end – Pensim2
 User friendly, transparent code in a standard format
 Easy to understand and flexible, enabling analysts to change
parameters & structure of model without understanding the
underlying code.
 Easy to maintain and handover
 Allowed independence of base data
SAS code generator - Genesis
 Developed with our own expertise and own resources
 Robust generator, producing reliable outputs
10
Model Development Unit
Strategy Directorate
Basic structure of Genesis
Dynamic Micro-Simulation Models at DWP
- Standard Excel sheets
used to define the model
GENESIS Model Engine
Simulation
Model
parameter
sheets (Excel)
SAS Base
Data
generates SAS model
and runs the simulation
SAS postsimulation
Data
SAS Static
code
- SAS chosen for the model engine and the output data,
as SAS is the standard data analysis tool used
11
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
12
Model Development Unit
Strategy Directorate
Development Standards
Two separate development teams, with different
development standards adopted
Genesis Model Engine
 Developed by IT staff
 Used standard IT project development procedures
Pensim2 Analytical team
 Economists and Statisticians
 Less structured approach to development
13
Model Development Unit
Strategy Directorate
Genesis Model Engine Development Standards
Highly structured approach to the development
 Project Management Protocol based on Prince2 methodology, although not
all aspects were needed
 Documentation – 2 types
 Development documentation produced at each stage of project, that
was signed off and used as input to the next stage
 Key Project documentation (*) - maintained and kept up-to-date with
each new release
 Development Processes –simple process diagrams to provide clear
guidelines, particularly useful for new staff
 Detailed Requirements – explaining the full functionality of the model
engine. Document translates Economist language into IT language (*)
14
Model Development Unit
Strategy Directorate
Genesis Model Engine Development Standards
 Design Diagrams and documentation – used in development stage only
 Change Control process –changes estimated and impacted before being
accepted/rejected/on hold (*)
 Problem Log process –problems recorded, prioritised, fixed/removed
from the scope/user error (*)
 Validation Process - detailed Test Plan based on Requirements document,
used to ensure that each requirement was fully satisfied
 Test Pack ensured existing functionality continued to work when changes
were made to the model engine (*)
 User Guide contains details of functionality written in simple format (*)
 Training materials, inc presentations and training text for self-study (*)
15
Model Development Unit
Strategy Directorate
Pensim2 Model Development
 Pensim2 Analytical development followed a less structured
approach
 Generally this was appropriate, as detailed requirements were not
possible to establish up-front
 Effort was predominantly spent on regression analysis
 Analysis results reviewed to determine the key components to
include and the most appropriate sources for the assumptions
 Using the standard Genesis templates, the Pensim2 model was
more easy to develop, rather than if a traditional DMS model had
been built
16
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
17
Model Development Unit
Strategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
No Hard Coding Rule which allows complete flexibility for:
• Structure of the simulation, i.e. order of events processed
• Data structures, i.e. data tables, data relationships and variables
• Parameters, e.g. indexes, rates, dates
• Regression Equations
• Selection filters
• Static SAS code, if required (e.g. addition of new cases to a model)
• Ability to add / remove modules easily
Each model is defined in simple Excel spreadsheets, using standard templates,
that users can easily understand and modify
18
Model Development Unit
Strategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
Data Access routines
 Enable easy access to data so user does not have to write complex code
 Default data access will process the current object in the current simulation
year
 Facility to process a variable belonging to a related object, e.g. partner PAR;
Relationships are not hard-coded and new relationships can easily be
defined
 Data manipulation routines provide commonly required data facilities,
e.g. “maximum value during past 10 years” = MA10
 E.g. MA10_PAR_pa_salary returns the maximum value in the past 10 years of
the salary variable for the individual’s partner, taken from the pa table
Keep Equations simple
 Equations, filters & process flows are defined in a simple, standard format,
that is unambiguous. Hence the models are easy to maintain
19
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
20
Model Development Unit
Strategy Directorate
Change Management
 Change Management Process is followed for all models
 Every change goes through the formal Change Control procedure, i.e.
requirements documented, impact assessed, agreed by users & signed off
 Change Register listing all changes, each with a separate reference number
 For each change to the model, a separate Change Control form is produced
 The model code contains the Change ref number, with a comment
 Some changes are repeated regularly, e.g. assumptions, so clear
documentation is helpful for the repeat change
 Questions arise after a change has been implemented - these can be easily
answered by referencing the Change Control documentation
21
Model Development Unit
Strategy Directorate
Problem Management
 Problem Management Process is followed for all models
 Every problem/error discovered in a model goes through the formal
Problem Log procedure, i.e. requirements documented, impact is assessed,
agreed by users & signed off
 Problem Log Register, listing all problems identified, including those that
turn out to be User Errors or misunderstandings; PL ref number assigned
 For each problem recorded, a separate Problem Log form is produced
 Users are notified of significant problems that affect the model results
 Many of the problems are minor and are picked up by the development
team – these are recorded formally, even if they do not impact the results
22
Model Development Unit
Strategy Directorate
User Model
 Our models fall into 2 categories:
 Maintained & owned by the Model Development Unit
 Developed by Model Development Unit; handed over to User
team, who then own and maintain the model
 Models owned by the Model Development Unit have a User Group
and/or a Steering Group consisting of representatives from each
team using the model
 The purpose of these groups is to agree the content, priority and
timing of changes made to the model
23
Model Development Unit
Strategy Directorate
User Group Purpose
Specifically for Pensim2, our largest model:
 Change Requests and Problem Logs are reviewed by the User
Group, where appropriate
 A key user sponsor is assigned to each change. The sponsor
ensures that the change is accurately specified & will sign-off
testing
 The User Group are responsible for signing off new versions of
the model before they are released
 The users also have a separate User Forum where they present
analysis carried out on the model outputs and explain how the
results have been used
24
Model Development Unit
Strategy Directorate
Release Management
Again, specifically for Pensim2:
 New Releases are implemented approximately every 6 to 9 months
 Each Release includes one or two key components, e.g. Private Pension
Reform, and a bundle of smaller changes and problem fixes
 The timing and content of each Release are agreed with the User Group
 When a new Release is built:
• Each change/fix is added to the model one at a time
• Hence each change is reviewed and signed off separately
• Enables us to fully understand and agree the effect of each change.
• Time consuming but gives confidence in the results
 Clear audit trail
 All previous versions of the model remain available
25
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development standards
4. Simplifying Complexity
5. Maintenance protocol
6. What worked well and Lessons Learned
26
Model Development Unit
Strategy Directorate
Original Objectives Satisfied
Pensim2 Feasibility Study document (2002) – Objectives
1. Flexibility
2. Robustness
3. Transparency
4. User-friendliness
5. Speed
6. Reliable Output
7. Availability
(i.e. desktop model)
8. Efficiency (in use of memory)
9. Timeliness
10. Done with own expertise
11. Done with own resources
12. Ease of handover
13. Ease of maintenance
14. Independence of base data
27
Model Development Unit
Strategy Directorate
Objectives Partially Satisfied
Pensim2 Feasibility Study document (2002) – Objectives
1. Flexibility
2. Robustness
3. Transparency
4. User-friendliness
5. Speed
6. Reliable Output
7. Availability
(i.e. desktop model)
8. Efficiency (in use of memory)
9. Timeliness
10. Done with own expertise
11. Done with own resources
12. Ease of handover
13. Ease of maintenance
14. Independence of base data
28
Model Development Unit
Strategy Directorate
Objectives that proved more tricky
Pensim2 Feasibility Study document (2002) – Objectives
1. Flexibility
2. Robustness
3. Transparency
4. User-friendliness
5. Speed
6. Reliable Output
7. Availability
(i.e. desktop model)
8. Efficiency (in use of memory)
9. Timeliness
10. Done with own expertise
11. Done with own resources
12. Ease of handover
13. Ease of maintenance
14. Independence of base data
29
Model Development Unit
Strategy Directorate
What went well
 Generic framework has proved successful and easy to use
 Genesis architecture has been remarkably robust
 Modules have been reused/copied from one model to another
 Quicker development of new dynamic micro-simulation models,
compared with a traditional build
 Automated documentation facility and error checking produced
using VBA, listing where variables are used and providing an
overview of each module, to ensure consistency
 Lessons learnt from Pensim2 development were used when
developing subsequent models
30
Model Development Unit
Strategy Directorate
Lessons Learned
Key lessons learnt from Pensim2 development:
 Tried to make the model too complicated – particularly the analytical side
 We should have developed a set of simple modules for a Phase 1 delivery,
then added complexity to the more important modules for Phase 2
 The structure of some of our modules was more complicated than the data
would support, e.g. Labour Market process
 Insufficient documentation produced by the Analytical team during the
development (e.g. Base Data Creation)
 Genesis Engine code is difficult to understand and hence modify, although
it is robust and does not require much modification
 Some Genesis Engine functionality was over-ambitious and subsequently
dropped (e.g. Alignment of polychotomous variables)
31
Model Development Unit
Strategy Directorate
Any questions?
32
Model Development Unit
Strategy Directorate