Model and assumption development

To develop assumptions for modelling policies in the Data Observatory, evidence from across academic studies, government and industry reports and public data are synthesised.

Appraisal method overview

Ex ante appraisal of policy options in the Data Observatory is implemented through a system dynamics (SD) modeling methodology. SD models are embedded in a systems thinking approach (McGonigle et al. 2021). They are used to simulate, through calculus, causal effects and feedback loops (both reinforcing and balancing), between elements of linked social and physical systems. SD’s “ontological basis relies on the concept that socio-technical systems can be modelled and studied as”information feedback control systems”” (Freeman, Yearworth and Cherruault, 2014, p. 3-14). These computerized models can support policy analysis and decision making (Duggan, 2016).

The SD model combines an overarching material flow layer (further extended by emissions factors) and sub-models representing key dynamics, including of the shortlisted policies. These sub-models link to the material flow layer by directly and indirectly affecting flows and stocks within it. As part of the modelling framework, the impacts of a given shortlisted policy instrument (and scenarios) is a function of:

Where these are otherwise not predefined, user-supplied parameters for shortlisted policies identified through a workshop, including:

  • When a lever is introduced
  • Where applicable, at what level a policy is set e.g. the level of a charge
  • The scope of a lever, particularly:
    • Geographical scope
    • Institutional scope i.e. to which institutional or industrial actor the lever would apply

Relationships and processes captured within the SD model, defined by a literature-based systems mapping of causal and feedback loops subsequently formalised within the model through mathematical functions. Identified dominant relationships and processes inform areas of focus as part of literature review on sub-model components and policies.

Potential effect sizes of policies estimated through an evidence review and synthesis of the literature, accounting for:

  • Likely penetration rates within the target group, for instance, a voluntary approach could be expected to have a lesser participation rate than a mandatory approach. 1 This may also vary by the level of an instrument such as a charge
  • The potential immediacy of impact following introduction of a policy
  • Long-run effects such as spurring innovation which may serve to alter relationships between modelled variables through time as well as resistance to external changes like inflation of a policy
  • Interaction effects between instruments, including accounting for their sequencing.

Model structure

Material flow layer

flowchart LR
  A(Net Trade) --> B(Placed on market)
  C(Domestic production) --> B(Placed on market/<br>sales)
  B-- Product lifetime --> D(Waste generated)
  D --> E(Waste collected)
  D --> F(Littering)
  E --> G(Formal domestic treatment)
  E ---> H(Treated overseas)
  G --> I(Recycling)
  G --> J(Residual treatment)
  J --> K(Incineration)
  J --> L(Landfill)
  I --> M(Mechanical)
  I --> N(Chemical)
  E --> O(Mismanaged e.g. <br> dumping)

Transfer coefficients used in the dynamic material flow model, derive from the static material flow analysis. Some variables are calculated as a residual through a mass-balance approach (e.g. littering). The table below shows for which variables captured within the material flow model, against-baseline change as a result of policy being applied link directly to Defra strategic ambitions and UK government targets:

  • Waste generated/residual treatment - zero avoidable (plastic) waste by 2042
  • Littering - Achieving the elimination of plastic pollution by 2040

Sub-models

  • Essence of the sub-model revolves around the ratio between price of virgin/secondary

  • Demand/supply facilitated by price dynamics

  • If demand for recyclate goes up, then there is a displacement on virgin (Virgin production capacity - 2016 SIC)

  • International market - other demand source

  • Need to add markets for each of the nodes of the value chain

  • Other modules will interface with these markets

  • User-side parameter here is a slider regulating recycled content

Policy sub-models

  • Targeted product: Drinks containers, variations to this including ‘on the go format’
  • Targeted materials: Plastic, Metal, (Glass in some regions)
  • Geography of introduction: Cross-UK
  • Planned date of introduction: 2027
  • Policy type: Price-based (charge-rebate)
  • Level of front-end fee: 20p
  • Functions through:
    • For each in-target unit placed on the market, a charge is levied on a seller by a Deposit Management Organisation(s). Consumer pay a deposit at the time of purchase also. The consumer can get a rebate on the charge by returning items to a reverse vending machine. If the value is unredeemed, it stay in the DMO system. A logistics network oversees transport from RVMs to bulking centres. Reuse would require cleaning facilities.
  • Anticipated impacts (and confidence):
    • Reduction in littering rates (high)
    • Relative increase in single-stream collection rates of target materials (high)
    • Improved recycling rates due to lessened contamination (high)
    • Possible reduced cost of recyclate and increased recycled content (medium)
  • Known uncertainties:
    • Possible balancing effects:
      • Domestic reprocessing does not keep up with increased feedstock
      • Impact on economics of household recycling as most valuable materials removed from waste stream
  • Key data sources and information:
    • Valpak preparatory study
    • AluPro - Aluminium
    • British Glass - Glass
    • Data apportion to England using population
  • Data gaps:
    • Baseline data outside of 2022
    • Projections of target materials beyond naive forecast
    • Import/domestic production split
    • What is occurring to the percentage of target materials not collected
    • How to measure the quality of recyclate

Counterfactual construction

Ex ante policy assessments requires baseline projections against which to compare scenarios with policies introduced. While a robust and credible baseline should be established, this is not necessarily a forecast of future developments, but rather a neutral viewpoint to provide a point of comparison against (Cambridge Econometrics, ). A counterfactual is constructed for key variables in the modelling. That for the quantity ‘placed on market’ (POM) is calculated through the following steps:

  1. A correlation analysis between key determinants highlighted in the literature such as final demand volume (population and/or GDP/GDHI), and variables to be modelled is undertaken to select drivers of potential greatest influence;
  2. A ratio is calculated between the exogenous factor showing a strongest correlation and outturn values for the variable of interest (technology coefficient);
  3. This ratio is forecasted into the future using a range of parametric time-series forecasting techniques which allow confidence intervals to be constructed;
  4. Published projections for the exogenous variables from e.g. the ONS for population or OECD for GDP are multiplied by the forecasted ratio to calculate possible future POM values.

Other variables such as the share across end of life treatment routes for waste are modelled either through the use of constant transfer coefficients through time, or accounting for trends in these e.g. anticipated recycling rates, where these are shown in the ex post data assessment or otherwise anticipated due to policies on the horizon.

Model assumption development and evidence review protocol

We undertake a systematic review2 of the evidence regarding the likely type and scale of effects of shortlisted policies. Sources are reviewed and synthesised in line with guidance under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework. An attempt is made to be as systematic as possible in reviewing the literature to be able to make a legitimate statement of what the evidence on policy instruments in scope says to date overall and to do so in a clear and consistent way that is accessible to a range of readers (DFID, 2010).

Evidence gathering

Eligibility - Inclusion criteria

Information sources: To establish a shortlist of sources, repositories and websites were selected which could help capture evidence from academic studies, government reports, industry reports and wider grey literature.

Search strategy: The following inclusion criteria (translated into search terms) were used to identify sources in the literature providing evidence on the instruments of interest:

  • Policy instrument type - we look for evidence on the following policy types in particular:

  • Policy focus - Looking specifically at instruments levied on material flows and stocks

  • Region of introduction - We prioritise evidence collected on instruments which have been applied in the UK on the expectation these will provide best insight to how instruments may integrate with the institutional and contextual specificities of countries across the UK. We do so through cross-referencing studies against a list of relevant UK policies identified through a search of legislation portal and OECD database.3 As in some cases there may be limited or no precidence of introducing particular tools in the UK, we next prioritise evidence from countries with similar PPP.

  • Outcomes and impacts - We collate evidence on policies in scope structured against a set of criteria or ‘Critical Success Factors’ (CSFs)4 detailed further under the section on synthesis of evidence, but with effectiveness of particular interest and therefore an explicit search term.

These inclusion criteria were translated into search terms and entered into the repositories. This was done in between October 2024 and February 2025 by two reviewers.

Table . Search terms by repository and returned hits

Search term/string Repository Results

Eligibility - Exclusion criteria

This stage identified _ sources. We excluded those viewed as unsuitable or out of scope based on the following exclusion criteria:

  1. Bias

Shortlisted sources

The process shortlisted _ sources, consisting of:

  • _ Academic sources

  • _ Government reports (including policy documents)

    • X Government reports were identified through a search of the Defra Science Search repository

    • Impact assessments

    • Post implementation reviews

  • _ Industry reports and wider grey literature

X were specific to the UK, X to Europe and X to the wider world.

Synthesis of evidence

The shortlisted evidence sources are diverse, including theoretical modelling studies, surveys, post-implementation reviews, consultations, single policy assessments in scientific papers and meta-analyses. The shortlisted sources also utilise a range of quantitative and qualitative ex ante and ex post methods including cost-benefit analysis, cost-effectiveness analysis, multi-criteria analysis, theoretical appraisal, regression discontinuity.

To help inform how transferrable insights are to the UK case, shortlisted sources were inventoried/coded in relation to information on: instrument type and sub-type, CE measure of relevance, country, research methodology, target activity and/or product group(s) and CE measure of relevance drawing on community-specific classifications. Quality of sources graded based on… Evidence on the following characteristics were primarily captured qualitatively:

  • Efficiency - We also consider evidence on how efficiently outcomes are delivered (value for money) and the role of instruments in moving the UK towards a more optimal distribution of goods, services and pollution (allocative efficiency). We include findings from both cost-effectiveness analysis5 and cost-benefit analysis.6

  • Financial cost to the public sector - A key part of moving from theoretical to actual benefits is political and administrative feasibility (the ability to put a policy into effect in a given context), with government affordability an important parts of this (OECD, 1999). In addition to considering evidence on abatement and compliance costs from the perspective of the regulatee, we look for evidence on direct and indirect administrative costs to government (including at policy design and enforcement stages) of instruments, as well as evidence on revenues generated which can offset these (HMT, 2022).

  • Long-run effects - We look for evidence on the long-run effects of instruments and their ability to meet regulatory aims persistently into the future while providing incentives for continued improvements beyond the minimum. This includes their continued effectiveness under a variety of circumstances and resilience to external changes such as inflation, ability to be updated in response to new information and capacity to harness technological change through providing incentives for innovation over time so as to lower the costs of achieving goals over time (Fiorino, 2004).

  • Distributional and equity effects - Considering the net-effects of instruments on different people and groups, and how these may exacerbate pre-existing inequities (Bryant and Bailey, 1997). Evidence of regressive effects across income strata, geography concerning any of the groups identified by the Equality Act 2010 as well as disproportionate burdens on small and micro businesses are sought to be captured (RPC, 2019).

  • Spillovers - We look for evidence on spillovers, both positive and negative and including: 1) Soft effects such as impacts on attitudes, awareness and learning; 2) Wider economic impacts including on innovation and trade; and 3) Perverse incentives and any contribution to potentially unintended consequences such as negatively impacting competition or giving rise to cross-media impacts.7

  • Strategic fit - How policy intervention supports ‘national, regional, local or organisational policies, initiatives and targets’, align with other projects and programmes and fits with wider business strategy of UK public bodies (HM Treasury, 2018).

While for the criteria of:

  • Effectiveness - There is often a high level of uncertainty regarding the impacts of policy change, with this based on a range of factors. To assess effectiveness, we capture evidence on the effect of instruments in relation to their objectives, which given the scope of our study includes measures such as reducing (primary) resource use, waste generation, reducing leakage and improving waste treatment and including the immediacy (indicative time required to implement) with which these effects arise and certainty of meeting aims (predictability).

Evidence was synthesised quantitatively. Evidence from these sources is synthesised into a set of ‘modelling blueprints’ made up of core required assumptions to input to policy sub-models in a transparent and consistent way (Donati et al. 2020; Hellweg et al. 2023). Synthesised values are documented in the assumption log of this site.

As part of the synthesis, a contingency analysis was undertaken to understand possible dependencies between policies and ordering effects. This also informed the interactions matrix developed.

  • Availability & quality of information - Ideally our evaluation of instruments should include a comprehensive assessment against all critical success factors, but often due to data and resource constraints, this is not possible.
  • Attribution - It is unlikely to be the case that all effects on outcomes such as pollution or resource use levels following the introduction of an instrument can be attributed to it. Attribution can be made difficult by the absence of a robust counterfactual i.e. what could have happened were the instrument not introduced. Comparisons of the performance of actors adopting an instrument versus those not can also be impacted by selection bias and other issues of representation. Policy instruments are also rarely used in isolation, such that it is often very difficult to separate individual contributions of policy measures while the joint effects on outcomes need to be recognised.
  • Transboundary and cross-media effects - Instruments don’t operate in a national vacuum, and spillover effects which might contribute to environmental degradation elsewhere and including in another environmental media, need to be considered.

Extending the analysis to other appraisal frameworks

Outputs of the modelling such as physical impacts of policies on material flows and emissions can be monetised to input to cost-benefit analysis - which has become the default standard for policy appraisal when formulating most new government regulation in the UK (Sunstein 2000). The general steps of a cost-benefit analysis (CBA) are (Boardman et al., 2006; Atkinson et al. 2018):

  1. Select the portfolio of options (projects or policies);
  2. Establish a baseline for comparison to indicate the counterfactual;
  3. Determine who has ‘standing’ and we are interested in counting the benefits and costs to;
  4. Identify, list and quantify potential physical impacts for the life of the project or policy (this is where outputs of the DO input directly);
  5. Monetize impacts for each time period;
    • Benefits - Outputs of the project which increase human welfare, the value of which are assessed based on how much someone is willing to pay for that benefit, even for the likes of avoided costs.
    • Costs - Outputs of a project that decrease human welfare and which include the costs of inputs.
  6. Aggregation then involves all benefits and costs being summed individually, while discounting for future benefits and costs depending on the social discount rate;8
  7. Sensitivity analysis can then be used as an indication of uncertainty by testing the sensitivity of the net benefit to changes in its determinants;9
  8. Results are presented either as a net benefit (B-C) or a benefit-cost ratio (B/C);
  9. Options are selected based on the following decision-rules:
    • Where options are mutually exclusive, the policy should be chosen that has the highest Net-Benefit.
    • If options can be chosen independently with no resource constraints, all options with a net benefit greater than 0 can be chosen.
    • If options can be chosen independently but resources are limited, then options can be ranked by the B/C ratio and those yielding the greatest net-benefit, selected.

When standard assumptions of welfare economics and CBA are met, optimisation models can play an important role in analysing the potential impact of policies or projects on the allocation of existing economic resources. However, where assumptions for welfare economics CBA are not met, Sharpe et al.(2021) recommend the use of ‘risk opportunity analysis’ (ROA) which is a more general form of cost-benefit analysis. ROA steps involve:

  1. System boundaries are delimited, and all relevant interactions and positive and negative feedbacks are identified (suitable models, if required, are chosen or designed);
  2. The potential effects (intended and unintended) of policy options in the economy are assessed, and uncertainty ranges estimated;
    1. Mapping the relationships between components of the economic system of concern, in terms of reinforcing and balancing feedbacks
    2. Identifying the likely effect of policy interventions on system behaviour, based on changes to the structure of relationships between components (including relationships created by other policies that already exist or are under consideration). This may be extended to include the creation of a range of scenarios and storylines of cumulative causation that result from policy action, where longer-term effects are likely to be important to policy objectives;
    3. Comparing likely effects in terms of:
      1. Direction of change (of any variables of policy interest)
      2. Magnitude of change (which may or may not be quantifiable)
      3. Pace of change
      4. Possible accumulation of risk and opportunity (option generation)
      5. Confidence, or range of uncertainty, in each of i to iv above.
  3. The risks and opportunities of options (including most likely, best-case and worst-case outcomes) are compared along multiple relevant metrics and dimensions (where probabilities may be quantifiable or unquantifiable). This includes consideration of systemic risk (breakdown of an existing system), and systemic opportunity (where policy generates a whole new system, or set of opportunities);
  4. The preferred option is determined by the decision-maker based on a qualitative judgment of the scale of the opportunities and risks, compared to the cost of the intervention. This will necessarily be a subjective judgment (since it incorporates a weighing of outcomes in different dimensions), informed by an objective assessment of likelihood and magnitude of possible outcomes in each of the relevant dimensions; and
  5. A clear statement of the reasoning behind the decision is recorded including the decisionmaking body’s assessment of the risks and opportunities in their various dimensions. (This can be helpful for transparency and for learning from experience).

Key differences in ROA to CBA are that multiple metrics are used, the focus is on expected processes that drive change rather than outcomes, unquantified risks and opportunities are central to consideration so expected values are not used and that policies are recommended that aim for maximum and self-reinforcing leverage in the desired direction of change.

Back to top

Footnotes

  1. Though even for mandatory approaches, levels of enforcement need to be considered↩︎

  2. This is implemented through “a clear protocol for systematically searching defined databases over a defined time period, with transparent criteria for the inclusion or exclusion of studies, as well as the analysis and reporting of study findings” (Waddington et al., 2012: 360).↩︎

  3. While the term ‘circular economy’ (CE) has only appeared more recently in policy [recommendation] documents released by UK [non-]public bodies, policies whose objectives are congruent with the CE concept by delivering against one of its ‘value drivers’ or measures, have been introduced in the UK since at least the 1970s. We therefore capture regulations that may not explicitly include reference to the CE, but can otherwise be recognised as relevant.↩︎

  4. In the HM Treasury Green Book as ‘attributes essential to the successful delivery of projects and programmes’ (HM Treasury, 2022)↩︎

  5. Cost-effectiveness analysis involves a monetary assessment of costs alone while not for benefits.↩︎

  6. Cost-benefit analysis involves measuring net benefits or benefit-cost ratios through both costs and benefits being monetised.↩︎

  7. Such as the IMO’s 2018 Ship Emissions Regulation leading to the installation of equipment helping meet atmospheric emissions regulations but by routing discharge into the ocean.↩︎

  8. We discount future values into those that are present-day through the use of a discount rate to account for society’s preferences for benefits now rather than later and its lesser concern for costs in the future versus those now on one hand, and the opportunity cost of capital given economic productivity on the other (Speer et al. 2015). The choice of discount rate embodies important normative assumptions regarding intergenerational equity and short-termism. Boardman et al. (2006) recommend a real social discount rate of 3.5% be used for studies 1-50 year into the future, with this increasing past a 50 year period. ↩︎

  9. Sensitivity analysis requires that the CBA is computed using different values of the parameters about which there is uncertainty. Such procedures require some assumption about likely minima and maxima, but do not necessarily make assumptions about the distribution of values between these limits. For example, if a discount rate of 4% is chosen as the central case, then 2 and 6% could also be chosen for a sensitivity analysis. One possible outcome is that the sign of the net benefits will be unaffected by these alternatives, in which case the analysis is said to be “robust” with respect to these assumptions. In other cases, changing assumptions may alter the CBA result to such an extent that the option is no longer net-welfare-enhancing.↩︎