-
Notifications
You must be signed in to change notification settings - Fork 0
/
design.qmd
130 lines (75 loc) · 8.04 KB
/
design.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
::: {.callout-important}
This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Design
## Introduction and overview
The design stage is where the Analyst translates the scope for the analysis agreed with the Commissioner into an actionable analytical plan. This chapter sets out recommended practices around designing the analysis and associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design, and design of multi-use and AI models.
### The analytical plan
The development of the analytical plan should consider:
* Methodology for producing results, including the treatment of uncertainty;
* Project management approach (for example Agile, Waterfall or a combination of approaches);
* Sourcing of inputs and assumptions;
* Data and file management;
* Change management and version control;
* Programming language and/or software;
* Code management, documentation and testing;
* Communication between stakeholders;
* Verification and validation procedures during the project lifetime;
* Documentation to be delivered;
* Process for updating the analytical plan;
* Process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date;
* Ethics;
* Reporting;
* Downstream application.
The use of [Reproducible Analytical Pipelines (RAP)](#rap) is encouraged as a means of effective project design.
Iteration between the Commissioner and the Analyst is normal and expected whilst the analytical design develops.
::: {.callout-tip}
# Reproducible analytical pipelines<a name="rap"></a>
The recommended approach for developing analysis in code is to use a [Reproducible Analytical Pipeline (RAP)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/). Reproducible Analytical Pipelines shall:
* Follow the practices set out in the [Analysis Function Quality Assurance of Code for Analysis and Research manual](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html).
* Meet the requirements of the [Reproducible Analytical Pipelines minimum viable product](https://github.com/best-practice-and-impact/rap_mvp_maturity_guidance/blob/master/Reproducible-Analytical-Pipelines-MVP.md).
:::
## Roles and responsibilities in the design stage
### The Commissioner's responsibilities during the design stage
The Commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the Commissioner may review the analytical plan.
The Commissioner's domain expertise can be a useful resource for the analyst in the design stage. The Commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs, all of which can inform the design.
### The Analyst's responsibilities during the design stage
The Analyst should:
* develop the method and plan to address the Commissioner’s needs;
* establish assurance requirements;
* develop the plan for proportionate verification and validation - see [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf);
* plan in sufficient time for the assurance activity;
* document the analytical plan in a proportionate manner; and,
* follow any organisation governance procedures for project design.
### The Assurer's responsibilities during the design stage
The Assurer should review the analytical plan to ensure that they are able to conduct the required assurance activities. They may provide feedback on the analytical plan. The Assurer should plan sufficient time for the assurance activity.
### The Approver's responsibilities during the design stage
In smaller projects, the Approver may not be heavily involved in the design stage. However, for business critical analysis, the Approver may want to confirm that organisational governance procedures for design have been followed.
## Assurance activities in the design stage
On completion of the design stage, the Assurer should be aware of the quality assurance tasks that will be required of them during the project lifetime and have assured the necessary elements of the analytical plan.
The assurance of the design stage should consider whether the analytical plan is likely to:
* Address commissioner's requirements - validation;
* Deliver as intended - verification;
* Be robust i.e. well-structured, data driven, with a sound overall design.
The assurance of the design stage may be carried out by the Assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance, and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases.
## Documentation in the design stage
The design process should be documented in a proportionate manner. A design document that records the [analytical plan](#the-analytical-plan) should be produced by the Analyst and signed-off by the Commissioner. The design document may be reviewed by the Assurer.
For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage.
It is best practice to use formal version control to track changes in the design document.
## Treatment of uncertainty in the design stage
During the design stage, Analysts should examine the planned analysis systematically for possible sources and types of uncertainty, to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error. This is discussed in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html)
## Black box models and the design stage
Using [black box models](definitions_and_key_concepts.qmd/#black-box-models) places greater weight on the design of the analysis and the assurance and validation of outputs by domain experts.
This [guidance on AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design of AI models, including risk assessment, impact assessment, bias audits and compliance audits.
In the Design of AI/ML models, the Analyst should:
* define the situation they wish to model;
* the prediction they wish to make;
* the data that could be used to make the prediction;
* carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach;
* consider how to separate the data for the design and testing of models - it's usual to design a model with a fraction of the data and then test it with the data that was not used in the design;
* consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions; and,
* consider whether to refer the model to their ethics committee, or a similar group - see the [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020).
## Multi-use models and the design stage
Designing multi-use models should take into account the needs of all users of the analysis. An Analysis Steering Group may be an effective means for communication about the design with a range of user groups.
The design of multi-use models may entail a modular structure with different Analysts and Assurers responsible for different elements. The design of assurance activities should capture both the assurance of individual modules and their integration.