From d9b0cdd25ba030345e652768526d4e7ada6b5036 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Mon, 16 Dec 2024 10:34:43 +0000 Subject: [PATCH 01/26] Added edits to definitions_and_key_concepts.qmd --- definitions_and_key_concepts.qmd | 93 ++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 29 deletions(-) diff --git a/definitions_and_key_concepts.qmd b/definitions_and_key_concepts.qmd index a4d1df1..079b276 100644 --- a/definitions_and_key_concepts.qmd +++ b/definitions_and_key_concepts.qmd @@ -10,7 +10,7 @@ This chapter sets out definitions and concepts that are used throughout the rest ## Analysis {.unnumbered} -Analysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. +Analysis is the collection, manipulation and interpretation of information and data for use in decision-making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. Examples of types of analysis that are frequently encountered in government are: @@ -26,13 +26,13 @@ Examples of types of analysis that are frequently encountered in government are: ## Assurance {.unnumbered} -Analytical assurance is the process and set of practices to ensure that the analysis is fit for purpose. +Analytical assurance is the process and set of practices that ensure analysis is fit for purpose. ## Assurance activities {.unnumbered} Assurance activities are any actions carried out in order to validate and verify analysis. -For example: +This may include: * analyst testing * peer review @@ -40,46 +40,69 @@ For example: ## Artificial Intelligence {.unnumbered} -Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models) +Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making and language understanding. AI models are a subset of [black box models](#black_box_models) ## Black box models {.unnumbered} -Black box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future. +The internal workings of black box models are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [AI](#Artificial Intelligence) models are the most common type of black box models used today. Other forms of black box models may be developed in the future. ## Business critical analysis {.unnumbered} -Business critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal implications. +Business critical analysis has significant influence over financial and funding decisions, is necessary to the achievement of a departmental business plan, or is analysis where an error could have a significant reputational, economic or legal implications. -The first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM). +The first edition of the AQuA book described business critical models. This has been updated to the more general term 'business critical analysis' because analysis can be business critical without including a model. Some departments may continue to use the term business critical models (BCM). ## Documentation {.unnumbered} ### Specification documentation {.unnumbered} -Specifications capture initial engagements with the commissioner. They describe the question, the context, and any boundaries of the analysis. This provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner. +Specifications capture the initial engagements with the commissioner. They describe the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and capturing what level of assurance is required by the commissioner. ### Design documentation {.unnumbered} -Design documents describe the analytical plan, including the methodology, inputs, and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the Analytical Assurer to verify whether the analysis meets the specified requirements. For more information on the design documentation, see the [Design](design.qmd) chapter. +Design documents describe the analytical plan, including the methodology, inputs and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements. + +You can read more about design documentation in the [Design](design.qmd) chapter. ### Assumptions log {.unnumbered} -A register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its effect and reliability and set out when it was made, why it was made, who made it and who signed it off. +A register of assumptions, whether provided by the commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should: + +* describe each assumption +* quantify its effect and reliability +* set out when it was made +* explain why it was made +* explain who made the assumption and who signed it off ### Decisions log {.unnumbered} -A register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off. +A register of decisions, whether provided by the commissioner or derived by the analysis. Decisions logs should: + +* describe each decision +* set out when it was made +* explain why it was made +* explain who made the decision and who signed it off ### Data log {.unnumbered} -A register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. +A register of data provided by the commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. + + +### User/technical documentation {.unnumbered} +All analysis shall have user-documentation, even if the only user is the analyst leading the analysis. This documentation should include: * -### User / technical documentation {.unnumbered} +* a summary of the analysis including the context to the question being asked +* what analytical methods were considered +* what analysis was planned and why +* what challenges were encountered and how they were overcome +* what verification and validation steps were performed -All analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. For modelling, the Analyst may include a model map that describes data flows and transformations. +Where relevant the analyst may include a model map that describes data flows and transformations. + +For analysis that is likely to be revisited or updated in the future, more comprehensive documentation should be provided to assist a future analyst. It may also be helpful to include guidance on what should be considered or updated. ### Assurance statement {.unnumbered} @@ -87,7 +110,7 @@ A brief description of the analytical assurance that have been performed to assu ::: {.callout-tip} # Example of publishing quality assurance tools -The Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. +The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. ::: @@ -95,45 +118,57 @@ The Department for Energy Security and Net Zero and Department for Business and Some models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. -Often, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area. +Often, a Steering Group is created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group would ideally have decision-making responsibilites in their area of work. ## Quality analysis {.unnumbered} -Quality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its effect, adequately communicated, documented and accepted by its commissioners. +Quality analysis is fit for the purpose it was commissioned to meet. It should be: + +* accurate +* appropriately assured +* evidenced +* bproportionate to its effect +* adequately communicated +* documented +* accepted by its commissioners ## Roles and responsibilities {.unnumbered} The AQuA book defines the following roles: -* **Commissioner** -* **Analyst** -* **Assurer** -* **Approver** +* commissioner +* analyst +* assurer +* approver -See [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details. +You can read more in the [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) section. ## Third party {.unnumbered} -Any individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company. +Any individual or group of individuals that is not a member of the same group as the those commissioning analysis. For example, they may be working for a different government department, a different function or an outside company. ## Uncertainty {.unnumbered} -Uncertainties are things that are not known, or are in a state of doubt, or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme, piece of analysis meeting its objectives.[^1] +Uncertainties are things that are not known, are in a state of doubt or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme or piece of analysis meeting its objectives.[^1] [^1]: https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf -There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required. +There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required. -The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty. +The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross-government group to help assessing and communicating uncertainty. ## Validation {.unnumbered} -Ensuring the analysis meets the needs of its intended users and the intended use environment. See @glover2014 for more information. +Validation ensures the analysis meets the needs of its intended users and the intended use environment. + +You can read more in [Verification and validation for the AQuA book] (https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. ## Verification {.unnumbered} -Ensuring the analysis meets it specified design requirements. See @glover2014 for more information. +Verification ensures the analysis meets it specified design requirements. + +You can read more in [Verification and validation for the AQuA book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. ## Version control {.unnumbered} -It is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. +It is important to ensure that the latest version of the analysis is being used and any changes made can be easily seen and quality assured by the analytical assurer. There are tools and templates that can be used to record any updates and checks made during a project. They can help to provide a log of the changes that have been made including why and when they were made, and who made them. From 2838110fc56ab0847b6a75fd2af0216f8774fcbd Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Mon, 16 Dec 2024 15:08:39 +0000 Subject: [PATCH 02/26] Added edits to proportionality.qmd --- proportionality.qmd | 60 +++++++++++++++++++++------------------------ 1 file changed, 28 insertions(+), 32 deletions(-) diff --git a/proportionality.qmd b/proportionality.qmd index 33b3b35..e2e08c4 100644 --- a/proportionality.qmd +++ b/proportionality.qmd @@ -13,50 +13,46 @@ The draft currently has no official status. It is a work in progress and is subj # Proportionality -All analysis shall be assured, and the assurance should be proportionate to the potential effect it will have, size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks. +All analysis shall be assured. The assurance should be proportionate to the potential effect it will have and the size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks. The assurer and the analyst shall be independent. The degree of separation depends on many factors including the importance of the output, and the size and complexity of the analysis. This does not mean that the analyst should not undertake assurance, rather that there shall also be some formal independent assurance. -## Introduction +## Factors for determining appropriate assurance -Think about and deliver appropriate (proportionate) levels of assurance for your analysis. There is a need to be confident in analysis delivered, but there is no point spending months assuring simple analysis that will have a minor influence on a decision. +While there is a need to be confident in the analysis it is not necessary to spend months assuring simple analysis that will have a minor influence on a decision. The level of analysis should be appropriate (proportionate) to the analysis. Table 3-1 provides a list of factors that should be considered when determining what level of assurance is appropriate. -Further detail and considerations may be found on the Data Quality Hub's [Quality Questions and Red Flags](https://dataqualityhub.github.io/resources-for-quality-analysis-external/) page. - - | Factor | Comments | |:--------|:----------| -| Business criticality | Different issues will affect business criticality such as financial, legal, operational, political and reputational effects. | -| Relevance of the analysis to the decision making process | When analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the evidence base. | +| Business criticality | Different issues will affect how business critical the analysis is. For example, its financial, legal, operational, political and reputational effects. | +| Relevance of the analysis to the decision-making process | When analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the whole evidence base. | | Type and complexity of analysis | Highly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts. | | Novelty of approach | A previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested. | -| Reusing or repurposing existing work | Reusing work that was carried out previously may require validation and verification to confirm that original approach - method, assumptions data etc. are still appropriate for the new requirement.| +| Reusing or repurposing existing work | Reusing work that was carried out previously may require validation and verification to confirm that original approach. For example, the original method and assumptions data are still appropriate for the new requirement. | | Level of precision required in outputs | Lower precision analysis often uses simplified assumption, models and data. The assurance approach is the same but will take less time than more precise analysis.| -| Amount of resource available for the analysis and assurance | The value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Analysis that is used for many purposes (e.g. population projections) may require greater levels of QA than might be suggested by any individual decision they support. | +| Amount of resource available for the analysis and assurance | The value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Analysis that is used for many purposes (for example, population projections) may require greater levels of quality assurance than might be suggested by any of the individual decisions they support. | | Longevity of the analysis | Ongoing analysis will require robust change control and regular review. | -| Affects on the public | Analysis which will have a significant affect on the public may require more assurance. | -| Repeat runs for the same analysis | Concentrate on version control and assurance of data and parameters for each run. | - -: Table 3-1 - Factors for determining appropriate assurance {.striped} +| Affects on the public | Analysis which will have a significant effect on the public may require more assurance. | +| Repeat runs for the same analysis | Assurance should concentrate on version control and the assurance of data and parameters for each run. | +: Table 3-1 Factors for determining appropriate assurance {.striped} -Figure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The important point is, the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis. +You can read more on the Data Quality Hub's [Quality Questions and Red Flags](https://dataqualityhub.github.io/resources-for-quality-analysis-external/). -![Figure 3-1 - Types of quality assurance - darker shades indicate the need for need for extra assurance activities and greater separation between the analyst and the assurer. The contours indicate the groups of activities that may be carried out for a particular level of business risk/complexity.](Figure 3-1 Types of Assurance Alternative 3.jpg){fig-alt="Figure 3-1 is diagram showing the relationship between risk, complexity and the requirement for assurance activity. There are two axes on the diagram. The X axis goes from simple analysis on the left to highly complex analysis on the right. The y axis goes from low business risk at the bottom to high business risk at the top. As risk and complexity increase there is a need for extra assurance activities as well as a higher degree of separation between the analyst and the assurer. For complex, high risk analysis this might include external peer review or audit. On the diagram, the increasing level of risk as we move from bottom left to top right is represented by darker shades."} +Figure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The need for more assurance interventions increases with the complexity of the analysis and the business risk associated with it. -The interventions in Figure 3-1 must not be viewed in isolation. The interventions should build on each other, for example some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, for example version control and analyst led testing. +![Figure 3-1 The darker shades in the diagram indicate the need for extra assurance activities and greater separation between the analyst and the assurer. The contours indicate the groups of activities that may be carried out for a particular level of business risk or complexity.](Figure 3-1 Types of Assurance Alternative 3.jpg){fig-alt="Figure 3-1 is diagram showing the relationship between risk, complexity and the requirement for assurance activity. There are two axes on the diagram. The X axis goes from simple analysis on the left to highly complex analysis on the right. The y axis goes from low business risk at the bottom to high business risk at the top. As risk and complexity increase there is a need for extra assurance activities as well as a higher degree of separation between the analyst and the assurer. For complex, high risk analysis this might include external peer review or audit. On the diagram, the increasing level of risk as we move from bottom left to top right is represented by darker shades."} -The total elimination of risk will never be achievable, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate. +The interventions in figure 3-1 must not be viewed in isolation. The interventions should build on each other, for example some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, such as version control and analyst-led testing. -Many of these interventions are mentioned elsewhere in the AQUA Book, and are not repeated here. +The total elimination of risk will never be achievable so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques but there are many different techniques explained in the AQuA book that need to be considered and implemented as appropriate. ## Structured assessment of business risk and complexity -To guide what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential affect of analytical errors, and the likelihood of errors occurring. In situations where the potential effect is high, it is more important that the likelihood of errors is reduced. +To determine what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential effect of analytical errors and the likelihood of errors occurring. In situations where the potential effect is high, it is more important that the likelihood of errors is reduced. -This can be visualised by considering the situation as a risk matrix, illustrated in Table 3-2. The effect analysis will have is usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left. +This can be visualised by considering the situation as a risk matrix (Table 3-2). The effect the analysis will have is usually be beyond the control of the analyst to change, so there will be few options to lessen the effect of a risk. However, there will usually be treatments (or mitigations) involving additional assurance measures that will allow the assessed business risk to become less likely to occur. @@ -124,30 +120,30 @@ Table 3-2 - Example of a risk matrix
-Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (see Figure 3-1). +Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (Figure 3-1). -For a situation where simple analysis is being employed, a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures, options like external peer review may be necessary. +For a situation where simple analysis is being employed a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures options such as external peer review may be necessary. -In cases where there is a need for analysis, but there are also significant time and/or resource constraints, it may not be possible to do as much assurance as usual. In these situations, concentrate on areas of greatest risk. These risks and limitations must also be communicated, along with appropriate caveats. +In cases where there is a need for analysis but there are also significant time or resource constraints, it may not be possible to do as much assurance as usual. In these situations the areas of greatest risk are the priority. These risks and limitations must also be communicated, along with appropriate caveats. | Assessed risk | Mitigations to consider | |------------------|------------------------------------------------------| -| High | The risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring. If treatment isn't an option, consideration must be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the SRO needs to fully understand the risk. | -| Medium | The risk should not be tolerated without SRO agreement. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations. | -| Low | The risk can be tolerated. Existing or planned mitigations should be continued, and new treatments may be considered. | -| Very Low | The risk can be tolerated. Existing/planned mitigations measures should be continued. | +| High | High risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring. If treatment isn't an option, consideration must be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the senior responsible owner needs to fully understand the risk. | +| Medium | Medium risk should not be tolerated without the agreement of the senior responsible owner. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations. | +| Low | Low risk can be tolerated. Continue with existing or planned mitigations and new treatments may be considered. | +| Very Low | Very low risk can be tolerated. Continue with existing or planned mitigations measures. | -: Table 3-3 - Responses to risk assessment levels {.striped} +: Table 3-3 Responses to risk assessment levels {.striped} -For further guidance on risk management, refer to the [Orange book](https://www.gov.uk/government/publications/orange-book) which covers risk management principles and risk control frameworks. +You can read more on risk management in the [Orange book](https://www.gov.uk/government/publications/orange-book) which covers risk management principles and risk control frameworks. ## Externally commissioned work -Proportionate assurance of externally commissioned work is just as important as for internally produced analysis. For the commissioner, to use the work they should be fully informed of the business risk associated with it. This should be provided by an appropriate mix of documented risk assessments provided as part of the work, and by joint risk assessments planned throughout the life of the project. For commissioned work the options for mitigation will be similar to those for internal analysis. +Proportionate assurance of externally commissioned work is just as important as for internally produced analysis. The commissioner should be fully informed of the business risk associated with the work. This should be provided by an appropriate mix of documented risk assessments provided as part of the work and by joint risk assessments planned throughout the life of the project. For commissioned work the options for mitigation will be similar to those for internal analysis. The difference will be in ensuring the assessment of risks and the applied mitigations are fully understood by the commissioner. ## Black-box models and business risk -Increasingly analysis may be underpinned by artificial intelligence or other forms of black-box models. With such models, the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible. +Increasingly analysis may be underpinned by Artificial intelligence (AI) or other forms of black-box models. With these models the need to understand business risk remains and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts and developing understanding of what mitigations are possible. From 0748ea4636936d737fe09b2784a5399f606580d6 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Mon, 16 Dec 2024 15:33:59 +0000 Subject: [PATCH 03/26] Added edits to engagement_and_scoping.qmd --- engagement_and_scoping.qmd | 84 +++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 43 deletions(-) diff --git a/engagement_and_scoping.qmd b/engagement_and_scoping.qmd index c864304..d7270de 100644 --- a/engagement_and_scoping.qmd +++ b/engagement_and_scoping.qmd @@ -6,78 +6,76 @@ The draft currently has no official status. It is a work in progress and is subj # Engagement and scoping +During the first stage of the analytical lifecycle the initial engagement takes place and the commissioner's requirements are scoped out. This stage identifies what is relevant for the analysis. +During this engagement and scoping stage the commissioner and the analyst shape the analysis by developing a shared understanding of the problem and the context. This shared understanding will be used as the basis for designing analysis that suits the commissioner’s requirements. -## Introduction and overview +## Roles and responsibilities +### The commissioner's responsibilities -The first stage of the analytical lifecycle is initial engagement and scoping out what the Commissioner requires. This information constrains what is relevant for the analysis. The Analyst works with the Commissioner to develop sufficient understanding of the problem to design a requisite analysis. +In the engagement and scoping stage the commissioner should: -During engagement and scoping, the Commissioner and the Analyst shape the analysis by developing a shared understanding of the problem and the context. This shared understanding will be used as the basis for designing analysis that suits the Commissioner’s requirements. +* communicate to the analyst the important aspects of the problem, scope, and programme constraints +* be available to engage with the Analyst to appropriately shape the work +* ensure that they understand risks where time and resource pressures constrain the approach +* communicate to the Analyst any sources of uncertainty they have identified as part of their wider considerations +* sign-off on the specification document produced by the analyst +* indicate the consequences for decision-making of different degrees of uncertainty, if possible, as this may enable the analyst to conduct their analysis at a proportionate level -## Roles and responsibilities in the engagement and scoping stage +### The analyst's responsibilities +In the engagement and scoping stage the analyst should: +* engage with the Commissioner to identify the question, the context, and the boundaries of the analysis, as well as constraints (for example, deadlines and available resource) assumptions, risks, identified uncertainties and business-criticality. +* create a specification document which captures the Commissioner's requirements +The specification document should provide a definition of the scope and project constraints. It should state the acceptable level of risk and the required level of assurance. It may also state the degree of uncertainty allowed for decision-making and record identified sources of uncertainty. The analyst should share this specification with the commissioner for sign-off. -### The Commissioner's responsibilities during the engagement and scoping stage +### The assurer's responsibilities -The Commissioner should: +In the engagement and scoping stage the assurer may confirm that the engagement process has been sufficient to fully understand the problem. For more business critical projects, the assurer may wish to confirm that the specification document adequately captures the outcomes of the engagement process. -* communicate to the Analyst the important aspects of the problem, scope, and programme constraints; -* be available to engage with the Analyst to appropriately shape the work; -* ensure that they understand risks where time and resource pressures constrain the approach; -* communicate to the Analyst any sources of uncertainty they have identified as part of their wider considerations; -* sign-off on the specification document produced by the analyst; and, -* if possible, indicate the consequences for decision-making of different degrees of uncertainty, as this may enable the Analyst to conduct their analysis at a proportionate level. +### The approver's responsibilities +In the engagement and scoping stage the approver should note the new project and confirm that resources and plans are in place for the appropriate assurance to take place. For example, they should ensure that the analyst and assurer are aware of local assurance protocols. The approver might provide support in securing a sufficiently qualified and experienced assurer. -### The Analyst's responsibilities during the engagement and scoping stage +The approver should ensure that there is sufficient governance in place to support the analyst and their role in the wider project or programme. This is particularly important if the analysis supports business critical decisions. This may need to be revisited at the design stage if a novel or riskier approach is required (for example, if Artifical Intelligence (AI) models are used). -The Analyst: +## Assurance activities -* Should engage with the Commissioner to identify the question, the context, and the boundaries of the analysis, as well as constraints (e.g. deadlines, available resource), assumptions, risks, identified uncertainties and business-criticality. -* Should create a specification document which captures the Commissioner's requirements. It should provide a definition of the scope and project constraints. It should state the acceptable level of risk, the required level of assurance. It may also state the degree of uncertainty allowed for decision-making, and record identified sources of uncertainty. The Analyst should share this specification with the Commissioner for sign-off. +If the commissioner is unable to present a well-defined problem, the engagement stage may require the use of problem structuring methods to develop a shared understanding of the requirements. Techniques such as the Strategic Choice Approach, Rich Pictures and Systems Thinking can help the analyst and commissioner to reach a joint understanding of the problem and define the scope of the work. -### The Assurer's responsibilities during the engagement and scoping stage +You can read more about these techniques in the [Systems Thinking Toolkit](https://www.gov.uk/government/publications/systems-thinking-for-civil-servants/toolkit) -The Assurer may confirm that the engagement process has been sufficient to fully understand the problem. For more business critical projects, they may wish to confirm that the specification document adequately captures the outcomes of the engagement process. +If the engagement and scoping techniques are complex or the project is deemed business critical, the assurer might also provide assurance of the engagement methodology (https://publications.tno.nl/publication/100301/Zs2SUz/wijnmalen-2012-natoclient.pdf). -### The Approver's responsibilities during the engagement and scoping stage +The engagement and scoping stage should lead to agreement between the analyst and commissioner about the outputs of the work, including acceptable levels of accuracy, precision and margins of error. This will inform the handling and the assurance of uncertainty in later stages. -The Approver should note the new project, confirm that resources and plans are in place for the appropriate assurance to take place. For example, they should ensure that the Analyst and Assurer are aware of local assurance protocols. The Approver might provide support in securing a sufficiently qualified and experienced Assurer. +The commissioner should communicate to the analyst any relevant information about data sources and data quality. This will be used to guide the design of data processing. -The Approver should ensure that there is sufficient governance in place to support the analyst and their role in the wider project or programme. This is particularly important if the analysis supports business critical decisions. This may need to be revisited at the design stage if a novel or riskier approach is required (for example if AI models are used). +The analyst and commissioner should also clarify risks and potential effects on the outcomes to inform the decisions around [proportionate assurance](proportionality.qmd). Constraints around resources and timelines should also be clarified and agreed. -## Assurance activities in the engagement and scoping stage +## Documentation -The engagement and scoping stage provides the Analyst with an understanding of the Commissioner's requirements. In some cases, the Commissioner may present a well-defined problem, while in other instances, the engagement stage may require problem structuring methods. Techniques such as the Strategic Choice Approach, Rich Pictures and Systems Thinking can help the Analyst and Commissioner to reach a joint understanding of the problem (see the [Systems Thinking Toolkit](https://www.gov.uk/government/publications/systems-thinking-for-civil-servants/toolkit) for further information). The engagement will lead to the Analyst and Commissioner being able to define the scope of the project in terms of the context and bounds of the analysis. In cases where the engagement and scoping techniques are complex and/or the project is deemed business critical, the Assurer might provide assurance of the engagement methodology (https://publications.tno.nl/publication/100301/Zs2SUz/wijnmalen-2012-natoclient.pdf) +The output of the engagement and scoping stage should be a [specification document](definitions_and_key_concepts.html#specification-documentation) that captures the joint understanding of the task between the commissioner and analyst. This document provides a reference for later [validation](definitions_and_key_concepts.html#validation) assurance activities (for example, by confirming that the analysis meets the specification). This document also provides the approver with evidence that the analysis meets the specification during the [delivery stage](delivery_and_communication.html). The document should be signed off by the commissioner and might also be reviewed by the assurer. -In addition to understanding the problem, the engagement and scoping stage should lead to agreement between the Analyst and Commissioner about the outputs to be delivered, including acceptable levels of accuracy, precision and margins of error. This will inform the handling of uncertainty and the assurance thereof in later stages. +## Treatment of uncertainty -The Commissioner should communicate to the Analyst what is known about data sources and data quality. This will be used to guide the design of data processing. +The engagement and scoping stage will inform the treatment of uncertainty by: -The Analyst and Commissioner should also clarify risks and potential effects on the outcomes to inform the decisions around [proportionate assurance](proportionality.qmd). Constraints around resource and timelines should also be clarified and agreed. +* providing a clear definition of the analytical question +* identifying sources of high or intractable uncertainty +* establishing an understanding of how the analysis will inform decisions -## Documentation in the engagement and scoping stage +You can read more about uncertainty in engagement and scoping in the [Uncertainty Toolkit](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_2.html#Jointly_agreeing_how_uncertainty_should_be_used) -The output of this stage should be a [specification document](definitions_and_key_concepts.html#specification-documentation) that captures the joint understanding of the task between the Commissioner and Analyst. This document provides a reference for later [validation](definitions_and_key_concepts.html#validation) assurance activities (i.e. that the analysis meets the specification). This document also provides evidence for the Approver during the [delivery stage](delivery_and_communication.html) that the analysis meets the specification. The document should be signed off by the Commissioner, and might be reviewed by the Assurer. +## Black box models -## Treatment of uncertainty in the engagement and scoping stage +Where the commissioner has engaged with the analyst to deliver [black box models](definitions_and_key_concepts.html#black-box-models) models such as AI or machine learning, the engagement and scoping stage should include discussions around ethics and risks in order to assess whether such models would be appropriate for addressing the given problem. For example, discussions might include considerations of regulations such as UK GDPR, organisational skills, internal governance and risk management. -The following aspects of the engagement and scoping stage will inform the treatment of uncertainty: +You can read more in the [Introdction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance). -* A clear definition of the analytical question -* Identification of sources of high and/or intractable uncertainty -* Establishing an understanding of how the analysis will inform decisions +## Multi-use models -Further details on Uncertainty in engagement and scoping can be found in the [Uncertainty Toolkit](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_2.html#Jointly_agreeing_how_uncertainty_should_be_used) - -## Black box models and the engagement and scoping stage - -Where the Commissioner has engaged with the Analyst to deliver [black box models](definitions_and_key_concepts.html#black-box-models) models such as AI/ML, the engagement and scoping stage should include discussions around ethics and risks in order to assess whether such models would be appropriate for addressing the given problem. For example, discussions might include considerations of regulations such as UK GDPR, organisational skills, and internal governance and risk management. For further details see https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance. - -## Multi-use models and the engagement and scoping stage - - -In the case of multi-use models, the Analyst may be required to engage with a group of end-users to develop an understanding of their respective requirements. As requirements might differ or contradict, techniques such as Strategic Options Development and Analysis ([SODA](https://www.ru.nl/publish/pages/938444/soda_-_the_principles.pdf)) and [Soft Systems Methodology] (https://en.wikipedia.org/wiki/Soft_systems_methodology) may be used to develop a shared understanding across multiple groups. +When working with multi-use models, the analyst may be required to engage with a group of end-users to develop an understanding of their respective requirements. Where requirements differ or contradict, techniques such as Strategic Options Development and Analysis ([SODA](https://www.ru.nl/publish/pages/938444/soda_-_the_principles.pdf)) and [Soft Systems Methodology] (https://en.wikipedia.org/wiki/Soft_systems_methodology) may be used to develop a shared understanding across multiple groups. From ce4f61bb44724679cfe4d70f5a2a5d5a351c64b7 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Tue, 17 Dec 2024 14:37:53 +0000 Subject: [PATCH 04/26] Added edits to quality_assurance_culture.qmd --- quality_assurance_culture.qmd | 96 +++++++++++++++++------------------ 1 file changed, 48 insertions(+), 48 deletions(-) diff --git a/quality_assurance_culture.qmd b/quality_assurance_culture.qmd index 3b11ca8..1d1a51b 100644 --- a/quality_assurance_culture.qmd +++ b/quality_assurance_culture.qmd @@ -6,53 +6,53 @@ The draft currently has no official status. It is a work in progress and is subj # Quality assurance culture -Creating and maintaining a strong culture of quality assurance is a vital element of ensuring analysis is high quality and robust. +Creating and maintaining a strong culture of quality assurance is vital to ensuring analysis is robust and of a high quality. -This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance that senior leaders should consider for their teams. However, everyone has a role to play in creating a strong quality assurance culture. As such, the approaches outlined are useful for everyone to understand, regardless of their role in a given team. +This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance in their teams. However, everyone has a role to play in creating a strong quality assurance culture and the approaches outlined are useful for everyone. For purposes of this chapter, culture is defined as the shared ways of working, beliefs and habits of an organisation.  +A strong culture of quality assurance means that quality assurance is understood, expected and valued by all those involved in the analytical process, including commissioners, analysts, users of analysis, managers, senior leaders and stakeholders.   -A strong culture of quality assurance means that quality assurance is understood, expected, and valued across all those involved in the analytical process, including commissioners, analysts, users of analysis, managers, senior leaders and stakeholders.   - - -Culture enables the risk management described [elsewhere in this document](proportionality.html#structured-assessment-of-business-risk-and-complexity). +This culture also enables effective [risk management](proportionality.html#structured-assessment-of-business-risk-and-complexity). ## Leadership -A quality assurance culture starts with senior leaders. They are accountable for the quality of analysis carried out in their departments. A important element of discharging this accountability is to clearly set out the priority of quality within their teams and create processes for embedding quality assurance.  +A quality assurance culture starts with senior leaders. They are accountable for the quality of analysis carried out in their departments. Senior leaders should clearly set out the priority of quality within their teams and create processes for embedding quality assurance.  -The guidance in Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the Accounting Officer. In practice the Accounting Officer may assign the responsibility to a senior leader reporting to the senior management board. The Accounting Officer may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero reports this annually - page 91 of [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf). +Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the accounting officer. In practice the accounting officer may assign the responsibility to a senior leader reporting to the senior management board. They may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero (DESNZ) reports this annually in the [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf). -Senior leaders should ensure there is clear messaging and standards on quality assurance, through guidance, training, and regular updates. Senior leaders can demonstrate the importance of quality assurance through long term initiatives, like setting up and embedding quality assurance processes within the team and creating roles and teams to support quality assurance. Senior leaders should also regularly talk about quality with their teams and highlight quality successes. - -::: {.callout-tip} -# Sharing best practice on quality assurance -HM Revenue and Customs have developed a Quality Champions network, made up of analysts from across the department. The network discuss quality assurance initiatives, quality issues and how they were resolved and shares wider best practice. -::: +Senior leaders should ensure there is clear messaging and standards on quality assurance through guidance, training and regular updates. Senior leaders can demonstrate the importance of quality assurance through long term initiatives. For example, by setting up and embedding quality assurance processes within the team and creating roles and teams to support quality assurance. Senior leaders should ensure teams have a common understanding of the quality standards required for thier work and regularly talk about quality with their teams, highlighting quality successes. -As part of a strong quality assurance culture, senior leaders should empower all those in the analytical process to identify any risks to the quality of the work, ensure people at any level can raise any quality concerns, and be able to discuss and constructively challenge each other if they feel those standards are not being met. To underpin this, senior leaders should ensure teams have a common understanding of the quality standards required for their work. +As part of a strong quality assurance culture, senior leaders should empower all those in the analytical process to identify any risks to the quality of the work, ensure people at any level can raise any quality concerns. Teams should be able to discuss and constructively challenge each other if they feel those standards are not being met. Creating transparency at all levels can help embed a culture of quality assurance. This includes peer review, open source of code (where possible), [external publication of models or methods](delivery_and_communication.html#open-publishing-of-analysis) and publication of the register of [Business Critical Analysis](delivery_and_communication.html#business-critical-analysis-register).  -Senior leaders should also develop processes so teams can report when things go wrong, be open and honest when issues occur, carry out reviews to understand the failures in the assurance process and share the lessons learnt across the analytical community. +Senior leaders should also develop processes that enable teams to report when things have gone wrong, be open and honest when issues occur, carry out reviews to understand the failures in the assurance process and share the lessons learnt across the analytical community. ::: {.callout-tip} # An open culture when things go wrong -When the Department for Education made an error producing the schools national funding formula allocations for 2024-25, they ran a detailed internal review to understand what went wrong and why it was not detected by the QA process. The department also commissioned and published an [external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its open learning culture and a culture of taking responsibility for mistakes. +When the Department for Education made an error producing the schools national funding formula allocations for 2024-25, they ran a detailed internal review to understand what went wrong and why it was not detected by the quality assurance process. The department also commissioned and [published an external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its culture of open learning taking responsibility for mistakes. ::: ## Capacity and capability -Senior leaders should create the conditions in which quality assurance processes can operate effectively, by ensuring staff have sufficient time for all stages of the analytical lifecycle, including design, quality assurance and documentation. The culture should also ensure staff can draw on expertise and experience from others, and have the access to the tools and data they need. +Senior leaders should create the conditions in which quality assurance processes can operate effectively by ensuring staff have sufficient time for all stages of the analytical lifecycle, including design, quality assurance and documentation. The culture should also ensure staff can draw on expertise and experience from others and have access to the tools and data they need. ### Capacity -There is a risk that work, and time pressures mean teams cut corners on quality or mistakes are made as analysis is rushed. Senior leaders can mitigate this through strong prioritisation, supporting teams to push back on lower value work or by making tough choices on team wide priorities. Through this prioritisation, senior leaders can emphasise the importance of quality. Senior leaders can support quality assurance, by ensuring it is considered throughout the lifecycle of a project, by all parties involved (analytical and non-analytical), and not simply considered at the end. They should support all parties to make time for adequate quality assurance, even when timescales are tight, as described in later chapters of this book. +There is a risk that work and time pressures could affect the quality of work. Senior leaders can mitigate this through strong prioritisation and supporting teams to push back on lower value work. Through this prioritisation senior leaders can emphasise the importance of quality. Senior leaders can also support quality assurance by ensuring all parties (analytical and non-analytical) consider it at all stages during the life cycle of a project. + +If time constraints mean insufficient assurance has taken place this should be explicitly acknowledged and reported in an assurance statement that sets out the known limitations and conditions associated with the analysis. + +If analysis requires a peer review this should be carried out by independent, skilled and competent individuals or groups. It can be difficult to identify available experts who are able to provide a review but there are several approaches to support and embed independent review. These approaches may include: -If time constraints mean insufficient assurance has taken place, senior leaders should ensure this is explicitly acknowledged and reported. This should be reported via an assurance statement that sets out the known limitations and conditions associated with the analysis. +* setting up specific teams to review and audit a sample of analytical projects +* developing assurance networks of analysts who can provide reviews when needed +* partnering with another department +* procuring an independent review from an independent source such as the [Government Actuary’s Department (GAD)](https://www.gov.uk/government/publications/gad-services/government-actuarys-department-services)), an academic institution or contractor -Where analysis requires peer review, this should be carried out by independent, skilled and competent individuals or groups. However, it can be difficult to identify available experts who are able to provide a review. There are several approaches to support and embed independent review, such as setting up specific teams to review and audit a sample of analytical projects, developing assurance networks of analysts who can provide reviews when needed, partnering with another Department, or procuring an independent review from an independent source such as the Government Actuary’s Department ([GAD](https://www.gov.uk/government/publications/gad-services/government-actuarys-department-services)), an academic institution or contractor. Team leaders can support this by making time for analysts to carry out peer reviews and ensuring analysts are clear that supporting peer reviews is part of their role. +Team leaders can support this by making time for analysts to carry out peer reviews and ensuring analysts are clear that supporting such reviews is part of their role. ::: {.callout-tip} # HM Revenue and Customs independent review team @@ -61,72 +61,72 @@ HMRC has a small analytical team which independently reviews analysis from acros ### Capability -There is a risk that errors occur because of a lack of skills or experience. Senior leaders can identify common skills gaps, creating training or mentoring to help fill gaps in analysts' knowledge. Processes to support knowledge sharing, innovation and dissemination of best practice will all help develop capability. Rolling out training on departmental assurance processes can also mitigate this risk. +There is a risk pf errors occuring because of a lack of skills or experience. Senior leaders can identify common skill or knowledge gaps and provide training or mentoring to help fill these gaps. Processes to support knowledge sharing, innovation and dissemination of best practice will all help develop capability. Rolling out training on departmental assurance processes can also mitigate this risk. -There are various cross-government resources designed to provide support and guidance for commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) aims to help policy professionals to work effectively with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. +There are various cross-government resources to support and guide commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) aims to help policy professionals to work effectively with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. ::: {.callout-tip} # Building assurance capability -The Department for Energy Security and Net Zero are building assurance capacity with a programme of quality assurance Colleges. The Colleges run regular virtual sessions, open to colleagues across Government and Partner Organisations. +DESNZ is building assurance capacity with a programme of quality assurance colleges. The colleges run regular virtual sessions, open to colleagues across government and partner organisations. -These sessions include an interactive activity looking at a purposefully sabotaged model, used to introduce and familiarise colleagues with the quality assurance logbook and the departmental system of actively monitoring models through these logbooks. +These sessions include an interactive activity looking at a purposefully sabotaged model, which is used to introduce and familiarise colleagues with the quality assurance logbook and the departmental system of actively monitoring models using the logbooks. -College participants also join the Modelling Integrity Network, as potential Assuring Analysts where these cannot be found by Lead Analysts in their own policy areas. +College participants also join the Modelling Integrity Network which can help provide assuring analyst capacity in policy areas where lead analysts do not have existing assurance support. ::: - ### Quality assurance champions -In an organisations with large analytical community, it is good practice for the senior leaders to appoint a quality assurance champion/s. This person/group can share best practice in implementation of quality assurance and provide advice on issues such as proportionality and communication of assurance. - +In organisations with large analytical community it is good practice for the senior leaders to appoint quality assurance champions who can share best practice in the implementation of quality assurance and provide advice on issues such as proportionality and communication. +::: {.callout-tip} +# Sharing best practice on quality assurance +HM Revenue and Customs have developed a Quality Champions network of analysts across the department. The network discuss quality assurance initiatives, quality issues and how they were resolved, and shares wider best practice. +::: ## Tools -There are risks of technology or analytical tools being out of date. This means that analysts cannot follow best practice or must spend time fixing processing issues instead of concentrating on quality. -Senior leaders can support teams by: -* making funding available for new tools or improving existing tools; -* gathering common issues; and, -* working with escalate them to appropriate points in their organisation. +There is a risk of technology or analytical tools being out of date. This means that analysts cannot follow best practice or must spend time fixing processing issues instead of concentrating on quality. Senior leaders can support teams by: + +* making funding available for new tools or improving existing tools +* gathering common issues +* escalating issues to appropriate points in their organisation ## Data -There is a risk that data quality and data understanding cause quality issues with analysis. Senior leaders can escalate data quality issues with wider data teams, and champion and oversee changes to improve data quality. +There is a risk that data quality and data understanding cause quality issues with analysis. Senior leaders can escalate data quality issues with wider data teams, championing and overseeing changes to improve data quality. ## Uncertainty in analysis -Uncertainty is intrinsic to government work. Analysis should support government decision making by appropriate treatment of uncertainty. Senior leaders are responsible for instilling a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed, and how to plan for uncertainty. The National Audit Office has [created guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately, and communicated. +Uncertainty is intrinsic to government work. Analysis should support government decision-making by treating of uncertainty approrpiately. Senior leaders are responsible for instilling a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has [created guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated. ## Governance and control -Governance supports a strong quality assurance culture, by overseeing the management and assurance of analysis. The [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) sets out the requirements for a governance framework for analysis. Each organisation should have a defined and established approach to assurance, which should be applied proportionately to the risk and value of the activity and integrated with the organisation's overall assurance framework.  +Governance supports a strong quality assurance culture by overseeing the effective management and assurance of analysis. The [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) sets out the requirements for a governance framework for analysis. Each organisation should have a defined and established approach to assurance. This should be applied proportionately to the risk and value of the activity and integrated with the organisation's overall assurance framework.  -Project level governance can provide oversight over a particular model or work area, allowing the Approver to ensure the analysis is fit for purpose. For example, formally agreeing assumptions (which may be recorded in an [assumptions log](definitions_and_key_concepts.html#assumptions-log)) will reduce the need for reworking the analysis providing more time for assurance. Projects governance can also fit within wider programme level governance.  +Project level governance can provide oversight over a particular model or work area. This will allow the approver to ensure the analysis is fit for purpose. For example, formally agreeing assumptions (which may be recorded in an [assumptions log](definitions_and_key_concepts.html#assumptions-log)) will reduce the need for reworking the analysis providing more time for assurance. Projects governance can also fit within the wider programme level governance.  -Analytical governance boards for new, high-profile or complex pieces of analysis, can allow senior analytical leaders and experts to provide oversight and challenge of analysis and ensure best practice is followed. These groups are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work.  +Analytical governance boards for new, high-profile or complex pieces of analysis can allow senior analytical leaders and experts to provide oversight and challenge of analysis and ensure best practice is followed. These boards are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work.  ## Transparency -Transparency at all levels can help embed a culture of quality assurance. For example peer review, sharing lessons learnt, and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where possible), all contribute to an open culture of high quality work. +Transparency at all levels can help embed a culture of quality assurance. For example, peer review, sharing lessons learnt and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where possible) can all contribute to an open culture of high quality work. ## Externally commissioned analysis +Analysis may be commissioned externally. For example, qualitative or quantitative research, or the production of models. In this case the commissioning department is accountable for analytical quality. The commissioning analyst should ensure that quality standards are clear, being met and that these standards are documented. -Analysis may be commissioned externally such as qualitative or quantitative research, or the production of models. In this case the commissioning department is accountable for analytical quality. The commissioning analyst should ensure that quality standards are clear and being met and that these standards are documented. - -For Arm’s Length Bodies[^1] (ALBs), the Commissioner of analysis is accountable for ensuring the requirements set out in the AQuA Book are met. - -The guidance mentioned above also applies to Arms Length Bodies. They may set out in a framework document how their Accounting Officer will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) +For Arm’s Length Bodies[^1] (ALBs), the commissioner of analysis is accountable for ensuring the requirements set out in the AQuA Book are met. +The guidance mentioned above also applies to ALBs. They may set out in a framework document how their accounting officer will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). -Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) +Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). +When working with third parties the commissioning department shall ensure it is clear which role or roles in which stages of the analytical lifecycle that third party is responsible for. For example, the third party may only undertake the analyst role in the analysis phase or they may undertake the analyst, assurer and approver roles in all stages of the lifecycle. -When working with third parties (e.g Arm's Length Bodies (ALBs)), the commissioning department shall ensure it is clear which role or roles in which stages of the analytical lifecycle that third party is responsible for. See [Roles and Responsibilities](analytical_lifecycle.qmd#roles_and_responsibilities) for details on the roles. +You can read more in the [Roles and Responsibilities](analytical_lifecycle.qmd#roles_and_responsibilities) section. -For example, the third party may only undertake the Analyst role in the analysis phase or they may undertake the Analyst, Assurer and Approver roles in all stages of the lifecycle. [^1]: ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, please see [Cabinet Office guidance on Classification of Public Bodies](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf) From c3c150a461e634556666a4695f5b9eb4e58f6052 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Tue, 17 Dec 2024 15:42:39 +0000 Subject: [PATCH 05/26] Added edits to analytical_lifecycle.qmd --- analytical_lifecycle.qmd | 97 +++++++++++++++++++++------------------- 1 file changed, 50 insertions(+), 47 deletions(-) diff --git a/analytical_lifecycle.qmd b/analytical_lifecycle.qmd index fdbd960..d0a57f0 100644 --- a/analytical_lifecycle.qmd +++ b/analytical_lifecycle.qmd @@ -6,45 +6,48 @@ The draft currently has no official status. It is a work in train and is subject # Roles and the analytical lifecycle -An analytical project can be viewed as a variation on an archetypal project defined by roles and stages. This chapter gives an overview of the roles and the project stages in the analytical lifecycle. Subsequent chapters go into the details of each stage, and the responsibilities of each role during a given stage. +An analytical project can be viewed as a variation on an archetypal project defined by roles and stages. This chapter gives an overview of the roles and the project stages in the analytical lifecycle. Chapters 6 to 9 give more explanation of each stage and the responsibilities of each role during a given stage. ## Roles and responsibilities Organisations may have their own titles for the main functional roles involved in analysis that are set out here. -Each role may be fulfilled by a team or committee of people. However, a single individual will have overall accountability (such as the chair of a committee) for each role. +Each role may be fulfilled by a team or committee of people. However, a single individual (for example, the chair of a committee) will have overall accountability for each role. +The AQuA Book defines four roles. -The AQuA book defines the following roles: +The commissioner (may be known as customer): -* **Commissioner** (may be known as customer) - + Requests the analysis and sets out their requirements - + Agrees what the analyst is going to do will satisfy the need - + Accepts the analysis and assurance as fit for purpose +* requests the analysis and sets out their requirements +* agrees what the analyst is going to do will satisfy the need +* accepts the analysis and assurance as fit for purpose -* **Analyst** - + Designs the approach, including the assurance, to meet the commissioner’s requirements - + Agrees the approach with the Commissioner - + Carries out the analysis - + Carries out their own assurance - + Acts on findings from the Assurer - + Can be a group of analysts, in which case the lead analyst is responsible +The analyst: -* **Assurer** (may be known as Analytical Assurer, Assuring Analyst) - + Reviews the assurance completed by the Analyst - + Carries out any further validation and verification they may see as appropriate - + Reports errors and areas for improvement to the analyst - + Re-reviews as required - + Confirms the work has been appropriately scoped, executed, validated and verified and documented to the Approver - + Can be a group of assurers. In which case the leader of the group is responsible. They must be independent from the analysts. +* designs the approach, including the assurance, to meet the commissioner’s requirements +* agrees the approach with the commissioner +* carries out the analysis +* carries out their own assurance +* acts on findings from the assurer +* can be a group of analysts, in which case the lead analyst is responsible -* **Approver** (may be known as Senior Analyst or Senior Responsible Officer (“SRO”)) - + Scrutinises the work of the Analyst and Assurer - + Confirms (if necessary) to the Analyst, Assurer and Commissioner that the work has been appropriately assured - -The roles of Analyst and Assurer shall be distinct from each other. The Analyst should carry out their own assurance but responsibility for formal assurance to the Approver and Commissioner lies with the Assurer. In some instances, particularly for quick and / or simple analysis, an individual may deliver more than one of the roles apart from the Assurer and Analyst roles which shall be separate from one another in all cases. +The assurer (may be known as the analytical assurer or assuring analyst): + +* reviews the assurance completed by the analyst +* carries out any further validation and verification they may see as appropriate +* reports errors and areas for improvement to the analyst +* rundertakes repeated reviews as required +* confirms the work has been appropriately scoped, executed, validated and verified and documented to the Approver +* can be a group of assurers, in which case the leader of the group is responsible +* must be independent from the analysts +The approver (may be known as senior analyst or senior responsible officer): + +* scrutinises the work of the analyst and assurer +* confirms (if necessary) to the analyst, assurer and commissioner that the work has been appropriately assured + +The roles of analyst and assurer shall be distinct from each other. The analyst should carry out their own assurance but responsibility for formal assurance to the approver and commissioner lies with the assurer. In some instances, particularly for quick or simple analysis, an individual may fulfill more than one of the roles, apart from the assurer and analyst roles which shall be separate from one another in all cases. ## The analytical lifecycle @@ -54,52 +57,52 @@ Quality assurance activities should take place throughout all stages of an analy Figure 2 is adapted from the [Government Functional Standard for Analysis](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). Analytical quality assurance activities should take place during every phase of the cycle and should consider proportionality, although analytical quality considerations may vary depending on project governance and the specific phase of the cycle. All projects will involve some element of every phase of the cycle, even if this is not clearly defined. -It is important that [proportionality](proportionality.qmd) is considered and that there is transparency of the analytical decisions, process, limitations and changes made at each stage to enable effective assurance and communication. This should be enabled by: +It is important that [proportionality](proportionality.qmd) is considered and that there is transparency of the analytical decisions, process, limitations and changes made at each stage to enable effective assurance and communication. This should be clearly shown in: -* Clear documentation of the analysis, assumptions and data, -* Clear records of the analytical decisions made; and -* Clear records of the quality assurance processes and checks completed. +* documentation of the analysis, assumptions and data +* records of the analytical decisions made +* records of the quality assurance processes and checks completed ### Engagement and scoping -Analytical projects typically start with customer engagement although other events may trigger analytical projects. Scoping ensures that an appropriate, common understanding of the problem is defined and that expectations are aligned with what can be delivered. During this phase the Commissioner plays an important role in communicating the questions to be addressed and working with the Analyst to ensure the requirements and scope are defined and understood. +Analytical projects typically start with customer engagement although other events may trigger analytical projects. Scoping ensures that an appropriate, common understanding of the problem is defined and that expectations are aligned with what can be produced. During this phase the commissioner plays an important role in communicating the questions to be addressed and working with the analyst to ensure the requirements and scope are defined and understood. -Where analysis requires multiple cycles, for example to develop, use and update analytical models, this phase may follow on from the Delivery and Communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project. +Where analysis requires multiple cycles, for example to develop, use and update analytical models, this phase may follow on from the delivery and communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project. -More effort may be needed to define the requirements and scope in this phase for research, evaluation or other projects that may need to seek a wider range of perspectives or for which subsequent phases and work may be delivered through a product or service. +In this phase more effort may be needed to define the requirements and scope in this phase for research, evaluation or other projects that may need to seek a wider range of perspectives or for which subsequent phases and work may be provided through a product or service. ### Design -During the design phase, the Analyst will convert the commission into an analytical plan, including the assurance required and ensuring it is sufficient to answer the questions posed. This phase includes the communication and approval of plans produced, and some iteration between the Commissioner and the Analyst is to be expected as the analytical solution is developed and limitations understood. +During the design phase, the analyst will convert the commission into an analytical plan. This will include the assurance required and ensuring the analysis is sufficient to answer the questions posed. This phase includes the communication and approval of plans produced, and some iteration between the commissioner and the analyst is to be expected as the analytical solution is developed and limitations understood. -For larger projects or those that require multiple cycles, the design phase may include consideration of the staging of work over the whole scope of the project as well as work required in each stage. Analysis plans for work that is dependent on insights from earlier stages may be high-level and necessitate a return to the design phase at a later date. +For larger projects or those that require multiple cycles, the design phase may include consideration of the staging of work over the whole scope of the project as well as the work required in each stage. Analysis plans for work that is dependent on insights from earlier stages may be high-level and necessitate a return to the design phase at a later date. ### Analysis -The analysis phase is where planned analysis is undertaken, and progress and relevance are monitored. During work, the design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered, and this phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the scoping or design phase. +The analysis phase is where planned analysis is undertaken, and progress and relevance are monitored. During work, the design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases the changes or limitations encountered may necessitate a return to either the scoping or design phase. -Throughout this phase, traceable documentation of the assurance activities undertaken shall also be produced. +Throughout this phase traceable documentation of the assurance activities undertaken shall also be produced. -In larger analytical projects, some outputs of the analysis may be completed at different times as work develops, and aspects of other phases may therefore take place concurrently. +In larger analytical projects, some outputs of the analysis may be completed at different times as work develops and aspects of other phases may therefore take place concurrently. ### Delivery, communication and sign-off -During the delivery stage, insights and analytical assurances are communicated to the Approver and the Commissioner. The aim is ensuring that these are sufficiently understood in order for the Approver and Commissioner to determine whether the work has been appropriately assured and meets their requirements. This may then trigger additional analysis and further assurance as analytical projects frequently need further iteration or extension to satisfy the Commissioner's needs. +During the delivery stage, insights and analytical assurances are communicated to the approver and the commissioner, These should be sufficiently understood in order for the approver and commissioner to determine whether the work has been appropriately assured and meets their requirements. Additional analysis and further assurance may be required as analytical projects frequently need further iteration or extension to satisfy the commissioner's needs. -Work in this stage can vary considerably depending on the commission, impact, approval processes and the nature of the project. Delivery and communication activities may include producing summary packs and reports, launching dashboards or websites and presentations to boards. +Work in this stage can vary considerably depending on the commission, impact, approval processes and the nature of the project. Delivery and communication activities may include producing summary packs and reports, launching dashboards or websites and presentations. -After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured, and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced. +After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced. ## Maintenance and continuous review -The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether +The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made as to whether: -*The inputs used remain appropriate -*The initial communication methods remain the best way to deliver the information -*Any software relied on continues to be supported and up to date -*The model continues to be calibrated appropriately (this is particularly important for [black box models](definitions_and_key_concepts.html#black-box-models)) +* the inputs used remain appropriate +* the initial communication methods remain the best way to deliver the information +* any software relied on continues to be supported and up to date +* the model continues to be calibrated appropriately (this is particularly important for [black box models](definitions_and_key_concepts.html#black-box-models)) Additionally, a robust version control process should be in place to ensure any changes to the analysis are appropriately assured. ## Urgent analysis -In some cases there may be a need for urgent analysis that cannot follow all the steps in this guide i.e. where the need for analysis outweighs the risk of poor quality. In this case analysts should follow the [Urgent data quality assurance guidance](https://www.gov.uk/government/publications/urgent-data-quality-assurance-guidance/urgent-data-quality-assurance-guidance). +In some cases there may be a need for urgent analysis that cannot follow all the steps in this guide. For example, where the need for analysis outweighs the risk of poor quality. In this case analysts should follow the [Urgent data quality assurance guidance](https://www.gov.uk/government/publications/urgent-data-quality-assurance-guidance/urgent-data-quality-assurance-guidance). From a703160e1e23fc07d83824d33c6a1b8dd95c90ab Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Wed, 18 Dec 2024 09:59:11 +0000 Subject: [PATCH 06/26] Added edits to design.qmd --- design.qmd | 126 ++++++++++++++++++++++++----------------------------- 1 file changed, 58 insertions(+), 68 deletions(-) diff --git a/design.qmd b/design.qmd index 33851d2..abd3a33 100644 --- a/design.qmd +++ b/design.qmd @@ -8,123 +8,113 @@ The draft currently has no official status. It is a work in progress and is subj ## Introduction and overview -The design stage is where the Analyst translates the scope for the analysis agreed with the Commissioner into an actionable analytical plan. This chapter sets out recommended practices around designing the analysis and associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design, and design of multi-use and AI models. +During the design stage the analyst creates an actionable analytical plan from the scope for the analysis agreed with the commissioner. This chapter sets out recommended practices around designing the analysis, deeciding on the associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design and the design of multi-use and Artifical Intelligence (AI) models. ### The analytical plan The development of the analytical plan should consider: -* Methodology for producing results, including the treatment of uncertainty; -* Project management approach (for example Agile, Waterfall or a combination of approaches); -* Sourcing of inputs and assumptions; -* Data and file management; -* Change management and version control; -* Programming language and/or software; -* Code management, documentation and testing; -* Communication between stakeholders; -* Verification and validation procedures during the project lifetime; -* Documentation to be delivered; -* Process for updating the analytical plan; -* Process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date; -* Ethics; -* Reporting; -* Downstream application. - -The use of [Reproducible Analytical Pipelines (RAP)](#rap) is encouraged as a means of effective project design. - -Iteration between the Commissioner and the Analyst is normal and expected whilst the analytical design develops. +* methodology for producing results, including the treatment of uncertainty +* project management approach (for example agile, waterfall or a combination of approaches) +* sourcing of inputs and assumptions +* data and file management +* change management and version control +* programming language and software +* code management, documentation and testing +* communication between stakeholders +* verification and validation procedures during the project lifetime +* documentation to be delivered +* process for updating the analytical plan +* process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date +* ethics +* reporting +* downstream application + +Iteration of the plan between the commissioner and the analyst is normal and expected while the analytical design develops. ::: {.callout-tip} # Reproducible analytical pipelines -The recommended approach for developing analysis in code is to use a [Reproducible Analytical Pipeline (RAP)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/). Reproducible Analytical Pipelines shall: +The recommended approach for developing analysis in code is to use a [Reproducible Analytical Pipeline (RAP)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/). RAPs shall: -* Follow the practices set out in the [Analysis Function Quality Assurance of Code for Analysis and Research manual](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html). -* Meet the requirements of the [Reproducible Analytical Pipelines minimum viable product](https://github.com/best-practice-and-impact/rap_mvp_maturity_guidance/blob/master/Reproducible-Analytical-Pipelines-MVP.md). +* follow the practices set out in the [Analysis Function Quality Assurance of Code for Analysis and Research manual](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) +* meet the requirements of the [RAPs minimum viable product](https://github.com/best-practice-and-impact/rap_mvp_maturity_guidance/blob/master/Reproducible-Analytical-Pipelines-MVP.md) ::: - - ## Roles and responsibilities in the design stage -### The Commissioner's responsibilities during the design stage -The Commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the Commissioner may review the analytical plan. - -The Commissioner's domain expertise can be a useful resource for the analyst in the design stage. The Commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs, all of which can inform the design. - - - -### The Analyst's responsibilities during the design stage -The Analyst should: +### The commissioner's responsibilities +The commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the commissioner may review the analytical plan. -* develop the method and plan to address the Commissioner’s needs; -* establish assurance requirements; -* develop the plan for proportionate verification and validation - see [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf); -* plan in sufficient time for the assurance activity; -* document the analytical plan in a proportionate manner; and, -* follow any organisation governance procedures for project design. +The commissioner's domain expertise can be a useful resource for the analyst in the design stage. The commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs, all of which can inform the design. +### The analyst's responsibilities -### The Assurer's responsibilities during the design stage -The Assurer should review the analytical plan to ensure that they are able to conduct the required assurance activities. They may provide feedback on the analytical plan. The Assurer should plan sufficient time for the assurance activity. +The analyst should: +* develop the method and plan to address the commissioner’s needs +* establish assurance requirements +* develop a plan for proportionate verification and validation as described in the [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf); +* plan in sufficient time for the assurance activity +* document the analytical plan in a proportionate manner +* follow any organisation governance procedures for project design +### The assurer's responsibilities +The Aasurer should review the analytical plan to ensure that they are able to conduct the required assurance activities. They may provide feedback on the analytical plan. The Assurer should plan sufficient time for the assurance activity. -### The Approver's responsibilities during the design stage -In smaller projects, the Approver may not be heavily involved in the design stage. However, for business critical analysis, the Approver may want to confirm that organisational governance procedures for design have been followed. +### The approver's responsibilities +In smaller projects, the approver may not be heavily involved in the design stage. However, for business critical analysis, the approver may want to confirm that organisational governance procedures for design have been followed. -## Assurance activities in the design stage +## Assurance activities - -On completion of the design stage, the Assurer should be aware of the quality assurance tasks that will be required of them during the project lifetime and have assured the necessary elements of the analytical plan. +When the design stage has been completed the assurer should be aware of the quality assurance tasks that will be required of them during the project lifetime and have assured the necessary elements of the analytical plan. The assurance of the design stage should consider whether the analytical plan is likely to: -* Address commissioner's requirements - validation; -* Deliver as intended - verification; -* Be robust i.e. well-structured, data driven, with a sound overall design. +* address commissioner's requirements (validation) +* deliver as intended (verification) +* be robust (for exmaple, provide a well-structured, data driven plan with a sound overall design) -The assurance of the design stage may be carried out by the Assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance, and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases. +The assurance of the design stage may be carried out by the assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance, and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases. -## Documentation in the design stage +## Documentation -The design process should be documented in a proportionate manner. A design document that records the [analytical plan](#the-analytical-plan) should be produced by the Analyst and signed-off by the Commissioner. The design document may be reviewed by the Assurer. +The design process should be documented in a proportionate manner. A design document that records the [analytical plan](#the-analytical-plan) should be produced by the analyst and signed-off by the commissioner. The design document may be reviewed by the assurer. For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage. - It is best practice to use formal version control to track changes in the design document. - - ## Treatment of uncertainty in the design stage -During the design stage, Analysts should examine the planned analysis systematically for possible sources and types of uncertainty, to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error. This is discussed in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html) +During the design stage, analysts should examine the planned analysis systematically for possible sources and types of uncertainty. This is to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error. + +You can read more in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html) -## Black box models and the design stage +## Black box models Using [black box models](definitions_and_key_concepts.qmd/#black-box-models) places greater weight on the design of the analysis and the assurance and validation of outputs by domain experts. This [guidance on AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design of AI models, including risk assessment, impact assessment, bias audits and compliance audits. -In the Design of AI/ML models, the Analyst should: +In the design of AI and machine learning models, the analyst should: -* define the situation they wish to model; -* the prediction they wish to make; -* the data that could be used to make the prediction; -* carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach; -* consider how to separate the data for the design and testing of models - it's usual to design a model with a fraction of the data and then test it with the data that was not used in the design; -* consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions; and, -* consider whether to refer the model to their ethics committee, or a similar group - see the [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020). +* define the situation they wish to model +* the prediction they wish to make +* the data that could be used to make the prediction +* carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach +* consider how to separate the data for the design and testing of models - it's usual to design a model with a fraction of the data and then test it with the data that was not used in the design +* consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions +* consider whether to refer the model to their ethics committee, or a similar group as described in the [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020) -## Multi-use models and the design stage +## Multi-use models Designing multi-use models should take into account the needs of all users of the analysis. An Analysis Steering Group may be an effective means for communication about the design with a range of user groups. -The design of multi-use models may entail a modular structure with different Analysts and Assurers responsible for different elements. The design of assurance activities should capture both the assurance of individual modules and their integration. +The design of multi-use models may entail a modular structure with different analysts and assurers responsible for different elements. The design of assurance activities should capture both the assurance of individual modules and their integration. From ab0d5b7c3360baab205b933725c8abee4949df2e Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Wed, 18 Dec 2024 10:25:05 +0000 Subject: [PATCH 07/26] Added edits to analysis.qmd --- analysis.qmd | 140 +++++++++++++++++++++++++++------------------------ 1 file changed, 75 insertions(+), 65 deletions(-) diff --git a/analysis.qmd b/analysis.qmd index d3214e8..39e8090 100644 --- a/analysis.qmd +++ b/analysis.qmd @@ -6,123 +6,133 @@ The draft currently has no official status. It is a work in progress and is subj # Analysis -## Introduction and overview - -The analysis stage is where planned analysis is undertaken and assured, and progress and relevance are monitored. During this stage, the design may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the design or scoping stage. +During the analysis stage the planned analysis is undertaken and assured, and progress and relevance are monitored. The design may be amended to account for any changing circumstances, emerging information, unexpected difficulties or limitations that may be encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may mean that the design or scoping stage need to be revisited to address these issues. ## Roles and responsibilities in the analysis stage -### The Commissioner's responsibilities during the analysis stage - -* The Commissioner should be available to provide input and clarifications to the Analyst. - -* The Commissioner’s should review any changes in design or methodology that the Analyst brings to their attention. - - - - -### The Analyst's responsibilities during the analysis stage +### The commissioner's responsibilities -* The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage). They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate. +The Commissioner should: -* When the analysis includes coding, the Analyst shall proportionately follow [best practice for code development](#assurance-of-code). +* be available to provide input and clarifications to the analyst +* review any changes in design or methodology that the analyst brings to their attention -* The Analyst shall produce documentation of the data (see [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach. -* The Analyst shall document any changes to the analytical plan in a proportionate manner. +### The analyst's responsibilities -* The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner's needs or whether there are any new requirements. +The analyst shall: +* follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage) +* provide traceable documentation of the assurance they have undertaken +* respond to recommendations from the assurer and act on them as appropriate +* proportionately follow [best practice for code development](#assurance-of-code), where relevant +* produce documentation of the data (as described in [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) +* produce documentation of the data (as described in [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used +* ensure all documentation is sufficient for the assurer to understand the approach +* document any changes to the analytical plan in a proportionate manner +* maintain appropriate contact with commissioner and assurer to provide an opportunity for them to advise on whether the analysis is still meeting the commissioner's needs or whether there are any new requirements -### The Assurer's responsibilities during the analysis stage +### The assurer's responsibilities -* The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required. +The assurer shall: -* When the analysis includes coding, the Assurer shall review that the work proportionately adheres to [best practice for code development](#assurance-of-code). +* review the assurance completed by the analyst +* carry out any further validation and verification they may see as appropriate +* report errors and areas for improvement to the analyst +* review that the work proportionately adheres to [best practice for code development](#assurance-of-code), where relevant. -* The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology. +The assurer may need to: +* re-review the analytical work completed, as required +* provide feedback on changes to the analytical plan, +* consider whether they are qualified to provide rigorous assurance on the revised methodology +### The approver's responsibilities +The approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage. -### The Approver's responsibilities during the analysis stage - -The Approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage. - - -## Assurance activities in the analysis stage +## Assurance activities ### Verification and validation -Verification that the implemented methodology meets the design requirements should be incorporated as part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling. These techniques extend to analysis more broadly. These include: +Verification that the implemented methodology meets the design requirements should be part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling but these techniques also extend to analysis more broadly. They include: -- Informal analysis: techniques that rely on human reasoning and subjectivity. -- Static analysis: tests that the implementation of the analysis before it is run. For example, checking that code adheres to code conventions, structural analysis of the code by examining graphs of control and data flows, . -- Dynamic analysis: tests the behaviour of the system, model or code to find errors that arise during execution. This includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing)) -- Symbolic analysis: particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during the execution of a model. Includes path tracing and cause-effect testing (see [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) ) -- Constraint analysis: particularly relevant to modelling and tests the implementation of constraints during model execution. This includes checking the assertions of the model and boundary analysis. -- Formal analysis: tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs. +* informal analysis - techniques that rely on human reasoning and subjectivity +* tatic analysis - tests that the implementation of the analysis before it is run (for example, checking that code adheres to code conventions) +* dynamic analysis - tests the behaviour of the system, model or code to find errors that arise during execution, includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing)) +* symbolic analysis - particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during [the execution of a model](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf), includes path tracing and cause-effect testing +* constraint analysis - particularly relevant to modelling and tests the implementation of constraints during model execution, includes checking the assertions of the model and boundary analysis +* formal analysis - tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs. +Validation refers to testing whether the product meets the requirements of users. It is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility. +Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. -Validation refers to testing whether the product meets the requirements of users. Hence, it is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility. +You can read more about [techniques that may be useful in validation of models](https://www.informs-sim.org/wsc11papers/016.pdf) -Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. Techniques that may be useful in validation of models are reviewed by [Sargent (2011)](https://www.informs-sim.org/wsc11papers/016.pdf). - -The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification. +The analyst has primary responsibility for conducting verification and validation. The assurer is responsible for reviewing the verification and validation that is carried out by the analyst, and for conducting or recommending additional verification and validation as required. The assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification. ### Data validity and data considerations -Testing data validity (i.e. that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics (outliers, trends, expected distributions etc), and assuring robust data management practices (e.g. automating data creation and data sourcing). +Testing data validity (for example, ensuring that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics such as outliers, trends and expected distributions, and assuring robust data management practices such as automating data creation and data sourcing. -It is rare to have the perfect dataset for an analytical commission. Reasons for this include: +It is rare to have the perfect dataset for an analytical commission. This could be because: -* The data is not available in the time frame required for the ideal analysis; -* The data definition does not perfectly align with the commission; -* There are data or coverage gaps; -* The data may be experimental or there are other reasons why it is not ‘mature’. +* the data is not available in the time frame required for the ideal analysis +* the data definition does not perfectly align with the commission +* there are data or coverage gaps +* the data may be experimental or there are other reasons why it is not mature -Often, no data is available that are directly and precisely relevant to the parameter and conditions of interest. In such cases, it is often possible to use surrogate data. This is measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. The use of surrogate data introduces further uncertainty, additional to that associated with the data itself. It may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. +When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. -The effect of using a proxy dataset should be explored and, if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration, and the decision to use a particular dataset or input, should be recorded for the benefit of the Assurer. +The effect of using a proxy dataset should be explored and if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration and the decision to use a particular dataset or input should be recorded for the assurer to verify. ### Assurance of code -The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation. The Analyst shall follow the guidance for good quality code development in a proportionate manner, and the Assurer shall review this accordingly. +The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on: + +* structuring code +* producing documentation +* using version control +* data management +* testing +* peer review +* automation +The analyst shall follow the guidance for good quality code development in a proportionate manner and the assurer shall review this accordingly. -## Documentation in the analysis stage -The Analyst should: +## Documentation -* Maintain appropriate records of the work; -* Fully document any code following agreed standards; -* Log the data, assumptions and inputs used in the analysis, and decisions made (see [documentation](definitions_and_key_concepts.html/#documentation)); -* Record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis; -* Produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation). For modelling, the Analyst may include a model map that describes data flows and transformations. +The analyst should: -## Treatment of uncertainty in the analysis stage +* aaintain appropriate records of the work +* fully document any code following agreed standards +* log the data, assumptions and inputs used in the analysis, and decisions made in appropriate [documentation](definitions_and_key_concepts.html/#documentation)) +* record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis +* produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation). -While the Scoping and Design stages identified and described risks and uncertainties, the Analysis stage aims to assess and quantify how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty. +For modelling, the analyst may include a model map that describes data flows and transformations. +## Treatment of uncertainty +While the scoping and design stages identified and described risks and uncertainties, the analysis stage aims to assess and quantify how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty. -## Black box models and the analysis stage +## Black box models -[Black box models](definitions_and_key_concepts.html/#black-box-models) such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. +[Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. -Assurance activities during the Analysis stage: +Assurance activities during the analysis stage: * may include performance testing in a live environment and -* should include the verification steps set out in the Design Phase +* should include the verification steps set out in the design stage * should include validation and verification of automatic tests to ensure the model behave as expected -See the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) for further details. - +You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) -## Multi-use models and the analysis stage +## Multi-use models -In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This calls for mechanisms for assuring that the changes integrate into the larger model as expected, for example, through the use of test-suites. +In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This requires mechanisms for assuring that the changes integrate into the larger model as expected. For example, through the use of test-suites. From 202b19578085a0ba0347e719ba498dd53480df46 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Wed, 18 Dec 2024 11:08:59 +0000 Subject: [PATCH 08/26] Added edits to delivery_and_communication.qmd --- delivery_and_communication.qmd | 232 ++++++++++++++++----------------- 1 file changed, 113 insertions(+), 119 deletions(-) diff --git a/delivery_and_communication.qmd b/delivery_and_communication.qmd index 4deff65..d529a20 100644 --- a/delivery_and_communication.qmd +++ b/delivery_and_communication.qmd @@ -6,198 +6,192 @@ The draft currently has no official status. It is a work in progress and is subj # Delivery, communication and sign-off -## Introduction and overview -The successful delivery of analysis to its Commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision making in your organisation and possibly inform the public. +The successful delivery of analysis to the commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision-making in your organisation and, possibly, inform the public. This chapter provides information on the processes around assurance of communication of analysis and delivery of analytical output. - ## Roles and responsibilities in delivery, communication and sign-off +### The commissioner's responsibilities +The commissioner shall: -### The Commissioner's responsibilities during delivery, communication and sign-off +* confirm that the analysis is likely to meet their needs +* use the analysis as specified +* understand and apply any limitations to its use -The Commissioner shall -* confirm that the analysis is likely to meet their needs; -* use the analysis as specified; and, -* understand and apply any limitations to its use. +### The analyst's responsibilities -### The Analyst's responsibilities during delivery, communication and sign-off +The analyst shall: -The Analyst -- shall follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the [business-critical analysis register](#business-critical-analysis-register), and making the analysis [publicly available](#open-publishing-of-analysis); -- shall decribe any limitations to using the analysis; -- should ensure that communication meets audience requirements e.g. on accessibility; -- should be prepared to respond to challenge from the Approver e.g. scruitiny from project or programme boards; -- may be required to communicate the assurance state to the Approver, if not done directly by the Assurer. +* follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the [business-critical analysis register](#business-critical-analysis-register) and making the analysis [publicly available](#open-publishing-of-analysis) +* describe any limitations to using the analysis +The analyst should: -### The Assurer's responsibilities during delivery, communication and sign-off +* ensure that communication meets audience requirements such as accessibility +* be prepared to respond to challenge from the approver or scruitiny from project or programme boards -The Assurer shall communicate the assurance state to the Approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented, and provides adequate handling of uncertainty. This communication may go via the Analyst. +The analyst may be required to communicate the assurance state to the approver if this is not done directly by the assurer. +### The assurer's responsibilities -### The Approver's responsibilities during delivery, communication and sign-off -* The Approver shall review the assurance evidence that has been provided to them. -* The Approver should provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose. -* The Approver shall be confident that the analysis meets the design requirements, is of sufficient quality and is adequately and proportionately documented. -* When they are satisfied with the validity and robustness of the analysis, the Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved. -* The Approver shall follow organisation governance procedures for sign-off, including updating of the [business-critical analysis register](#business-critical-analysis-register), where appropriate. +The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may go via the analyst. +### The approver's responsibilities -## Assurance activities in the delivery, communication and sign-off stage +The approver shall: +* review the assurance evidence that has been provided to them +* be confident that the analysis meets the design requirements, is of sufficient quality and is adequately and proportionately documented +* shall follow organisation governance procedures for sign-off, including updating of the [business-critical analysis register](#business-critical-analysis-register), where appropriate +The approver should: -### Delivery +* provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose +* provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved when they are satisfied with the validity and robustness of the analysis -When delivering a piece of analysis, the Analyst and/or Assurer should communicate its assurance state to the Approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance and to demonstrate that the analysis is ready for delivery, for example: - -* It uses suitable data and assumptions; -* It has provisions for regular review; -* It meets the purpose of its commission; -* It has been carried out correctly and to its agreed specification; -* It has a risk assessment and statement against the programme risk register; -* It meets analytical standards, such as those around coding standards and documentation; -* It adheres to any professional codes of practice (e.g. [The Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/) -* Where appropriate the analysis is accompanied by a completed [assurance statement](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fassets.publishing.service.gov.uk%2Fmedia%2F65c5021f9c5b7f0012951b83%2F20231101-analysis-evidence-quality-assurance-report-gov-uk-E02.docx&wdOrigin=BROWSELINK). +## Assurance activities -Though not strictly assurance, the analyst should also consider areas such as security ratings, retention policies, intellectual property, ethics and related concerns. +### Delivery +When delivering a piece of analysis assurer, or analyst, should communicate its assurance state to the approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance. They should also demonstrate that the analysis is ready for delivery. This may include confirming that the analysis: + +* uses suitable data and assumptions +* has provisions for regular review +* meets the purpose of its commission +* has been carried out correctly and to its agreed specification +* has a risk assessment and statement against the programme risk register; +* meets analytical standards, such as those around coding standards and documentation; +* adheres to any professional codes of practice (for example, [The Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/) +* is accompanied by a completed [assurance statement](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fassets.publishing.service.gov.uk%2Fmedia%2F65c5021f9c5b7f0012951b83%2F20231101-analysis-evidence-quality-assurance-report-gov-uk-E02.docx&wdOrigin=BROWSELINK), where appropriate +Though not strictly assurance, the analyst should also consider areas such as: -The Approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard, which considers the [proportionality](proportionality.html) of the work. The Approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required. +* security ratings +* retention policies +* intellectual property +* ethics and related concerns +The approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard, which considers the [proportionality](proportionality.html) of the work. The approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required. -The exact nature of any scrutiny made by the Approver should be proportionate to the effect the analysis is likely to have, the governance process of their programme/ organisation, and follow the principles of proportionality described in [Chapter 3](proportionality.html) of this document. +The exact nature of any scrutiny made by the approver should be proportionate to the effect the analysis is likely to have, the governance process of their programme/ organisation, and follow the [principles of proportionality](proportionality.html). -To ensure that the analysis is used as intended, the Commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the Analyst. +To ensure that the analysis is used as intended, the commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the analyst. ### Communication -The effective and transparent communication is essential to enable analysis to be adopted and trusted by the Commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including: +The effective and transparent communication is essential to ensure analysis is adopted and trusted by the commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including: -* Commissioners and users of the analysis; -* External scrutiny including the [Public Accounts Committee](https://committees.parliament.uk/committee/127/public-accounts-committee/), the [National Audit Office](https://www.nao.org.uk/), internal and external audit; -* The public, through publications and [Freedom of Information Act](https://www.legislation.gov.uk/ukpga/2000/36/contents) requests; -* Academic experts, possibly through a departmental [Areas of Research Interest](https://www.gov.uk/government/publications/dwp-areas-of-research-interest-2023/dwp-areas-of-research-interest-2023) document. -* Governmental partners - both national and international +* commissioners and users of the analysis +* external scrutiny including the [Public Accounts Committee](https://committees.parliament.uk/committee/127/public-accounts-committee/), the [National Audit Office](https://www.nao.org.uk/), internal and external audit +* the public, through publications and [Freedom of Information Act](https://www.legislation.gov.uk/ukpga/2000/36/contents) requests +* academic experts, possibly through a departmental [Areas of Research Interest](https://www.gov.uk/government/publications/dwp-areas-of-research-interest-2023/dwp-areas-of-research-interest-2023) document +* national and international governmental partners The form of communication should be tailored to the audience. The communication should be quality assured in a proportionate manner to ensure an accurate reflection of the analytical results. -The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that people with disabilities can use them. More specifically, people can: perceive, understand, navigate, and interact with the web. +The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that they meet accessibility guidelines. -If publishing the outcome of any analysis is required, the Analyst should follow departmental and statutory guidance. Some examples are given below: +If the outcome of any analysis needs to be published the analyst should follow departmental and statutory guidance. There are different guidelines depending on the work that is being published. These are: -* If you are publishing statistics you will need to follow your organisation's guidance and the [regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/) ; -* If you are publishing research, you shall follow your organisations guidance and the [Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols); -* If you are publishing an evaluation, refer to any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force); +* [The regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/) ; +* [The Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols) +* any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force) for evaluations ### Sign-off -The exact nature of the approval process may vary depending on: -* The effect the analysis is likely to have; -* The approval process of the organisation, and -* The nature of the programme, project or board approving the analysis. +The exact nature of the approval process may vary depending on the: -The formality of the sign-off process should be governed by organisational procedures, and be proportionate to the analysis. +* effect the analysis is likely to have; +* approval process of the organisation, and +* nature of the programme, project or board approving the analysis -The Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project / programme board where the decision to approve the analysis was made or similar. +The formality of the sign-off process should be governed by organisational procedures and be proportionate to the analysis. +The approver should provide the analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project or programme board where the decision to approve the analysis was made. -## Documentation in the delivery, communication and sign-off stage -When the Analyst and Assurer are satisfied that the analysis is ready to hand over to the Commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation may include: +## Documentation -* [Specification](definitions_and_key_concepts.html#specification-documentation) and [design documentation](definitions_and_key_concepts.html#design-documentation) -* Logs of [data](definitions_and_key_concepts.html#data-log), [assumptions](definitions_and_key_concepts.html#assumptions-log) and [decisions](definitions_and_key_concepts.html#decisions-log) including their source, ownership, reliability and any sensitivity analysis carried out; -* [User and technical documentation](definitions_and_key_concepts.html#user-technical-documentation) -* Advice on uncertainty and its affect on the outputs of the analysis; -* A description of the limits of the analysis and what it can and cannot be used for; -* Any materials for presenting the analysis to the Commissioner, for example slide decks or reports; -* A record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting; -* The code-base, when it has been agreed to [publish the analysis openly](#open-publishing-of-analysis) -* The test plan and results of the tests made against that plan; -* A statement of assurance; -* A statement that ethical concerns have been addressed, especially in cases that include the application of [black-box models](definitions_and_key_concepts.html#black-box-models); +When the analyst and assurer are satisfied that the analysis is ready to hand over to the commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation may include: +* [specification](definitions_and_key_concepts.html#specification-documentation) and [design documentation](definitions_and_key_concepts.html#design-documentation) +* logs of [data](definitions_and_key_concepts.html#data-log), [assumptions](definitions_and_key_concepts.html#assumptions-log) and [decisions](definitions_and_key_concepts.html#decisions-log) including their source, ownership, reliability and any details of any sensitivity analysis carried out +* [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation) +* advice on uncertainty and its affect on the outputs of the analysis +* a description of the limits of the analysis and what it can and cannot be used for +* any materials for presenting the analysis to the commissioner (for example, slide decks or reports) +* a record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting; +* the code-base, when it has been agreed to [publish the analysis openly](#open-publishing-of-analysis) +* the test plan and results of the tests made against that plan +* a statement of assurance +* a statement confirming that ethical concerns have been addressed, especially in cases that include the application of [black-box models](definitions_and_key_concepts.html#black-box-models) - +## Treatment of uncertainty -## Treatment of uncertainty in the delivery, communication and sign-off stage +Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis, providing valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis. This includes: -Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis. This includes: +* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/) +* The [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_6.html) +* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) -* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/); -* The [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_6.html); -* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/); +## Black-box models -Each provides valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis. +The approver is responsible for signing-off that all risks and ethical considerations around the use of black-box models have been addressed. +This may include: -## Black-box models and the delivery, communication and sign-off stage - - -The Approver is responsible for signing-off that all risks and ethical considerations around the use of black-box models have been addressed. -This may include * formal consultation and approval by an ethics committee or similar * provisions for regular review -* communicating the "health" of the model at regular intervals to the commissioner i.e. is it continuing to behave as expected -The aspects to be considered are detailed in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance). +* communicating the health of the model at regular intervals to the commissioner (for exampe, confirming that the model is continuing to behave as expected) +You can read more in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) +## Multi-use models -## Multi-use models and the delivery, communication and sign-off stage - -There is a greater risk that multi-use models may be used for purposes outside the intended scope. This places a greater onus on the Analyst to clearly communicate to all users the limitations and intended use. The Analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended. +There is a greater risk that multi-use models may be used for purposes outside the intended scope. This places a greater onus on the analyst to clearly communicate to all users the limitations and intended use. The analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended. ## Analytical transparency -Enabling the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register) and [making analysis open](#open-publishing-of-analysis). Further guidance on transparency can be found [here](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/). +Enabling the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register) and [making analysis open](#open-publishing-of-analysis) and [transparency](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/). ### Business-critical analysis register -This section applies to publishing lists of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA), including models. - -1. Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of BCA -in use within their organisations at least annually. -2. Each department and ALB should decide what is defined as business -critical based on the extent to which they influence significant financial and funding -decisions; are necessary to the achievement of a departmental business plan, or -where an error could lead to serious financial, legal or reputational damage. -3. Departments and ALBs should align their definitions and thresholds of business -criticality with their own risk framework respectively. The thresholds should be -agreed by the Director of Analysis or equivalent. -4. ALB’s are responsible for publishing their own BCA list, unless agreed otherwise -with the department. The ALB’s Accounting Officer is accountable for -ensuring publication and the sponsor department’s AO oversees this. -5. The BCA lists should include all business-critical analysis unless there is an -internally documented reason that the analysis should be excluded, agreed with the Director -of Analysis (or equivalent) and the agreement documented. -6. Justification for not publishing a model in the list may include exemptions -under the Freedom of Information (FOI) Act 2000 where relevant, for example, -including, but not limited to: National Security, policy under development or -prejudicing commercial interests. -7. In addition to these exemptions, there may be further reasons where the risk of -negative consequence is deemed to outweigh the potential benefits resulting from -publication of the model. One example is where population behaviour may change -in response to awareness of a model or modelling. -8. For clarity, the name of model/analysis and what it is used for should be included, -alongside links to published material. -9. To ensure the list is accessible, content and structure should follow guidance -for [writing plainly](https://www.gov.uk/guidance/content-design/writing-for-gov-uk) +Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA) in use with their organisations at least annually. This includes models. The list should meet accessibility guidelines. + +Each department and ALB should decide what is defined as business critical based on the extent to which they influence significant financial and funding decisions, are necessary to the achievement of a departmental business plan, or where an error could lead to serious financial, legal or reputational damage. + +The definitions and thresholds of business criticality should be aligned with their own risk framework respectively. The thresholds should be +agreed by the director of analysis or equivalent. + +ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is accountable for ensuring publication and the sponsor department’s accounting officer oversees this. + +The BCA lists should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This shoud be agreed with the director of analysis (or equivalent) with that agreement documented. + +Justification for not publishing a model in the list may include, but is not limited to: + +* exemptions under the Freedom of Information (FOI) Act 2000 +* national security +* policy under development +* commercial interests + +In addition to these exemptions, there may be further reasons where the risk of negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. For example, where population behaviour may change in response to awareness of a model or modelling. + +For clarity, the name of the analysis or model and what it is used for should be included alongside links to any published material. + ### Open publishing of analysis -To facilitate public scrutiny, departments may choose to make analysis/models (e.g. source code or spreadsheets) and details which may include data, assumptions, methodology and outputs open to the public. Open publishing source code and other elements of analysis allows others to reuse and build on the work (https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open). Practical guidance to open coding can be found [here](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable). +To facilitate public scrutiny, departments may choose to make the analysis or model (for example, source code or spreadsheets) and any relevant data, assumptions, methodology and outputs open to the public. [Open publishing source code](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) and other elements of analysis allows others to reuse and build on the work. -Publication of analysis should also draw on the [guidance for BCA lists](#business-critical-analysis-register) related to accessibility and justification for when publication may not be appropriate. For analysis that is extremely complex in nature, it may be more appropriate to publish summary information instead, to aid the accessibility. +You can read more about making [source code open and reusable](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) +The [guidance for publishing BCA lists](#business-critical-analysis-register) should be applied to the publication of analysis as in some cases it might not be appropriate to publish the work. For example, if the analysis is extremely complex it may be more appropriate to publish summary information to make the analysis more accessible. -[^1]: ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, please see [Cabinet Office guidance on Classification of Public Bodies](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf) +[^1]: ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, you can read more in the [Cabinet Office guidance on Classification of Public Bodies](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf) From e85e3c950681fc821d359aa86ab348389c18b89f Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Wed, 18 Dec 2024 11:15:57 +0000 Subject: [PATCH 09/26] Added edits to resources.qmd --- resources.qmd | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/resources.qmd b/resources.qmd index 003a098..032cae7 100644 --- a/resources.qmd +++ b/resources.qmd @@ -8,7 +8,7 @@ The draft currently has no official status. It is a work in progress and is subj ## Written resources -The key additional resources referred to within the AQuA Book are collated here for easy reference. +The key additional resources referred to in the AQuA Book: * The [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) * The [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020) @@ -17,27 +17,27 @@ The key additional resources referred to within the AQuA Book are collated here ### Guidance and advice for performing analysis -* [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) provides support for handling uncertainty in analysis. -* [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) aims to help policy professionals work effectively with statisticians and other analysts. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. +* The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) provides support for handling uncertainty in analysis. +* [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) helps policy professionals work effectively with statisticians and other analysts. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. * The [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020) guides appropriate and responsible data use in government and the wider public sector. It helps public servants understand ethical considerations, address these within their projects, and encourages responsible innovation. * The [Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework) supports the production of sustainable high quality data. * The [Reproducible Analytical Pipelines](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) guidance sets out what a Reproducible Analytical Pipeline is and points to resources for analysts who need to build them. ### Guidance and advice for performing assurance -* The [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf) provides a structured approach to review models which organisations can use to determine whether the modelling outputs they produce are reasonable, robust and have a minimal likelihood of errors being made; -* [Department for Energy Security and Net Zero modelling tools and QA guidance](https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fpublications%2Fenergy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance&data=05%7C02%7Cmodellingintegrity%40energysecurity.gov.uk%7Cc42f8ed850c24245b91f08dc28c9b298%7Ccbac700502c143ebb497e6492d1b2dd8%7C0%7C0%7C638430094188459392%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6pFuTNcJcz5EddPnuIE8CmbU%2BrzUcZvYKdHpdVPkGgk%3D&reserved=0) provides resources to help quality assure new and existing models, including those developed by third parties; -* [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design and assurance of AI models, including risk assessment, bias audits and considering the ongoing 'health' of the model. +* The [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf) provides a structured approach to review models which organisations can use to determine whether the modelling outputs they produce are reasonable, robust and have a minimal likelihood of errors being made. +* [Department for Energy Security and Net Zero modelling tools and QA guidance](https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fpublications%2Fenergy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance&data=05%7C02%7Cmodellingintegrity%40energysecurity.gov.uk%7Cc42f8ed850c24245b91f08dc28c9b298%7Ccbac700502c143ebb497e6492d1b2dd8%7C0%7C0%7C638430094188459392%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6pFuTNcJcz5EddPnuIE8CmbU%2BrzUcZvYKdHpdVPkGgk%3D&reserved=0) provides resources to help quality assure new and existing models, including those developed by third parties. +* [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design and assurance of AI models, including risk assessment, bias audits and considering the ongoing health of the model. * [The Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides guidance on assuring code. ### Guidance and advice for communicating analysis -* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/); -* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/); -* The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that people with disabilities can use them. More specifically, people can: perceive, understand, navigate, and interact with the web. -* If you are publishing statistics you shall follow your organisation's guidance and the [regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/wp-content/uploads/2018/10/Publishing_official_statistics_National_Statistics_1218.pdf); -* If you are publishing research, you shall follow your organisations guidance and the [Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols); -* If you are publishing an evaluation, refer to any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force); +* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/) explores ways of communicating uncertainty in statistics. +* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) explains how to communicate information about quality, uncertainty and change to users. +* The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that they are accessible. +* If you are publishing statistics you shall follow your organisation's guidance and the [regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/wp-content/uploads/2018/10/Publishing_official_statistics_National_Statistics_1218.pdf). +* If you are publishing research, you shall follow your organisations guidance and the [Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols). +* If you are publishing an evaluation, refer to any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force). ## External sources of quality assurance From dd8aad5c3c7fb601e7eb0a4aa2eb87934fde7521 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Wed, 18 Dec 2024 11:44:26 +0000 Subject: [PATCH 10/26] Added edits to intro.qmd --- intro.qmd | 112 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 59 insertions(+), 53 deletions(-) diff --git a/intro.qmd b/intro.qmd index 2ad3462..af74e1a 100644 --- a/intro.qmd +++ b/intro.qmd @@ -1,86 +1,76 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: # Introduction -The AQuA book provides guidance on producing quality analysis for government. It aims to support well informed decision making to create better outcomes and improve the lives of citizens. +The Analytical Quality Assurance (AQuA) Book provides guidance on producing quality analysis for government to support well informed decision-making that creates better outcomes and improve the lives of citizens. -The AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence. +The AQuA Book has made a significant contribution to the cultural change in assurance practices in government by clearly setting out the core framework for assuring all forms of analytical evidence. -The last version of the **Analytical Quality Assurance (AQuA) Handbook** was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, implemenation and operational excellence. +## The updated version (2025) -The world of analysis has developed since we published the first edition of the AQuA Book. +The last version of the AQuA Book was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then assurance has become part of the fabric of good practice for developing evidence to support policy development, implementation and operational excellence. -Increasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence. +The world of analysis has developed since we published the first edition of the AQuA Book. Increasingly, in our data driven world, the insights provided by analysis underpin almost all policies and help to support operational excellence. At the same time, our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. We have now broadened the range of methods we use to include open-source software, machine learning and Artificial Intelligence (AI). -Users of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on: +Based on feedback from our users, for this new edition we have added guidance on: -* multi-use models - large models used for many purposes with many stakeholders; -* assuring "black box"[^1] analysis, including artificial intelligence; -* development, maintenance and continuous review; -* working with third parties such as contractors and academic groups and, -* publishing models. +* multi-use models - large models used for many purposes with many stakeholders +* assuring black box [^1] analysis, including AI +* development, maintenance and continuous review +* working with third parties such as contractors and academic groups +* publishing models -We provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide applicable to all types of analysis. +We provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide relevant to all types of analysis. -The AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”. - -For those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10. - -The AQuA Book is a vital supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). This Standard refers extensively to the AQuA book and notes that "detailed guidance on the analytical cycle and management of analysis, included in the Aqua Book should be followed." +The AQuA Book is a vital supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). This Standard refers extensively to the AQuA Book and notes that the "detailed guidance on the analytical cycle and management of analysis included in the Aqua Book should be followed." It is also referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and other Functional Standards, such as the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. -## Who is the AQuA Book for? +## Who the AQuA Book is for -In this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed. +The new edition is relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose. -We would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes: +We would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision-making in government. The book can help users of analysis make the most of work that has been commissioned and senior leaders with an interest in analytical assurance. The AQuA Book is also for anyone carrying out analysis, including: -* Users of analysis – helping you to get the most out of your commission; -* Those who carry out analysis such as members of the government analytical professions, including: - + operational researchers, statisticians and economists; - + geographers; - + finance professionals; - + actuaries; - + social researchers carrying out qualitative research; - + data scientists developing advanced analytics; - + and anyone else carrying out analysis. -* Senior leaders with an interest in analytical assurance. +* operational researchers, statisticians and economists; +* geographers; +* finance professionals; +* actuaries; +* social researchers carrying out qualitative research; +* data scientists developing advanced analytics; -## Why should I pay attention to this guidance? -Here are a few reasons why. - -* **Your analytical insights will be used for major decisions and operations.** You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage; -* **Trust is hard to obtain but easy to lose.** A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted; -* **Prevention is better than cure.** Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use; -* **Providing quality analysis provides the confidence that is needed for transparency and public openness;** -* **Assurance is required for audit purposes**[^2]; and -* **Professional pride in your work.** ## How to use this book -The first four chapters of this book cover definitions and themes, whilst the second half of the book goes into more detail on the [analytical life cycle](analytical_lifecycle.qmd). This can be pictured as follows: +The AQuA Book has been developed to help the analysis community: + +* publish analytical insights that will be used for major decisions and operations +* minimising the risk of errors arising that cause operational, business or reputational damage +* promote trust in analysts' work +* ensure appropriate quality assurance in place to help to manage mistakes, handle changes to requirements and ensure appropriate re-use +* promote confidence in analysis that is needed for transparency and public openness +* support the analytical assurance that is required for audit purposes [^2] + +The first four chapters of this book cover definitions and themes, while the second half of the book goes into more detail on the [analytical life cycle](analytical_lifecycle.qmd). This can be pictured as follows: ![Figure 1 - The analytical cycle](analytical_lifecycle.jpg){fig-alt="This diagram shows the cycle of activities that make up a typical analytical workflow and shows how these map to the chapters of the AQuA Book. The cycle moves from the start point to engagement and scoping, then design, analysis and finally delivery and communication. After the final stage the cycle either ends at sign off or returns to the engagement and scoping stage. The figure explains that the cycle is often iterative."} Each chapter in the second half of the book is structured as follows: -* Introduction and overview +* an overview of the stage and chapter * Roles and responsibilities * Assurance activities * Documentation * Uncertainty * Black box models * Multi-use models -* Any other elements specific to the stage of the life cycle - -This guidance uses the following terms to indicate whether recommendations are mandatory or advisory. +* any other guidance specific to the stage of the life cycle -The terms are: +Thie AQuA Book uses the following terms to indicate whether recommendations are mandatory or advisory: * **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times * **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis @@ -89,22 +79,38 @@ The terms are: * **‘can’** denotes both capability and possibility * **is/are** is used for a description -These are the same terms as those in the [UK Government Functional Standards](https://www.gov.uk/government/collections/functional-standards). +This is consistent with the [UK Government Functional Standards](https://www.gov.uk/government/collections/functional-standards). ## Principles of analytical quality assurance {.unnumbered} -No single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. However, the following principles support commissioning and production of fit-for-purpose analysis: +No single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. There are some important principles that support that commissioning and production of fit-for-purpose analysis. + +### Proportionate assurance + +Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational effects. + +You can read more about proportionality in chapter [3] + +### Assurance throughout development + +Quality assurance should be considered throughout the whole life cycle of the analysis. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. + +You can read more on the analysis life cycle in chapter [5]. + +### Verification and validation + +Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate and fit for the purpose for which it is being used (validation). -**Proportionate:** Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational effects. More details can be found in chapter [3] +You can read more on verification and validation in chapters [5-9]. -**Assurance throughout development:** Quality assurance should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. More details on the analysis life cycle can be seen in chapter [5]. +### Uncertainty -**Verification and validation:** Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation). Validation and verification are covered in more depth in chapters [5-9]. +It is important to accept that uncertainty is inherent in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty. -**Accept that uncertainty is inherent** in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty . Further support can be found in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io) +You can read more about uncertainty in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io). -**Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. +### Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. ::: {.callout-tip collapse="true"} ### RIGOUR From 71f04a3aa8286db87a9279b214bb16bf7f7b25ad Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 09:25:41 +0000 Subject: [PATCH 11/26] Made small fixes and proofed intro.qmd --- intro.qmd | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/intro.qmd b/intro.qmd index af74e1a..f38e7f0 100644 --- a/intro.qmd +++ b/intro.qmd @@ -13,7 +13,7 @@ The AQuA Book has made a significant contribution to the cultural change in assu The last version of the AQuA Book was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then assurance has become part of the fabric of good practice for developing evidence to support policy development, implementation and operational excellence. -The world of analysis has developed since we published the first edition of the AQuA Book. Increasingly, in our data driven world, the insights provided by analysis underpin almost all policies and help to support operational excellence. At the same time, our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. We have now broadened the range of methods we use to include open-source software, machine learning and Artificial Intelligence (AI). +The world of analysis has developed since we published the first edition of the AQuA Book. Increasingly, in our data driven world, the insights provided by analysis underpin almost all policies and help to support operational excellence. At the same time, our working practices have developed. For example, the dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. We have now broadened the range of methods we use to include open-source software, machine learning and Artificial Intelligence (AI). Based on feedback from our users, for this new edition we have added guidance on: @@ -29,21 +29,18 @@ The AQuA Book is a vital supporting guide for the [Analysis Function Standard](h It is also referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and other Functional Standards, such as the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. - ## Who the AQuA Book is for The new edition is relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose. We would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision-making in government. The book can help users of analysis make the most of work that has been commissioned and senior leaders with an interest in analytical assurance. The AQuA Book is also for anyone carrying out analysis, including: -* operational researchers, statisticians and economists; -* geographers; -* finance professionals; -* actuaries; -* social researchers carrying out qualitative research; -* data scientists developing advanced analytics; - - +* operational researchers, statisticians and economists +* geographers +* finance professionals +* actuaries +* social researchers carrying out qualitative research +* data scientists developing advanced analytics ## How to use this book @@ -51,13 +48,13 @@ The AQuA Book has been developed to help the analysis community: * publish analytical insights that will be used for major decisions and operations * minimising the risk of errors arising that cause operational, business or reputational damage -* promote trust in analysts' work -* ensure appropriate quality assurance in place to help to manage mistakes, handle changes to requirements and ensure appropriate re-use -* promote confidence in analysis that is needed for transparency and public openness +* create greater trust in analysts' work +* ensure appropriate quality assurance is in place to help to manage mistakes, handle changes to requirements and ensure appropriate re-use of analysis +* develop the confidence in analysis that is needed for transparency and public openness * support the analytical assurance that is required for audit purposes [^2] The first four chapters of this book cover definitions and themes, while the second half of the book goes into more detail on the [analytical life cycle](analytical_lifecycle.qmd). This can be pictured as follows: -![Figure 1 - The analytical cycle](analytical_lifecycle.jpg){fig-alt="This diagram shows the cycle of activities that make up a typical analytical workflow and shows how these map to the chapters of the AQuA Book. The cycle moves from the start point to engagement and scoping, then design, analysis and finally delivery and communication. After the final stage the cycle either ends at sign off or returns to the engagement and scoping stage. The figure explains that the cycle is often iterative."} +![Figure 1 - The analytical cycle](analytical_lifecycle.jpg){fig-alt="This diagram shows the cycle of activities that make up a typical analytical workflow and shows how these map to the chapters of the AQuA Book. The cycle moves from the start point to engagement and scoping, then design, analysis and finally to delivery and communication. After the final stage the cycle either ends at sign-off or returns to the engagement and scoping stage. The figure explains that the cycle is often iterative."} Each chapter in the second half of the book is structured as follows: @@ -70,9 +67,9 @@ Each chapter in the second half of the book is structured as follows: * Multi-use models * any other guidance specific to the stage of the life cycle -Thie AQuA Book uses the following terms to indicate whether recommendations are mandatory or advisory: +The AQuA Book uses the following terms to indicate whether recommendations are mandatory or advisory: -* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times +* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances and at all times * **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis * **‘may’** denotes approval * **‘might’** denotes a possibility @@ -81,7 +78,6 @@ Thie AQuA Book uses the following terms to indicate whether recommendations are This is consistent with the [UK Government Functional Standards](https://www.gov.uk/government/collections/functional-standards). - ## Principles of analytical quality assurance {.unnumbered} No single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. There are some important principles that support that commissioning and production of fit-for-purpose analysis. @@ -108,7 +104,7 @@ You can read more on verification and validation in chapters [5-9]. It is important to accept that uncertainty is inherent in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty. -You can read more about uncertainty in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io). +You can read more about uncertainty in the [Uncertainty Toolkit for Analysts in Government](analystsuncertaintytoolkit.github.io). ### Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. From de58b887f3744188edc8aeda13644c2e4eac4ab1 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 09:43:39 +0000 Subject: [PATCH 12/26] Made small fixes and proofed definitions_and_key_concepts.qmd --- definitions_and_key_concepts.qmd | 35 ++++++++++++++++---------------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/definitions_and_key_concepts.qmd b/definitions_and_key_concepts.qmd index 079b276..2a46300 100644 --- a/definitions_and_key_concepts.qmd +++ b/definitions_and_key_concepts.qmd @@ -1,5 +1,5 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: @@ -11,7 +11,6 @@ This chapter sets out definitions and concepts that are used throughout the rest ## Analysis {.unnumbered} Analysis is the collection, manipulation and interpretation of information and data for use in decision-making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. - Examples of types of analysis that are frequently encountered in government are: * actuarial @@ -40,7 +39,7 @@ This may include: ## Artificial Intelligence {.unnumbered} -Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making and language understanding. AI models are a subset of [black box models](#black_box_models) +Artificial Intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making and language understanding. AI models are a subset of [black box models](#black_box_models) ## Black box models {.unnumbered} @@ -50,25 +49,25 @@ The internal workings of black box models are not visible or easily understood. ## Business critical analysis {.unnumbered} -Business critical analysis has significant influence over financial and funding decisions, is necessary to the achievement of a departmental business plan, or is analysis where an error could have a significant reputational, economic or legal implications. +Business critical analysis refers to analysis that has a significant influence over financial and funding decisions, is necessary to the achievement of a departmental business plan or is analysis where an error could have a significant reputational, economic or legal implications. -The first edition of the AQuA book described business critical models. This has been updated to the more general term 'business critical analysis' because analysis can be business critical without including a model. Some departments may continue to use the term business critical models (BCM). +The first edition of the AQuA Book described business critical models. This has been updated to the more general term 'business critical analysis' because analysis can be business critical without including a model. Some departments may still use the term business critical models (BCM). ## Documentation {.unnumbered} ### Specification documentation {.unnumbered} -Specifications capture the initial engagements with the commissioner. They describe the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and capturing what level of assurance is required by the commissioner. +Specification documentation records the initial engagements with the commissioner. It describe the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and define what level of assurance is required by the commissioner. ### Design documentation {.unnumbered} -Design documents describe the analytical plan, including the methodology, inputs and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements. +Design documents describe the analytical plan, including the methodology, inputs and software that will be used. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements. You can read more about design documentation in the [Design](design.qmd) chapter. ### Assumptions log {.unnumbered} -A register of assumptions, whether provided by the commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should: +The assumptions log is a register of assumptions, whether provided by the commissioner or gathered from the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should: * describe each assumption * quantify its effect and reliability @@ -78,7 +77,7 @@ A register of assumptions, whether provided by the commissioner or derived by th ### Decisions log {.unnumbered} -A register of decisions, whether provided by the commissioner or derived by the analysis. Decisions logs should: +The decisions log is a register of decisions, whether provided by the commissioner or derived by the analysis. Decisions logs should: * describe each decision * set out when it was made @@ -90,9 +89,9 @@ A register of decisions, whether provided by the commissioner or derived by the A register of data provided by the commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. -### User/technical documentation {.unnumbered} +### User and technical documentation {.unnumbered} -All analysis shall have user-documentation, even if the only user is the analyst leading the analysis. This documentation should include: * +All analysis shall have user documentation, even if the only user is the analyst leading the analysis. This documentation should include: * a summary of the analysis including the context to the question being asked * what analytical methods were considered @@ -110,15 +109,15 @@ A brief description of the analytical assurance that have been performed to assu ::: {.callout-tip} # Example of publishing quality assurance tools -The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. +The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with the quality assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the departments to ensure analysis meets the standards set out in the AQuA Book and provide assurance to users of the analysis that proportionate quality assurance has been completed. ::: ## Multi-use models {.unnumbered} -Some models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. +Multi-use models are used by more than one user or group of users for related but different purposes. These are often complex and large. -Often, a Steering Group is created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group would ideally have decision-making responsibilites in their area of work. +A Steering Group may be created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group would ideally have decision-making responsibilites in their area of work. ## Quality analysis {.unnumbered} @@ -127,14 +126,14 @@ Quality analysis is fit for the purpose it was commissioned to meet. It should b * accurate * appropriately assured * evidenced -* bproportionate to its effect +* proportionate to its effect * adequately communicated * documented * accepted by its commissioners ## Roles and responsibilities {.unnumbered} -The AQuA book defines the following roles: +The AQuA Book defines the following roles: * commissioner * analyst @@ -160,13 +159,13 @@ The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertainty Validation ensures the analysis meets the needs of its intended users and the intended use environment. -You can read more in [Verification and validation for the AQuA book] (https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. +You can read more in [Verification and validation for the AQuA Book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. ## Verification {.unnumbered} Verification ensures the analysis meets it specified design requirements. -You can read more in [Verification and validation for the AQuA book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. +You can read more in [Verification and Validation for the AQuA Book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover. ## Version control {.unnumbered} From b5eb07e7157b7df8a59cd5cc78f83fca7416bb06 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 09:51:09 +0000 Subject: [PATCH 13/26] Added small fixes and proofed proportionality.qmd --- proportionality.qmd | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/proportionality.qmd b/proportionality.qmd index e2e08c4..bc3a3a4 100644 --- a/proportionality.qmd +++ b/proportionality.qmd @@ -6,20 +6,22 @@ format: --- ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: # Proportionality -All analysis shall be assured. The assurance should be proportionate to the potential effect it will have and the size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks. +All analysis shall be assured. The assurer and the analyst shall be independent. The degree of separation depends on many factors including the importance of the output, and the size and complexity of the analysis. This does not mean that the analyst should not undertake assurance, rather that there shall also be some formal independent assurance. +The assurance should be proportionate to the potential effect it will have and the size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks. + ## Factors for determining appropriate assurance -While there is a need to be confident in the analysis it is not necessary to spend months assuring simple analysis that will have a minor influence on a decision. The level of analysis should be appropriate (proportionate) to the analysis. +While there is a need to be confident in the analysis, it is not necessary to spend months assuring simple analysis that will have a minor influence on a decision. The level of analysis should be appropriate (proportionate) to the analysis. Table 3-1 provides a list of factors that should be considered when determining what level of assurance is appropriate. @@ -29,7 +31,7 @@ Table 3-1 provides a list of factors that should be considered when determining | Relevance of the analysis to the decision-making process | When analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the whole evidence base. | | Type and complexity of analysis | Highly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts. | | Novelty of approach | A previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested. | -| Reusing or repurposing existing work | Reusing work that was carried out previously may require validation and verification to confirm that original approach. For example, the original method and assumptions data are still appropriate for the new requirement. | +| Reusing or repurposing existing work | Reusing work may require validation and verification to confirm that original approach. For example, the original method and assumptions data are still appropriate for the new requirement. | | Level of precision required in outputs | Lower precision analysis often uses simplified assumption, models and data. The assurance approach is the same but will take less time than more precise analysis.| | Amount of resource available for the analysis and assurance | The value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Analysis that is used for many purposes (for example, population projections) may require greater levels of quality assurance than might be suggested by any of the individual decisions they support. | | Longevity of the analysis | Ongoing analysis will require robust change control and regular review. | @@ -44,16 +46,15 @@ Figure 3-1 shows some assurance techniques that might be considered for differen ![Figure 3-1 The darker shades in the diagram indicate the need for extra assurance activities and greater separation between the analyst and the assurer. The contours indicate the groups of activities that may be carried out for a particular level of business risk or complexity.](Figure 3-1 Types of Assurance Alternative 3.jpg){fig-alt="Figure 3-1 is diagram showing the relationship between risk, complexity and the requirement for assurance activity. There are two axes on the diagram. The X axis goes from simple analysis on the left to highly complex analysis on the right. The y axis goes from low business risk at the bottom to high business risk at the top. As risk and complexity increase there is a need for extra assurance activities as well as a higher degree of separation between the analyst and the assurer. For complex, high risk analysis this might include external peer review or audit. On the diagram, the increasing level of risk as we move from bottom left to top right is represented by darker shades."} -The interventions in figure 3-1 must not be viewed in isolation. The interventions should build on each other, for example some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, such as version control and analyst-led testing. +The interventions in figure 3-1 must not be viewed in isolation. The interventions should build on each other. For example, some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, such as version control and analyst-led testing. -The total elimination of risk will never be achievable so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques but there are many different techniques explained in the AQuA book that need to be considered and implemented as appropriate. +The total elimination of risk will never be achieved, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques but there are many different techniques explained in the AQuA Book that need to be considered and implemented as appropriate. ## Structured assessment of business risk and complexity -To determine what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential effect of analytical errors and the likelihood of errors occurring. In situations where the potential effect is high, it is more important that the likelihood of errors is reduced. - -This can be visualised by considering the situation as a risk matrix (Table 3-2). The effect the analysis will have is usually be beyond the control of the analyst to change, so there will be few options to lessen the effect of a risk. However, there will usually be treatments (or mitigations) involving additional assurance measures that will allow the assessed business risk to become less likely to occur. +A structured approach should be taken to determine what assurance is needed when reviewing business risks. Business risk should be viewed as the combination of the potential effect of analytical errors and the likelihood of errors occurring. In situations where the potential effect is high, it is more important that the likelihood of errors is reduced than the level of the effect. +This can be visualised by considering the situation as a risk matrix (table 3-2). The effect the analysis will have is usually be beyond the control of the analyst to change, so there will be few options to lessen the effect of a risk. However, there will usually be treatments (or mitigations) involving additional assurance measures that will allow the assessed business risk to become less likely to occur.
@@ -120,7 +121,7 @@ Table 3-2 - Example of a risk matrix
-Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (Figure 3-1). +Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate mitigations must be considered to reduce the probability of errors occurring. This will depend on the mitigations already in place and on the complexity of the analysis (figure 3-1). For a situation where simple analysis is being employed a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures options such as external peer review may be necessary. @@ -128,14 +129,14 @@ In cases where there is a need for analysis but there are also significant time | Assessed risk | Mitigations to consider | |------------------|------------------------------------------------------| -| High | High risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring. If treatment isn't an option, consideration must be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the senior responsible owner needs to fully understand the risk. | -| Medium | Medium risk should not be tolerated without the agreement of the senior responsible owner. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations. | -| Low | Low risk can be tolerated. Continue with existing or planned mitigations and new treatments may be considered. | +| High | High risk should not be tolerated. New assurance measures must be considered to treat the likelihood of errors occurring. If treatment isn't an option, consideration must be given to terminating or transferring the risk. If it remains necessary to tolerate the risk the senior responsible owner needs to fully understand this risk. | +| Medium | Medium risk should not be tolerated without the agreement of the senior responsible owner. New assurance measures should be put in place to treat the likelihood of errors occurring. Continue with planned or existing mitigations. | +| Low | Low risk can be tolerated. Continue with existing or planned mitigations. New treatments may also be considered. | | Very Low | Very low risk can be tolerated. Continue with existing or planned mitigations measures. | : Table 3-3 Responses to risk assessment levels {.striped} -You can read more on risk management in the [Orange book](https://www.gov.uk/government/publications/orange-book) which covers risk management principles and risk control frameworks. +You can read more on risk management in the [Orange Book](https://www.gov.uk/government/publications/orange-book) which covers risk management principles and risk control frameworks. ## Externally commissioned work From 6bf64e9d3c76790674a20b517fd5acb4c0f6f674 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:02:04 +0000 Subject: [PATCH 14/26] Made small fixes and proofed quality_assurance_culture.qmd --- quality_assurance_culture.qmd | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/quality_assurance_culture.qmd b/quality_assurance_culture.qmd index 1d1a51b..d39fdfd 100644 --- a/quality_assurance_culture.qmd +++ b/quality_assurance_culture.qmd @@ -1,5 +1,5 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: @@ -8,7 +8,7 @@ The draft currently has no official status. It is a work in progress and is subj Creating and maintaining a strong culture of quality assurance is vital to ensuring analysis is robust and of a high quality. -This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance in their teams. However, everyone has a role to play in creating a strong quality assurance culture and the approaches outlined are useful for everyone. +This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance in their teams. However, everyone has a role to play in creating a strong quality assurance culture and the approaches outlined here are useful for everyone. For purposes of this chapter, culture is defined as the shared ways of working, beliefs and habits of an organisation.  @@ -20,7 +20,7 @@ This culture also enables effective [risk management](proportionality.html#struc A quality assurance culture starts with senior leaders. They are accountable for the quality of analysis carried out in their departments. Senior leaders should clearly set out the priority of quality within their teams and create processes for embedding quality assurance.  -Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the accounting officer. In practice the accounting officer may assign the responsibility to a senior leader reporting to the senior management board. They may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero (DESNZ) reports this annually in the [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf). +Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the accounting officer. In practice the accounting officer may assign the responsibility to a senior leader reporting to the senior management board. They may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero (DESNZ) reports this in each [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf). Senior leaders should ensure there is clear messaging and standards on quality assurance through guidance, training and regular updates. Senior leaders can demonstrate the importance of quality assurance through long term initiatives. For example, by setting up and embedding quality assurance processes within the team and creating roles and teams to support quality assurance. Senior leaders should ensure teams have a common understanding of the quality standards required for thier work and regularly talk about quality with their teams, highlighting quality successes. @@ -32,7 +32,7 @@ Senior leaders should also develop processes that enable teams to report when th ::: {.callout-tip} # An open culture when things go wrong -When the Department for Education made an error producing the schools national funding formula allocations for 2024-25, they ran a detailed internal review to understand what went wrong and why it was not detected by the quality assurance process. The department also commissioned and [published an external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its culture of open learning taking responsibility for mistakes. +When the Department for Education made an error producing the schools national funding formula allocations for 2024-25 they ran a detailed internal review to understand what went wrong and why it was not detected by the quality assurance process. The department also commissioned and [published an external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its culture of open learning taking responsibility for mistakes. ::: ## Capacity and capability @@ -41,7 +41,7 @@ Senior leaders should create the conditions in which quality assurance processes ### Capacity -There is a risk that work and time pressures could affect the quality of work. Senior leaders can mitigate this through strong prioritisation and supporting teams to push back on lower value work. Through this prioritisation senior leaders can emphasise the importance of quality. Senior leaders can also support quality assurance by ensuring all parties (analytical and non-analytical) consider it at all stages during the life cycle of a project. +There is a risk that work and time pressures could affect the quality of work. Senior leaders can mitigate this through strong prioritisation and supporting teams to push back on lower value work. Through this prioritisation senior leaders can emphasise the importance of quality. Senior leaders can also support quality assurance by ensuring all parties (both analytical and non-analytical) consider it at all stages during the life cycle of a project. If time constraints mean insufficient assurance has taken place this should be explicitly acknowledged and reported in an assurance statement that sets out the known limitations and conditions associated with the analysis. @@ -55,7 +55,7 @@ If analysis requires a peer review this should be carried out by independent, sk Team leaders can support this by making time for analysts to carry out peer reviews and ensuring analysts are clear that supporting such reviews is part of their role. ::: {.callout-tip} -# HM Revenue and Customs independent review team +# HM Revenue and Customs (HMRC) independent review team HMRC has a small analytical team which independently reviews analysis from across the department, including a sample of HMRC’s business-critical models. The reviews provide assurance for high profile analysis and support the sharing of best practice. ::: @@ -63,7 +63,7 @@ HMRC has a small analytical team which independently reviews analysis from acros There is a risk pf errors occuring because of a lack of skills or experience. Senior leaders can identify common skill or knowledge gaps and provide training or mentoring to help fill these gaps. Processes to support knowledge sharing, innovation and dissemination of best practice will all help develop capability. Rolling out training on departmental assurance processes can also mitigate this risk. -There are various cross-government resources to support and guide commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) aims to help policy professionals to work effectively with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. +There are various cross-government resources to support and guide commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) supports policy professionals working with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. ::: {.callout-tip} # Building assurance capability @@ -71,17 +71,17 @@ DESNZ is building assurance capacity with a programme of quality assurance colle These sessions include an interactive activity looking at a purposefully sabotaged model, which is used to introduce and familiarise colleagues with the quality assurance logbook and the departmental system of actively monitoring models using the logbooks. -College participants also join the Modelling Integrity Network which can help provide assuring analyst capacity in policy areas where lead analysts do not have existing assurance support. +College participants also join the Modelling Integrity Network and help to provide assuring analyst capacity in policy areas where lead analysts do not have existing assurance support. ::: ### Quality assurance champions -In organisations with large analytical community it is good practice for the senior leaders to appoint quality assurance champions who can share best practice in the implementation of quality assurance and provide advice on issues such as proportionality and communication. +In organisations with large analytical communities it is good practice for the senior leaders to appoint quality assurance champions who can share best practice in the implementation of quality assurance and provide advice on issues such as proportionality and communication. ::: {.callout-tip} # Sharing best practice on quality assurance -HM Revenue and Customs have developed a Quality Champions network of analysts across the department. The network discuss quality assurance initiatives, quality issues and how they were resolved, and shares wider best practice. +HMRC have developed a Quality Champions network of analysts across the department. The network discuss quality assurance initiatives, quality issues, how issues are resolved and shares wider best practice. ::: ## Tools @@ -98,7 +98,7 @@ There is a risk that data quality and data understanding cause quality issues wi ## Uncertainty in analysis -Uncertainty is intrinsic to government work. Analysis should support government decision-making by treating of uncertainty approrpiately. Senior leaders are responsible for instilling a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has [created guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated. +Uncertainty is an inevitable part of government work. Analysis should support government decision-making by treating uncertainty approrpiately. Senior leaders are responsible for creating a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has published [guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated. ## Governance and control @@ -106,11 +106,11 @@ Governance supports a strong quality assurance culture by overseeing the effecti Project level governance can provide oversight over a particular model or work area. This will allow the approver to ensure the analysis is fit for purpose. For example, formally agreeing assumptions (which may be recorded in an [assumptions log](definitions_and_key_concepts.html#assumptions-log)) will reduce the need for reworking the analysis providing more time for assurance. Projects governance can also fit within the wider programme level governance.  -Analytical governance boards for new, high-profile or complex pieces of analysis can allow senior analytical leaders and experts to provide oversight and challenge of analysis and ensure best practice is followed. These boards are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work.  +Analytical governance boards for new, high-profile or complex pieces of analysis can allow senior analytical leaders and experts to provide oversight, challenge and ensure best practice is followed. These boards are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work.  ## Transparency -Transparency at all levels can help embed a culture of quality assurance. For example, peer review, sharing lessons learnt and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where possible) can all contribute to an open culture of high quality work. +Transparency at all levels can help embed a culture of quality assurance. For example, peer review, sharing lessons learnt and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where appropriate) can all contribute to an open culture of high quality work. ## Externally commissioned analysis @@ -120,7 +120,7 @@ For Arm’s Length Bodies[^1] (ALBs), the commissioner of analysis is accountabl The guidance mentioned above also applies to ALBs. They may set out in a framework document how their accounting officer will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). -Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). +Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of Managing Public Money and the Analysis Function Standard. When working with third parties the commissioning department shall ensure it is clear which role or roles in which stages of the analytical lifecycle that third party is responsible for. For example, the third party may only undertake the analyst role in the analysis phase or they may undertake the analyst, assurer and approver roles in all stages of the lifecycle. From 4b242d3261614ba08e9270106616b221f4c3608c Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:11:26 +0000 Subject: [PATCH 15/26] Made small fixes and proofed analytical_lifecycle.qmd --- analytical_lifecycle.qmd | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/analytical_lifecycle.qmd b/analytical_lifecycle.qmd index d0a57f0..27f7dbf 100644 --- a/analytical_lifecycle.qmd +++ b/analytical_lifecycle.qmd @@ -1,5 +1,5 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in train and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: @@ -10,7 +10,6 @@ An analytical project can be viewed as a variation on an archetypal project defi ## Roles and responsibilities - Organisations may have their own titles for the main functional roles involved in analysis that are set out here. Each role may be fulfilled by a team or committee of people. However, a single individual (for example, the chair of a committee) will have overall accountability for each role. @@ -67,35 +66,34 @@ It is important that [proportionality](proportionality.qmd) is considered and th Analytical projects typically start with customer engagement although other events may trigger analytical projects. Scoping ensures that an appropriate, common understanding of the problem is defined and that expectations are aligned with what can be produced. During this phase the commissioner plays an important role in communicating the questions to be addressed and working with the analyst to ensure the requirements and scope are defined and understood. -Where analysis requires multiple cycles, for example to develop, use and update analytical models, this phase may follow on from the delivery and communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project. +Where analysis requires multiple cycles (for example to develop, use and update analytical models), this phase may follow on from the delivery and communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project. In this phase more effort may be needed to define the requirements and scope in this phase for research, evaluation or other projects that may need to seek a wider range of perspectives or for which subsequent phases and work may be provided through a product or service. ### Design -During the design phase, the analyst will convert the commission into an analytical plan. This will include the assurance required and ensuring the analysis is sufficient to answer the questions posed. This phase includes the communication and approval of plans produced, and some iteration between the commissioner and the analyst is to be expected as the analytical solution is developed and limitations understood. +During the design phase, the analyst will convert the commission into an analytical plan. This will include the assurance required and ensuring the analysis is sufficient to answer the questions posed. This phase includes the communication and approval of plans. Some iteration between the commissioner and the analyst is to be expected as the analytical solution is developed and limitations understood. For larger projects or those that require multiple cycles, the design phase may include consideration of the staging of work over the whole scope of the project as well as the work required in each stage. Analysis plans for work that is dependent on insights from earlier stages may be high-level and necessitate a return to the design phase at a later date. ### Analysis -The analysis phase is where planned analysis is undertaken, and progress and relevance are monitored. During work, the design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases the changes or limitations encountered may necessitate a return to either the scoping or design phase. +The analysis phase is where planned analysis is undertaken and progress and relevance are monitored. The design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases the changes or limitations encountered may mean the scoping or design phase needes to be revisited. -Throughout this phase traceable documentation of the assurance activities undertaken shall also be produced. +Throughout this phase traceable documentation of the assurance activities that have been undertaken shall also be produced. In larger analytical projects, some outputs of the analysis may be completed at different times as work develops and aspects of other phases may therefore take place concurrently. ### Delivery, communication and sign-off -During the delivery stage, insights and analytical assurances are communicated to the approver and the commissioner, These should be sufficiently understood in order for the approver and commissioner to determine whether the work has been appropriately assured and meets their requirements. Additional analysis and further assurance may be required as analytical projects frequently need further iteration or extension to satisfy the commissioner's needs. +During the delivery stage, insights and analytical assurances are communicated to the approver and the commissioner. These should be sufficiently understood for the approver and commissioner to determine whether the work has been appropriately assured and meets their requirements. Additional analysis and further assurance may be required as analytical projects frequently need further iteration or extension to satisfy the commissioner's needs. Work in this stage can vary considerably depending on the commission, impact, approval processes and the nature of the project. Delivery and communication activities may include producing summary packs and reports, launching dashboards or websites and presentations. After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced. - ## Maintenance and continuous review -The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made as to whether: +The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis all aspects of the lifecycle should be regularly updated. For example, consideration should be made as to whether: * the inputs used remain appropriate * the initial communication methods remain the best way to deliver the information From befe078ac1d379b647249819a53a3068528593ba Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:15:22 +0000 Subject: [PATCH 16/26] Made small fixes and proofed engagement_and_scoping.qmd --- engagement_and_scoping.qmd | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/engagement_and_scoping.qmd b/engagement_and_scoping.qmd index d7270de..ad77bd6 100644 --- a/engagement_and_scoping.qmd +++ b/engagement_and_scoping.qmd @@ -1,5 +1,5 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: @@ -8,7 +8,7 @@ The draft currently has no official status. It is a work in progress and is subj During the first stage of the analytical lifecycle the initial engagement takes place and the commissioner's requirements are scoped out. This stage identifies what is relevant for the analysis. -During this engagement and scoping stage the commissioner and the analyst shape the analysis by developing a shared understanding of the problem and the context. This shared understanding will be used as the basis for designing analysis that suits the commissioner’s requirements. +During this engagement and scoping stage the commissioner and the analyst shape the analysis by developing a shared understanding of the problem and the context. This shared understanding will be used as the basis for designing analysis that meets the commissioner’s requirements. ## Roles and responsibilities @@ -27,7 +27,7 @@ In the engagement and scoping stage the commissioner should: In the engagement and scoping stage the analyst should: -* engage with the Commissioner to identify the question, the context, and the boundaries of the analysis, as well as constraints (for example, deadlines and available resource) assumptions, risks, identified uncertainties and business-criticality. +* engage with the commissioner to identify the question, the context, and the boundaries of the analysis, as well as constraints (for example, deadlines and available resource) assumptions, risks, identified uncertainties and business-criticality * create a specification document which captures the Commissioner's requirements The specification document should provide a definition of the scope and project constraints. It should state the acceptable level of risk and the required level of assurance. It may also state the degree of uncertainty allowed for decision-making and record identified sources of uncertainty. The analyst should share this specification with the commissioner for sign-off. @@ -46,7 +46,7 @@ The approver should ensure that there is sufficient governance in place to suppo If the commissioner is unable to present a well-defined problem, the engagement stage may require the use of problem structuring methods to develop a shared understanding of the requirements. Techniques such as the Strategic Choice Approach, Rich Pictures and Systems Thinking can help the analyst and commissioner to reach a joint understanding of the problem and define the scope of the work. -You can read more about these techniques in the [Systems Thinking Toolkit](https://www.gov.uk/government/publications/systems-thinking-for-civil-servants/toolkit) +You can read more about these techniques in the [Systems Thinking Toolkit](https://www.gov.uk/government/publications/systems-thinking-for-civil-servants/toolkit). If the engagement and scoping techniques are complex or the project is deemed business critical, the assurer might also provide assurance of the engagement methodology (https://publications.tno.nl/publication/100301/Zs2SUz/wijnmalen-2012-natoclient.pdf). @@ -58,7 +58,7 @@ The analyst and commissioner should also clarify risks and potential effects on ## Documentation -The output of the engagement and scoping stage should be a [specification document](definitions_and_key_concepts.html#specification-documentation) that captures the joint understanding of the task between the commissioner and analyst. This document provides a reference for later [validation](definitions_and_key_concepts.html#validation) assurance activities (for example, by confirming that the analysis meets the specification). This document also provides the approver with evidence that the analysis meets the specification during the [delivery stage](delivery_and_communication.html). The document should be signed off by the commissioner and might also be reviewed by the assurer. +The output of the engagement and scoping stage should be a [specification document](definitions_and_key_concepts.html#specification-documentation) that captures the commissioner and analyst's joint understanding of the task. This document provides a reference for later [validation](definitions_and_key_concepts.html#validation) assurance activities (for example, by confirming that the analysis meets the specification). This document also provides the approver with evidence that the analysis meets the specification during the [delivery stage](delivery_and_communication.html). The document should be signed off by the commissioner and might also be reviewed by the assurer. ## Treatment of uncertainty @@ -68,11 +68,11 @@ The engagement and scoping stage will inform the treatment of uncertainty by: * identifying sources of high or intractable uncertainty * establishing an understanding of how the analysis will inform decisions -You can read more about uncertainty in engagement and scoping in the [Uncertainty Toolkit](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_2.html#Jointly_agreeing_how_uncertainty_should_be_used) +You can read more about uncertainty in engagement and scoping in the [Uncertainty Toolkit](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_2.html#Jointly_agreeing_how_uncertainty_should_be_used). ## Black box models -Where the commissioner has engaged with the analyst to deliver [black box models](definitions_and_key_concepts.html#black-box-models) models such as AI or machine learning, the engagement and scoping stage should include discussions around ethics and risks in order to assess whether such models would be appropriate for addressing the given problem. For example, discussions might include considerations of regulations such as UK GDPR, organisational skills, internal governance and risk management. +Where the commissioner has engaged with the analyst to deliver [black box models](definitions_and_key_concepts.html#black-box-models) models such as AI or machine learning, the engagement and scoping stage should include discussions around ethics and risks to assess whether such models would be appropriate for addressing the given problem. For example, discussions might include considerations of regulations such as UK GDPR, organisational skills, internal governance and risk management. You can read more in the [Introdction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance). From b0424c320840da70ea00d47cdf1ad0a565d89a6c Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:25:45 +0000 Subject: [PATCH 17/26] Made small fixes and proofed design.qmd --- design.qmd | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/design.qmd b/design.qmd index abd3a33..bbbdd44 100644 --- a/design.qmd +++ b/design.qmd @@ -1,14 +1,12 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: # Design -## Introduction and overview - -During the design stage the analyst creates an actionable analytical plan from the scope for the analysis agreed with the commissioner. This chapter sets out recommended practices around designing the analysis, deeciding on the associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design and the design of multi-use and Artifical Intelligence (AI) models. +During the design stage the analyst creates an actionable [analytical plan]((#the-analytical-plan) from the scope for the analysis agreed with the commissioner. This chapter sets out recommended practices around designing the analysis, deeciding on the associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design and the design of multi-use and Artifical Intelligence (AI) models. ### The analytical plan @@ -45,9 +43,10 @@ The recommended approach for developing analysis in code is to use a [Reproducib ## Roles and responsibilities in the design stage ### The commissioner's responsibilities + The commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the commissioner may review the analytical plan. -The commissioner's domain expertise can be a useful resource for the analyst in the design stage. The commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs, all of which can inform the design. +The commissioner's expertise can be a useful resource for the analyst in the design stage. The commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs. All of these can inform the design. ### The analyst's responsibilities @@ -62,10 +61,12 @@ The analyst should: * follow any organisation governance procedures for project design ### The assurer's responsibilities -The Aasurer should review the analytical plan to ensure that they are able to conduct the required assurance activities. They may provide feedback on the analytical plan. The Assurer should plan sufficient time for the assurance activity. + +The aasurer should review the analytical plan to ensure that it is able to conduct the required assurance activities. They may provide feedback on the analytical plan. The assurer should plan sufficient time for the assurance activity. ### The approver's responsibilities + In smaller projects, the approver may not be heavily involved in the design stage. However, for business critical analysis, the approver may want to confirm that organisational governance procedures for design have been followed. @@ -79,14 +80,14 @@ The assurance of the design stage should consider whether the analytical plan is * deliver as intended (verification) * be robust (for exmaple, provide a well-structured, data driven plan with a sound overall design) -The assurance of the design stage may be carried out by the assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance, and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases. +The assurance of the design stage may be carried out by the assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases. ## Documentation -The design process should be documented in a proportionate manner. A design document that records the [analytical plan](#the-analytical-plan) should be produced by the analyst and signed-off by the commissioner. The design document may be reviewed by the assurer. +The design process should be documented in a proportionate manner. A design document that records the analytical plan should be produced by the analyst and signed-off by the commissioner. The design document may be reviewed by the assurer. -For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage. +For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage. It is best practice to use formal version control to track changes in the design document. @@ -94,7 +95,7 @@ It is best practice to use formal version control to track changes in the design During the design stage, analysts should examine the planned analysis systematically for possible sources and types of uncertainty. This is to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error. -You can read more in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html) +You can read more in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html). ## Black box models From 3081bb3ae724d870088d38be7ba03ee2cf4210b8 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:31:33 +0000 Subject: [PATCH 18/26] Made small fixes and proofed analysis.qmd --- analysis.qmd | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/analysis.qmd b/analysis.qmd index 39e8090..246d26d 100644 --- a/analysis.qmd +++ b/analysis.qmd @@ -1,19 +1,19 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: # Analysis -During the analysis stage the planned analysis is undertaken and assured, and progress and relevance are monitored. The design may be amended to account for any changing circumstances, emerging information, unexpected difficulties or limitations that may be encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may mean that the design or scoping stage need to be revisited to address these issues. +During the analysis stage, the planned analysis is undertaken and assured, and progress and relevance are monitored. The design may be amended to account for any changing circumstances, emerging information, unexpected difficulties or limitations that may be encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases changes or limitations encountered may mean that the design or scoping stage need to be revisited to address these issues. ## Roles and responsibilities in the analysis stage ### The commissioner's responsibilities -The Commissioner should: +The commissioner should: * be available to provide input and clarifications to the analyst * review any changes in design or methodology that the analyst brings to their attention @@ -40,7 +40,7 @@ The assurer shall: * review the assurance completed by the analyst * carry out any further validation and verification they may see as appropriate * report errors and areas for improvement to the analyst -* review that the work proportionately adheres to [best practice for code development](#assurance-of-code), where relevant. +* review that the work proportionately adheres to [best practice for code development](#assurance-of-code), where relevant The assurer may need to: @@ -64,13 +64,13 @@ Verification that the implemented methodology meets the design requirements shou * dynamic analysis - tests the behaviour of the system, model or code to find errors that arise during execution, includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing)) * symbolic analysis - particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during [the execution of a model](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf), includes path tracing and cause-effect testing * constraint analysis - particularly relevant to modelling and tests the implementation of constraints during model execution, includes checking the assertions of the model and boundary analysis -* formal analysis - tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs. +* formal analysis - tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs -Validation refers to testing whether the product meets the requirements of users. It is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility. +Validation is testing whether the product meets the requirements of users. It is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility. -Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. +Validation of models includes testing the validity of the conceptual model and the operational validity of any computerized model. -You can read more about [techniques that may be useful in validation of models](https://www.informs-sim.org/wsc11papers/016.pdf) +You can read more about [techniques that may be useful in validation of models](https://www.informs-sim.org/wsc11papers/016.pdf). The analyst has primary responsibility for conducting verification and validation. The assurer is responsible for reviewing the verification and validation that is carried out by the analyst, and for conducting or recommending additional verification and validation as required. The assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification. @@ -85,7 +85,7 @@ It is rare to have the perfect dataset for an analytical commission. This could * there are data or coverage gaps * the data may be experimental or there are other reasons why it is not mature -When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. +When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter (or of the parameter of interest under different conditions) that is related to the parameter and conditions of interest. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. The effect of using a proxy dataset should be explored and if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration and the decision to use a particular dataset or input should be recorded for the assurer to verify. @@ -108,29 +108,30 @@ The analyst shall follow the guidance for good quality code development in a pro The analyst should: -* aaintain appropriate records of the work +* maintain appropriate records of the work * fully document any code following agreed standards * log the data, assumptions and inputs used in the analysis, and decisions made in appropriate [documentation](definitions_and_key_concepts.html/#documentation)) * record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis -* produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation). +* produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation) For modelling, the analyst may include a model map that describes data flows and transformations. ## Treatment of uncertainty -While the scoping and design stages identified and described risks and uncertainties, the analysis stage aims to assess and quantify how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty. +While the scoping and design stages identified and described risks and uncertainties, the analysis stage assesses and quantifies how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the analyst and assurer should assure the appropriate treatment of uncertainty. ## Black box models -[Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. +[Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. -Assurance activities during the analysis stage: +Assurance activities of these models during the analysis stage should: -* may include performance testing in a live environment and -* should include the verification steps set out in the design stage -* should include validation and verification of automatic tests to ensure the model behave as expected +* include the verification steps set out in the design stage +* include validation and verification of automatic tests to ensure the model behave as expected -You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) +They may include performance testing in a live environment. + +You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance). ## Multi-use models From 127406b65606bad9f3d2bba43b66085e2a04fc62 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 10:59:02 +0000 Subject: [PATCH 19/26] Made small fixes and proofed delivery_and_communication.qmd --- delivery_and_communication.qmd | 46 ++++++++++++++++------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/delivery_and_communication.qmd b/delivery_and_communication.qmd index d529a20..56f5765 100644 --- a/delivery_and_communication.qmd +++ b/delivery_and_communication.qmd @@ -1,15 +1,14 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: # Delivery, communication and sign-off - The successful delivery of analysis to the commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision-making in your organisation and, possibly, inform the public. -This chapter provides information on the processes around assurance of communication of analysis and delivery of analytical output. +This chapter provides information on the assurance of communication of analysis and delivery of analytical output. ## Roles and responsibilities in delivery, communication and sign-off @@ -39,7 +38,7 @@ The analyst may be required to communicate the assurance state to the approver i ### The assurer's responsibilities -The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may go via the analyst. +The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may be undertaken by the analyst. ### The approver's responsibilities @@ -65,8 +64,8 @@ When delivering a piece of analysis assurer, or analyst, should communicate its * has provisions for regular review * meets the purpose of its commission * has been carried out correctly and to its agreed specification -* has a risk assessment and statement against the programme risk register; -* meets analytical standards, such as those around coding standards and documentation; +* has a risk assessment and statement against the programme risk register +* meets analytical standards, such as those around coding standards and documentation * adheres to any professional codes of practice (for example, [The Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/) * is accompanied by a completed [assurance statement](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fassets.publishing.service.gov.uk%2Fmedia%2F65c5021f9c5b7f0012951b83%2F20231101-analysis-evidence-quality-assurance-report-gov-uk-E02.docx&wdOrigin=BROWSELINK), where appropriate @@ -77,7 +76,7 @@ Though not strictly assurance, the analyst should also consider areas such as: * intellectual property * ethics and related concerns -The approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard, which considers the [proportionality](proportionality.html) of the work. The approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required. +The approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard. The approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required. The exact nature of any scrutiny made by the approver should be proportionate to the effect the analysis is likely to have, the governance process of their programme/ organisation, and follow the [principles of proportionality](proportionality.html). @@ -99,7 +98,7 @@ The Analysis Function's [Making Analytical Publications Accessible Toolkit](http If the outcome of any analysis needs to be published the analyst should follow departmental and statutory guidance. There are different guidelines depending on the work that is being published. These are: -* [The regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/) ; +* [The regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/) * [The Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols) * any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force) for evaluations @@ -107,8 +106,8 @@ If the outcome of any analysis needs to be published the analyst should follow d The exact nature of the approval process may vary depending on the: -* effect the analysis is likely to have; -* approval process of the organisation, and +* effect the analysis is likely to have +* approval process of the organisation * nature of the programme, project or board approving the analysis The formality of the sign-off process should be governed by organisational procedures and be proportionate to the analysis. @@ -139,39 +138,39 @@ Government has produced a range of guidance to support analysts in presenting an * The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/) * The [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_6.html) -* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) +* The Government Analysis Function guidance note on [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) ## Black-box models + +The approver is responsible for the sign-off that confirms that all risks and ethical considerations around the use of black-box models have been addressed. -The approver is responsible for signing-off that all risks and ethical considerations around the use of black-box models have been addressed. This may include: * formal consultation and approval by an ethics committee or similar * provisions for regular review * communicating the health of the model at regular intervals to the commissioner (for exampe, confirming that the model is continuing to behave as expected) -You can read more in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) +You can read more in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance). ## Multi-use models -There is a greater risk that multi-use models may be used for purposes outside the intended scope. This places a greater onus on the analyst to clearly communicate to all users the limitations and intended use. The analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended. +There is a greater risk that multi-use models may be used for purposes outside the intended scope. This means it is important that the analyst very clearly communicates to all users the limitations and intended use. The analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended. ## Analytical transparency -Enabling the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register) and [making analysis open](#open-publishing-of-analysis) and [transparency](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/). +Supporting and encouraging the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register), [making analysis open](#open-publishing-of-analysis) and ensuring [transparency](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/). ### Business-critical analysis register -Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA) in use with their organisations at least annually. This includes models. The list should meet accessibility guidelines. +Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA) in use within their organisations at least annually. This includes models. The list should meet accessibility guidelines. Each department and ALB should decide what is defined as business critical based on the extent to which they influence significant financial and funding decisions, are necessary to the achievement of a departmental business plan, or where an error could lead to serious financial, legal or reputational damage. -The definitions and thresholds of business criticality should be aligned with their own risk framework respectively. The thresholds should be -agreed by the director of analysis or equivalent. +The definitions and thresholds of business criticality should be aligned with their organisation's own risk framework. The thresholds should be agreed by the director of analysis or equivalent. -ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is accountable for ensuring publication and the sponsor department’s accounting officer oversees this. +ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is responsible for ensuring publication and the sponsor department’s accounting officer oversees this. -The BCA lists should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This shoud be agreed with the director of analysis (or equivalent) with that agreement documented. +The BCA list should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This shoud be agreed with the director of analysis (or equivalent) and that agreement should be documented. Justification for not publishing a model in the list may include, but is not limited to: @@ -180,16 +179,15 @@ Justification for not publishing a model in the list may include, but is not lim * policy under development * commercial interests -In addition to these exemptions, there may be further reasons where the risk of negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. For example, where population behaviour may change in response to awareness of a model or modelling. +In addition to these exemptions there may be further reasons where the risk of negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. For example, where population behaviour may change in response to awareness of a model or modelling. For clarity, the name of the analysis or model and what it is used for should be included alongside links to any published material. - ### Open publishing of analysis -To facilitate public scrutiny, departments may choose to make the analysis or model (for example, source code or spreadsheets) and any relevant data, assumptions, methodology and outputs open to the public. [Open publishing source code](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) and other elements of analysis allows others to reuse and build on the work. +To facilitate public scrutiny departments may choose to make the analysis or model (for example, source code or spreadsheets) and any relevant data, assumptions, methodology and outputs open to the public. [Open publishing source code](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) and other parts of the analysis allows others to reuse and build on the work. -You can read more about making [source code open and reusable](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) +You can read more about making [source code open and reusable](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable). The [guidance for publishing BCA lists](#business-critical-analysis-register) should be applied to the publication of analysis as in some cases it might not be appropriate to publish the work. For example, if the analysis is extremely complex it may be more appropriate to publish summary information to make the analysis more accessible. From 00b2be2a323c6dd1b3c8381a40a51024922a2845 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 11:08:03 +0000 Subject: [PATCH 20/26] Made small fixes and proofed resources.qmd --- resources.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/resources.qmd b/resources.qmd index 032cae7..75894c6 100644 --- a/resources.qmd +++ b/resources.qmd @@ -21,14 +21,14 @@ The key additional resources referred to in the AQuA Book: * [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) helps policy professionals work effectively with statisticians and other analysts. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence. * The [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020) guides appropriate and responsible data use in government and the wider public sector. It helps public servants understand ethical considerations, address these within their projects, and encourages responsible innovation. * The [Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework) supports the production of sustainable high quality data. - * The [Reproducible Analytical Pipelines](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) guidance sets out what a Reproducible Analytical Pipeline is and points to resources for analysts who need to build them. +* The [Reproducible Analytical Pipelines](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) guidance sets out what a Reproducible Analytical Pipeline is and points to resources for analysts who need to build them. ### Guidance and advice for performing assurance * The [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf) provides a structured approach to review models which organisations can use to determine whether the modelling outputs they produce are reasonable, robust and have a minimal likelihood of errors being made. * [Department for Energy Security and Net Zero modelling tools and QA guidance](https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fpublications%2Fenergy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance&data=05%7C02%7Cmodellingintegrity%40energysecurity.gov.uk%7Cc42f8ed850c24245b91f08dc28c9b298%7Ccbac700502c143ebb497e6492d1b2dd8%7C0%7C0%7C638430094188459392%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6pFuTNcJcz5EddPnuIE8CmbU%2BrzUcZvYKdHpdVPkGgk%3D&reserved=0) provides resources to help quality assure new and existing models, including those developed by third parties. -* [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design and assurance of AI models, including risk assessment, bias audits and considering the ongoing health of the model. -* [The Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides guidance on assuring code. +* [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design and assurance of AI models, including risk assessment, bias audits and considering the ongoing health of the model. +* The {Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides guidance on assuring code. ### Guidance and advice for communicating analysis From ad5c471d2303294212e12e2480311c3b10653643 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Thu, 19 Dec 2024 11:08:33 +0000 Subject: [PATCH 21/26] Fixed typo in improving_the_book.qmd --- improving_the_book.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/improving_the_book.qmd b/improving_the_book.qmd index c4f3391..045b2c1 100644 --- a/improving_the_book.qmd +++ b/improving_the_book.qmd @@ -1,5 +1,5 @@ ::: {.callout-important} -This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. **At this stage, the site does not fully incorporate requirements from accessibility legislation.** +This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. **At this stage, the site does not fully incorporate requirements from accessibility legislation.** The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. ::: From 833ac280c9c3514d36b5cd7d7e56510ecc2a45c7 Mon Sep 17 00:00:00 2001 From: Hurstharrier <93716598+Hurstharrier@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:07:05 +0000 Subject: [PATCH 22/26] Corrected one further typo --- quality_assurance_culture.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quality_assurance_culture.qmd b/quality_assurance_culture.qmd index d39fdfd..eda1636 100644 --- a/quality_assurance_culture.qmd +++ b/quality_assurance_culture.qmd @@ -90,7 +90,7 @@ There is a risk of technology or analytical tools being out of date. This means * making funding available for new tools or improving existing tools * gathering common issues -* escalating issues to appropriate points in their organisation +* escalating issues to riate points in their organisation ## Data @@ -98,7 +98,7 @@ There is a risk that data quality and data understanding cause quality issues wi ## Uncertainty in analysis -Uncertainty is an inevitable part of government work. Analysis should support government decision-making by treating uncertainty approrpiately. Senior leaders are responsible for creating a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has published [guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated. +Uncertainty is an inevitable part of government work. Analysis should support government decision-making by treating uncertainty appropriately. Senior leaders are responsible for creating a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has published [guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated. ## Governance and control From 060dfe69d4bd810857180f4c40a595f5fad37929 Mon Sep 17 00:00:00 2001 From: Hurstharrier <93716598+Hurstharrier@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:21:54 +0000 Subject: [PATCH 23/26] Corrected one typo --- design.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design.qmd b/design.qmd index bbbdd44..570060f 100644 --- a/design.qmd +++ b/design.qmd @@ -62,7 +62,7 @@ The analyst should: ### The assurer's responsibilities -The aasurer should review the analytical plan to ensure that it is able to conduct the required assurance activities. They may provide feedback on the analytical plan. The assurer should plan sufficient time for the assurance activity. +The assurer should review the analytical plan to ensure that it is able to conduct the required assurance activities. They may provide feedback on the analytical plan. The assurer should plan sufficient time for the assurance activity. ### The approver's responsibilities From 813aed84347b6031809f5ae36511d651255eb7cb Mon Sep 17 00:00:00 2001 From: Hurstharrier <93716598+Hurstharrier@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:36:06 +0000 Subject: [PATCH 24/26] Returned some text to the document Added back the bit on extrapolating parameters Reformatted the paragraph on assurance activities so that the "should" and "may" activities are in the same list. --- analysis.qmd | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/analysis.qmd b/analysis.qmd index 246d26d..0d0654f 100644 --- a/analysis.qmd +++ b/analysis.qmd @@ -85,7 +85,7 @@ It is rare to have the perfect dataset for an analytical commission. This could * there are data or coverage gaps * the data may be experimental or there are other reasons why it is not mature -When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter (or of the parameter of interest under different conditions) that is related to the parameter and conditions of interest. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. +When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter (or of the parameter of interest under different conditions) that is related to the parameter and conditions of interest. This implies extrapolating between parameters, or between conditions for the same parameter. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest. The effect of using a proxy dataset should be explored and if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration and the decision to use a particular dataset or input should be recorded for the assurer to verify. @@ -124,12 +124,11 @@ While the scoping and design stages identified and described risks and uncertain [Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis. -Assurance activities of these models during the analysis stage should: +Assurance activities of these models during the analysis stage: -* include the verification steps set out in the design stage -* include validation and verification of automatic tests to ensure the model behave as expected - -They may include performance testing in a live environment. +* should include the verification steps set out in the design stage +* should include validation and verification of automatic tests to ensure the model behave as expected +* may include performance testing in a live environment You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance). From 5c89eb19a469dd16d32dedbc75985d6b72c07f88 Mon Sep 17 00:00:00 2001 From: Hurstharrier <93716598+Hurstharrier@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:45:34 +0000 Subject: [PATCH 25/26] Changed the AO to be accountable @valentine-scroll - can you review. My change is based on the RACI matrix https://en.wikipedia.org/wiki/Responsibility_assignment_matrix The Accounting officer is accountable i.e. The one ultimately answerable for the correct and thorough completion of the deliverable or task. They usually delegate responsibility to some one else. --- delivery_and_communication.qmd | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/delivery_and_communication.qmd b/delivery_and_communication.qmd index 56f5765..83d73a3 100644 --- a/delivery_and_communication.qmd +++ b/delivery_and_communication.qmd @@ -41,7 +41,7 @@ The analyst may be required to communicate the assurance state to the approver i The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may be undertaken by the analyst. -### The approver's responsibilities +### The approver's bilities The approver shall: @@ -142,7 +142,7 @@ Government has produced a range of guidance to support analysts in presenting an ## Black-box models -The approver is responsible for the sign-off that confirms that all risks and ethical considerations around the use of black-box models have been addressed. +The approver is ble for the sign-off that confirms that all risks and ethical considerations around the use of black-box models have been addressed. This may include: @@ -168,9 +168,9 @@ Each department and ALB should decide what is defined as business critical based The definitions and thresholds of business criticality should be aligned with their organisation's own risk framework. The thresholds should be agreed by the director of analysis or equivalent. -ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is responsible for ensuring publication and the sponsor department’s accounting officer oversees this. +ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is accountable for ensuring publication and the sponsor department’s accounting officer oversees this. -The BCA list should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This shoud be agreed with the director of analysis (or equivalent) and that agreement should be documented. +The BCA list should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This should be agreed with the director of analysis (or equivalent) and that agreement should be documented. Justification for not publishing a model in the list may include, but is not limited to: From 933d5feb51a9d8d3a05d8e085785e2a4a2bd1fc1 Mon Sep 17 00:00:00 2001 From: valentine-scroll Date: Mon, 6 Jan 2025 11:11:09 +0000 Subject: [PATCH 26/26] Fixed typo in delivery_and_communication.qmd --- delivery_and_communication.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/delivery_and_communication.qmd b/delivery_and_communication.qmd index 83d73a3..7af02ff 100644 --- a/delivery_and_communication.qmd +++ b/delivery_and_communication.qmd @@ -41,7 +41,7 @@ The analyst may be required to communicate the assurance state to the approver i The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may be undertaken by the analyst. -### The approver's bilities +### The approver's abilities The approver shall: