Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All chapters scroll edits #286

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d9b0cdd
Added edits to definitions_and_key_concepts.qmd
valentine-scroll Dec 16, 2024
2838110
Added edits to proportionality.qmd
valentine-scroll Dec 16, 2024
0748ea4
Added edits to engagement_and_scoping.qmd
valentine-scroll Dec 16, 2024
ce4f61b
Added edits to quality_assurance_culture.qmd
valentine-scroll Dec 17, 2024
c3c150a
Added edits to analytical_lifecycle.qmd
valentine-scroll Dec 17, 2024
a703160
Added edits to design.qmd
valentine-scroll Dec 18, 2024
ab0d5b7
Added edits to analysis.qmd
valentine-scroll Dec 18, 2024
202b195
Added edits to delivery_and_communication.qmd
valentine-scroll Dec 18, 2024
e85e3c9
Added edits to resources.qmd
valentine-scroll Dec 18, 2024
dd8aad5
Added edits to intro.qmd
valentine-scroll Dec 18, 2024
71f04a3
Made small fixes and proofed intro.qmd
valentine-scroll Dec 19, 2024
de58b88
Made small fixes and proofed definitions_and_key_concepts.qmd
valentine-scroll Dec 19, 2024
b5eb07e
Added small fixes and proofed proportionality.qmd
valentine-scroll Dec 19, 2024
6bf64e9
Made small fixes and proofed quality_assurance_culture.qmd
valentine-scroll Dec 19, 2024
4b242d3
Made small fixes and proofed analytical_lifecycle.qmd
valentine-scroll Dec 19, 2024
befe078
Made small fixes and proofed engagement_and_scoping.qmd
valentine-scroll Dec 19, 2024
b0424c3
Made small fixes and proofed design.qmd
valentine-scroll Dec 19, 2024
3081bb3
Made small fixes and proofed analysis.qmd
valentine-scroll Dec 19, 2024
127406b
Made small fixes and proofed delivery_and_communication.qmd
valentine-scroll Dec 19, 2024
00b2be2
Made small fixes and proofed resources.qmd
valentine-scroll Dec 19, 2024
ad5c471
Fixed typo in improving_the_book.qmd
valentine-scroll Dec 19, 2024
833ac28
Corrected one further typo
Hurstharrier Jan 6, 2025
060dfe6
Corrected one typo
Hurstharrier Jan 6, 2025
813aed8
Returned some text to the document
Hurstharrier Jan 6, 2025
5c89eb1
Changed the AO to be accountable
Hurstharrier Jan 6, 2025
933d5fe
Fixed typo in delivery_and_communication.qmd
valentine-scroll Jan 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 64 additions & 29 deletions definitions_and_key_concepts.qmd
rebecca-wodcke-gad marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This chapter sets out definitions and concepts that are used throughout the rest

## Analysis {.unnumbered}

Analysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process.
Analysis is the collection, manipulation and interpretation of information and data for use in decision-making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process.

Examples of types of analysis that are frequently encountered in government are:

Expand All @@ -26,114 +26,149 @@ Examples of types of analysis that are frequently encountered in government are:

## Assurance {.unnumbered}

Analytical assurance is the process and set of practices to ensure that the analysis is fit for purpose.
Analytical assurance is the process and set of practices that ensure analysis is fit for purpose.

## Assurance activities {.unnumbered}

Assurance activities are any actions carried out in order to validate and verify analysis.

For example:
This may include:

* analyst testing
* peer review
* reconciliation of results to independent sources

## Artificial Intelligence {.unnumbered}

Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models)
Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making and language understanding. AI models are a subset of [black box models](#black_box_models)


## Black box models {.unnumbered}

Black box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future.
The internal workings of black box models are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [AI](#Artificial Intelligence) models are the most common type of black box models used today. Other forms of black box models may be developed in the future.


## Business critical analysis {.unnumbered}

Business critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal implications.
Business critical analysis has significant influence over financial and funding decisions, is necessary to the achievement of a departmental business plan, or is analysis where an error could have a significant reputational, economic or legal implications.

The first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM).
The first edition of the AQuA book described business critical models. This has been updated to the more general term 'business critical analysis' because analysis can be business critical without including a model. Some departments may continue to use the term business critical models (BCM).

## Documentation {.unnumbered}

### Specification documentation {.unnumbered}

Specifications capture initial engagements with the commissioner. They describe the question, the context, and any boundaries of the analysis. This provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner.
Specifications capture the initial engagements with the commissioner. They describe the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and capturing what level of assurance is required by the commissioner.

### Design documentation {.unnumbered}

Design documents describe the analytical plan, including the methodology, inputs, and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the Analytical Assurer to verify whether the analysis meets the specified requirements. For more information on the design documentation, see the [Design](design.qmd) chapter.
Design documents describe the analytical plan, including the methodology, inputs and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements.

You can read more about design documentation in the [Design](design.qmd) chapter.

### Assumptions log {.unnumbered}

A register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its effect and reliability and set out when it was made, why it was made, who made it and who signed it off.
A register of assumptions, whether provided by the commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should:

* describe each assumption
* quantify its effect and reliability
* set out when it was made
* explain why it was made
* explain who made the assumption and who signed it off

### Decisions log {.unnumbered}

A register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.
A register of decisions, whether provided by the commissioner or derived by the analysis. Decisions logs should:

* describe each decision
* set out when it was made
* explain why it was made
* explain who made the decision and who signed it off

### Data log {.unnumbered}

A register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder.
A register of data provided by the commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder.


### User/technical documentation {.unnumbered}

All analysis shall have user-documentation, even if the only user is the analyst leading the analysis. This documentation should include: *

### User / technical documentation {.unnumbered}
* a summary of the analysis including the context to the question being asked
* what analytical methods were considered
* what analysis was planned and why
* what challenges were encountered and how they were overcome
* what verification and validation steps were performed

All analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. For modelling, the Analyst may include a model map that describes data flows and transformations.
Where relevant the analyst may include a model map that describes data flows and transformations.

For analysis that is likely to be revisited or updated in the future, more comprehensive documentation should be provided to assist a future analyst. It may also be helpful to include guidance on what should be considered or updated.

### Assurance statement {.unnumbered}

A brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.

::: {.callout-tip}
# Example of publishing quality assurance tools
The Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed.
The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed.

:::

## Multi-use models {.unnumbered}

Some models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**.

Often, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area.
Often, a Steering Group is created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group would ideally have decision-making responsibilites in their area of work.

## Quality analysis {.unnumbered}

Quality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its effect, adequately communicated, documented and accepted by its commissioners.
Quality analysis is fit for the purpose it was commissioned to meet. It should be:

* accurate
* appropriately assured
* evidenced
* bproportionate to its effect
* adequately communicated
* documented
* accepted by its commissioners

## Roles and responsibilities {.unnumbered}

The AQuA book defines the following roles:

* **Commissioner**
* **Analyst**
* **Assurer**
* **Approver**
* commissioner
* analyst
* assurer
* approver

See [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details.
You can read more in the [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) section.

## Third party {.unnumbered}
Any individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company.
Any individual or group of individuals that is not a member of the same group as the those commissioning analysis. For example, they may be working for a different government department, a different function or an outside company.

## Uncertainty {.unnumbered}

Uncertainties are things that are not known, or are in a state of doubt, or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme, piece of analysis meeting its objectives.[^1]
Uncertainties are things that are not known, are in a state of doubt or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme or piece of analysis meeting its objectives.[^1]

[^1]: https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf

There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required.
There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required.

The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.
The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross-government group to help assessing and communicating uncertainty.

## Validation <a name="def_validation"></a>{.unnumbered}

Ensuring the analysis meets the needs of its intended users and the intended use environment. See @glover2014 for more information.
Validation ensures the analysis meets the needs of its intended users and the intended use environment.

You can read more in [Verification and validation for the AQuA book] (https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover.

## Verification {.unnumbered}

Ensuring the analysis meets it specified design requirements. See @glover2014 for more information.
Verification ensures the analysis meets it specified design requirements.

You can read more in [Verification and validation for the AQuA book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover.

## Version control {.unnumbered}

It is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom.
It is important to ensure that the latest version of the analysis is being used and any changes made can be easily seen and quality assured by the analytical assurer. There are tools and templates that can be used to record any updates and checks made during a project. They can help to provide a log of the changes that have been made including why and when they were made, and who made them.

Loading