Skip to content

Commit

Permalink
Made small fixes and proofed quality_assurance_culture.qmd
Browse files Browse the repository at this point in the history
  • Loading branch information
valentine-scroll authored Dec 19, 2024
1 parent b5eb07e commit 6bf64e9
Showing 1 changed file with 14 additions and 14 deletions.
28 changes: 14 additions & 14 deletions quality_assurance_culture.qmd
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
::: {.callout-important}
This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
Expand All @@ -8,7 +8,7 @@ The draft currently has no official status. It is a work in progress and is subj

Creating and maintaining a strong culture of quality assurance is vital to ensuring analysis is robust and of a high quality.

This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance in their teams. However, everyone has a role to play in creating a strong quality assurance culture and the approaches outlined are useful for everyone.
This chapter particularly addresses senior leaders and describes their role in developing a strong culture of quality assurance. It outlines processes and approaches to support and embed quality assurance in their teams. However, everyone has a role to play in creating a strong quality assurance culture and the approaches outlined here are useful for everyone.

For purposes of this chapter, culture is defined as the shared ways of working, beliefs and habits of an organisation. 

Expand All @@ -20,7 +20,7 @@ This culture also enables effective [risk management](proportionality.html#struc

A quality assurance culture starts with senior leaders. They are accountable for the quality of analysis carried out in their departments. Senior leaders should clearly set out the priority of quality within their teams and create processes for embedding quality assurance. 

Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the accounting officer. In practice the accounting officer may assign the responsibility to a senior leader reporting to the senior management board. They may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero (DESNZ) reports this annually in the [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf).
Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) assigns accountability for ensuring appropriate assurance processes are in place to the accounting officer. In practice the accounting officer may assign the responsibility to a senior leader reporting to the senior management board. They may collect information on the state of assurance processes and include this in their annual report. The Department of Energy Strategy and Net Zero (DESNZ) reports this in each [DESNZ Annual report](https://assets.publishing.service.gov.uk/media/6532741b26b9b1000faf1ca7/CCS0123681176-001_PN6763756_BEIS_2022-23_Annual_Report_Web_Accessible.pdf).

Senior leaders should ensure there is clear messaging and standards on quality assurance through guidance, training and regular updates. Senior leaders can demonstrate the importance of quality assurance through long term initiatives. For example, by setting up and embedding quality assurance processes within the team and creating roles and teams to support quality assurance. Senior leaders should ensure teams have a common understanding of the quality standards required for thier work and regularly talk about quality with their teams, highlighting quality successes.

Expand All @@ -32,7 +32,7 @@ Senior leaders should also develop processes that enable teams to report when th

::: {.callout-tip}
# An open culture when things go wrong
When the Department for Education made an error producing the schools national funding formula allocations for 2024-25, they ran a detailed internal review to understand what went wrong and why it was not detected by the quality assurance process. The department also commissioned and [published an external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its culture of open learning taking responsibility for mistakes.
When the Department for Education made an error producing the schools national funding formula allocations for 2024-25 they ran a detailed internal review to understand what went wrong and why it was not detected by the quality assurance process. The department also commissioned and [published an external, independent review](https://assets.publishing.service.gov.uk/media/65819c6f23b70a000d234c08/Independent_review_to_assess_the_error_made_in_the_production_of_the_schools_block_NFF.pdf) to assess the error and put forward recommendations. The independent review praised the team for its culture of open learning taking responsibility for mistakes.
:::

## Capacity and capability
Expand All @@ -41,7 +41,7 @@ Senior leaders should create the conditions in which quality assurance processes

### Capacity

There is a risk that work and time pressures could affect the quality of work. Senior leaders can mitigate this through strong prioritisation and supporting teams to push back on lower value work. Through this prioritisation senior leaders can emphasise the importance of quality. Senior leaders can also support quality assurance by ensuring all parties (analytical and non-analytical) consider it at all stages during the life cycle of a project.
There is a risk that work and time pressures could affect the quality of work. Senior leaders can mitigate this through strong prioritisation and supporting teams to push back on lower value work. Through this prioritisation senior leaders can emphasise the importance of quality. Senior leaders can also support quality assurance by ensuring all parties (both analytical and non-analytical) consider it at all stages during the life cycle of a project.

If time constraints mean insufficient assurance has taken place this should be explicitly acknowledged and reported in an assurance statement that sets out the known limitations and conditions associated with the analysis.

Expand All @@ -55,33 +55,33 @@ If analysis requires a peer review this should be carried out by independent, sk
Team leaders can support this by making time for analysts to carry out peer reviews and ensuring analysts are clear that supporting such reviews is part of their role.

::: {.callout-tip}
# HM Revenue and Customs independent review team
# HM Revenue and Customs (HMRC) independent review team
HMRC has a small analytical team which independently reviews analysis from across the department, including a sample of HMRC’s business-critical models. The reviews provide assurance for high profile analysis and support the sharing of best practice.
:::

### Capability

There is a risk pf errors occuring because of a lack of skills or experience. Senior leaders can identify common skill or knowledge gaps and provide training or mentoring to help fill these gaps. Processes to support knowledge sharing, innovation and dissemination of best practice will all help develop capability. Rolling out training on departmental assurance processes can also mitigate this risk.

There are various cross-government resources to support and guide commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) aims to help policy professionals to work effectively with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence.
There are various cross-government resources to support and guide commissioners and users of analysis. For example, the Analysis Function's [Advice for policy professionals using statistics and analysis](https://analysisfunction.civilservice.gov.uk/policy-store/advice-for-policy-professionals-using-statistics/) supports policy professionals working with analysts and analysis. It introduces some important statistical ideas and concepts to help policy professionals ask the right questions when working with statistical evidence.

::: {.callout-tip}
# Building assurance capability
DESNZ is building assurance capacity with a programme of quality assurance colleges. The colleges run regular virtual sessions, open to colleagues across government and partner organisations.

These sessions include an interactive activity looking at a purposefully sabotaged model, which is used to introduce and familiarise colleagues with the quality assurance logbook and the departmental system of actively monitoring models using the logbooks.

College participants also join the Modelling Integrity Network which can help provide assuring analyst capacity in policy areas where lead analysts do not have existing assurance support.
College participants also join the Modelling Integrity Network and help to provide assuring analyst capacity in policy areas where lead analysts do not have existing assurance support.
:::


### Quality assurance champions

In organisations with large analytical community it is good practice for the senior leaders to appoint quality assurance champions who can share best practice in the implementation of quality assurance and provide advice on issues such as proportionality and communication.
In organisations with large analytical communities it is good practice for the senior leaders to appoint quality assurance champions who can share best practice in the implementation of quality assurance and provide advice on issues such as proportionality and communication.

::: {.callout-tip}
# Sharing best practice on quality assurance
HM Revenue and Customs have developed a Quality Champions network of analysts across the department. The network discuss quality assurance initiatives, quality issues and how they were resolved, and shares wider best practice.
HMRC have developed a Quality Champions network of analysts across the department. The network discuss quality assurance initiatives, quality issues, how issues are resolved and shares wider best practice.
:::

## Tools
Expand All @@ -98,19 +98,19 @@ There is a risk that data quality and data understanding cause quality issues wi

## Uncertainty in analysis

Uncertainty is intrinsic to government work. Analysis should support government decision-making by treating of uncertainty approrpiately. Senior leaders are responsible for instilling a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has [created guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated.
Uncertainty is an inevitable part of government work. Analysis should support government decision-making by treating uncertainty approrpiately. Senior leaders are responsible for creating a culture in which the proportionate handling of uncertainty is part of all analytical work. To do so, senior leaders should gain an understanding of how to identify uncertainty, how uncertainty can be analysed and how to plan for uncertainty. The National Audit Office has published [guidance for decision makers and senior leaders](https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf) on managing uncertainty. It is the responsibility of decision makers to challenge analysts, as well as other members of a project team, on whether uncertainties have been considered, treated appropriately and communicated.

## Governance and control

Governance supports a strong quality assurance culture by overseeing the effective management and assurance of analysis. The [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) sets out the requirements for a governance framework for analysis. Each organisation should have a defined and established approach to assurance. This should be applied proportionately to the risk and value of the activity and integrated with the organisation's overall assurance framework. 

Project level governance can provide oversight over a particular model or work area. This will allow the approver to ensure the analysis is fit for purpose. For example, formally agreeing assumptions (which may be recorded in an [assumptions log](definitions_and_key_concepts.html#assumptions-log)) will reduce the need for reworking the analysis providing more time for assurance. Projects governance can also fit within the wider programme level governance. 

Analytical governance boards for new, high-profile or complex pieces of analysis can allow senior analytical leaders and experts to provide oversight and challenge of analysis and ensure best practice is followed. These boards are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work. 
Analytical governance boards for new, high-profile or complex pieces of analysis can allow senior analytical leaders and experts to provide oversight, challenge and ensure best practice is followed. These boards are multi-disciplinary and can cover a range of analytical approaches based on their expertise and experience. This can help ensure that innovations and new approaches are disseminated across teams, and standards are applied equally across similar work. 

## Transparency

Transparency at all levels can help embed a culture of quality assurance. For example, peer review, sharing lessons learnt and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where possible) can all contribute to an open culture of high quality work.
Transparency at all levels can help embed a culture of quality assurance. For example, peer review, sharing lessons learnt and [making analysis open](delivery_and_communication.qmd#open-publishing-of-analysis) (where appropriate) can all contribute to an open culture of high quality work.

## Externally commissioned analysis

Expand All @@ -120,7 +120,7 @@ For Arm’s Length Bodies[^1] (ALBs), the commissioner of analysis is accountabl

The guidance mentioned above also applies to ALBs. They may set out in a framework document how their accounting officer will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2).

Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money) and the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2).
Third parties may set out in a framework document how they will demonstrate compliance with Annex 4.2 of Managing Public Money and the Analysis Function Standard.

When working with third parties the commissioning department shall ensure it is clear which role or roles in which stages of the analytical lifecycle that third party is responsible for. For example, the third party may only undertake the analyst role in the analysis phase or they may undertake the analyst, assurer and approver roles in all stages of the lifecycle.

Expand Down

0 comments on commit 6bf64e9

Please sign in to comment.