Comments from Andrew Duncan from the Turing Centre
A few days late, but I’ve finished going through the book.

General comments:

1. It might help to have a very explicit taxonomy of the models you have in mind. For example, would you consider a simple data-driven statistical model to be an AI/ML model, e.g. something you would write as a Structural Equation Model or a Generalised Linear model? I wouldn’t consider most of these to be black-box, but they do require data.
2. Is the book focused on the commissioning of a static piece of analysis? Do the models developed have any longevity within these organisations, or are they single-use? In the former, then maybe one needs to have a bit more product focused viewpoint, e.g. planning where it is going to run (cloud, local machine, etc) and what resources are needed to enable this.
3. Are all your models built in isolation? Do you combine / compose models? In the latter case, are there any standards / best-practices for interoperability which you might encourage?

Specific Comments:

• Section 2: “Artificial Intelligence models (including Machine Learning) are the most common type of black box models used today. Other forms of black box models may arise in future."

It's not just about the nature of the model, it’s also about the provenance. For example, it’s quite common for an organisation to purchase a piece of commercial modelling software where the inner workings are proprietary and protected IP, in which case it is black box. I would argue this is far more a common existing setting for bb models (unless you’re developing all your models in-house). Additionally, as models grow in complexity, there will come a point where they just have to be considered black-box, simply because the inner workings are too complex and multi-faceted.

• Figure 3-1: Version control is certainly important for maintaining longevity of a software base, but is it really relevant to assurance? I would assume this would be mandated for all software engineering activities.

• Section 6.6: Concerns about ethics, etc would pertain to any data-driven model (not just AI) -- or are we using AI as a catch-all for all data-driven models (including white-box/ grey-box models) – see previous comment.

• Section 7.6: For data-driven black-box models, it’s also important that the Analyst produces an estimate of how long they expect the trained model to remain within validity thresholds (e.g. due to concept drift), and thus come up with a plan for retraining / updating etc. This needs to be accompanied by a plan of how additional / refreshed data will be brought in to refresh the model and what resource (compute) is required.

• Section 8.3.1: For data-driven models, especially black-box, most V&V methods are statistical: i.e. evaluating validity over a sample of the dataset.

@lmadavies @Hurstharrier . The branch that I created for these changes is now a fair bit behind main. So it would be good to create a new branch from main for these edits, to minimise conflicts. Have either of you started to work on these edits on a local branch, or can you wait until we have committed the outstanding PRs (which should hopefully be done tomorrow). It would be best to create the new branch only when you are ready to start work on the edits so that we try keep the new branch as synced up with main as possible

@irisoren-sg I haven't started adding in the comments as I didn't think I had the permissions set up but may do now. Happy to wait until you have made the outstanding PRs before starting on this. You can either start a new branch or rebase the current one, shouldn't make too much difference.

@lmadavies I have deleted the branch and will start a new one once we have merged the big outstanding commits. I would prefer to minimise the chance of getting rebase conflict resolution problems. I'll let you know when the branch is ready

@lmadavies Branch https://github.com/best-practice-and-impact/aqua_book_revision/tree/261-all-chapters-black-box-model-comments is ready for editing

Initial updates to the pages have been made - I would like to do a final review before merging.

I have also tried to address the comments from Turing. The ones I have not included currently are:

• Figure 3-1: Version control is certainly important for maintaining longevity of a software base, but is it really relevant to assurance? I would assume this would be mandated for all software engineering activities.

• Section 6.6: Concerns about ethics, etc would pertain to any data-driven model (not just AI) -- or are we using AI as a catch-all for all data-driven models (including white-box/ grey-box models) – see previous comment.

• Section 8.3.1: For data-driven models, especially black-box, most V&V methods are statistical: i.e. evaluating validity over a sample of the dataset.

@lmadavies I have reviewed the additions. I have made one minor change. On the other points

@valentine-scroll our editor will take a look too as she is reviewing the whole document and it makes sense to include this branch

On the bits you haven't changed.

Version control - strictly speaking he is right but in my experience many of us don't treat models as pieces of software that require maintenance etc. Therefore, I think this is a useful reminder.

ON ethics, etc. It seems to me that ethics concerns are more of an issue with AI - just read the papers! Happy to include a reference to ethics for all analysis but I think we need to also make the point for AI.

On data driven models - I agree but am not sure what to add or amend. Am open to suggestions.

Version control - I think we do want to include version control in the "types of assurance" figure. Andrew may be specifically thinking about git version control which is a core software engineering activity but the book is meant to cover all types of model (including non-coding models) so therefore I think the meaning is broader here and relates to assurance. Personally, I would argue that every piece of analysis has a level of version control (minimum: when it was produced and whether this is a dev, pre-qa or live version) but won't add anything else at this stage as this goes slightly beyond AI/black box.

For the maintenance point, I have added a few points about this already. Agree that it is an important point to remember specifically for AI models.

On ethics - You have a reference to ethics in the Analytical Plan section on the design page. I will add something into section 6.3 Assurance activities in the engagement and scoping stage to mention ethics again.

On data driven models - Agree not sure what to amend or add here.

@lmadavies
VC - I agree with your views but don't think we need to go any further - thanks for adding the maintenance point
Ethics - good idea for 6.3
Data driven models - just leave that be. I think we could tie ourselves in knots if we do much more