Metrics evaluation dataset

[leriomaggio]

This PR introduces in the backend a new EvaluationDataSource created to calculate the metrics for the learning machine.
This special data source holds reference to the original FER validation set, with selected samples (60) used for model prediction.

These samples have been chosen as a result of a data-driven analysis whose thorough documentation is available into a dedicated notebook, which is also part of the current PR (pls see notebooks/FER_AutoEncoder.ipynb)

The PR also adds in a new LearningMachine Model (i.e. VGG13-FER) which is the DL model originally used in the FERPlus dataset.
This model is currently under training - meanwhile the whole infrastructure is available and integrated in the backend for potential future use.

[OliverDavis]

This is brilliant! Working perfectly here. Are we ready to merge?

[leriomaggio]

Not yet.

I moved the PR to draft as I am now finalising several changes I worked on in the last couple of days which I am also willing to include in the PR.

These include:

• a new model for the learning machine (also trained, and several checkpoints saved)
• integration of the new FERPlus dataset as a backend datasource
• (therefore) new AutoEncoder and brand new selection of pool for candidate faces for metric calculation (from FERPlus)

I was just checking the last couple of things, and how the pretrained can be used in case.

Will push the latest soon

[leriomaggio]

Ok @OliverDavis should be ready now for testing, and review.

Please note that I've changed the defaults for the backend in this MR, therefore using the new LearningMachine model, and the new FER+ dataset.
This means that it would take a few seconds the first time it will start, to download the new dataset.
Please keep an eye on the logs in the console for the backend :)

The pretrained option for the model should still be disabled, although fully available at different checkpoints (see learning_machine.py#L335), corresponding to different "starting" grades of learning of the model.

Final performance of this model are also quite decent too, on a full scale training:

Training: ACC: 0.934; 	 MCC: 0.911; 
Validation:  0.802; MCC: 0.732
Test:  ACC: 0.821; MCC: 0.755

Please feel free to let me know if theres's anything that's not clear, or requires fix :)

Once this is merged, we might want to discuss how do we want to save annotations.

TY

[OliverDavis]

I'm getting this when I run the backend:

File "app.py", line 6, in
from endpoints import faces, get_face, annotate, test_face
File "/.../The-Learning-Machine/backend/endpoints.py", line 4, in
from numpy.typing import ArrayLike
ModuleNotFoundError: No module named 'numpy.typing'

Am I missing something from the Conda environment? (Or just missing something :-)

[leriomaggio]

I'm getting this when I run the backend:

File "app.py", line 6, in from endpoints import faces, get_face, annotate, test_face File "/.../The-Learning-Machine/backend/endpoints.py", line 4, in from numpy.typing import ArrayLike ModuleNotFoundError: No module named 'numpy.typing'

Am I missing something from the Conda environment? (Or just missing something :-)

I think this is because you don't probably have the nptyping package installed within your env. (or it could not be found for some reason)

That could be easily fixed with the following command:

pip install nptyping

Please let me know if that worked :)

[OliverDavis]

Thanks! Solved with pip -U numpy, updating numpy to version > 1.20.

[OliverDavis]

I've tried training this for a while, and I'm not sure it's learning anything at the moment: I'm stuck around 0.16 on the graph. Is that random chance with six categories? Very occasionally I get a jump up or down, but it quickly returns to 0.16. The donut plots change, but they remain identical for all the faces (i.e. all of them change in the same way). This might be expected behaviour, if training is very, very slow, although it seemed faster with the previous version.