impl-NN-from-scratch

Implementation of a Neural Network from scratch in Python for the Machine Learning course.

Authors:

• Diletta Goglia - M.Sc. in Artificial Intelligence, University of Pisa

• Paolo Murgia - M.Sc. in Artificial Intelligence, University of Pisa

Description

Project implementation for Machine Learning exam, Master's Degree Course in Computer Science, Artificial Intelligence curriculum, University of Pisa.

Professor: Alessio Micheli.

For more further info please read the report.

Abstract

The project consists in the implementation of an Artificial Neural Network built from scratch using Python, without using pre-built libraries. The overall validation schema consists in a preliminary screening phase to reduce the hyperparameters search space, followed by a first coarse grid-search and a second but finer one. All the explored models are validated with a 5-fold cross validation. The resulting model is a 2 hidden layer network with 20 units each and ReLU activation for both layers.

Code implementation.

For clarity, transparency and accessibility purposes, we decided to write our code following the ”tacit and explicit conventions applied in Scikit-learn and its API”, and soto follow the notation of the glossary, eg. using standard terms for methods, attributes, etc.
This well-known ”best practice” allowed us to write a good-quality code, well-commented and easy for reading, understanding and experiments reproducibility.

References

Useful sources used & documentation:

• Property vs. Getters and Setters in Python - Datacamp
• Numpy documentation
• Implementing a Neural Network from Scratch in Python - WildML
• sklearn.preprocessing.OneHotEncoder
• tqdm (progress bar)
• Building a Neural Network From Scratch Using Python (Part 1)
• How to build your own Neural Network from scratch in Python
• How to code a neural network from scratch in Python
• Naming with Underscores in Python
• Hyperparameters tuning for ML
• Early stopping for training
• Glorot initialization for weights
• Performance comparison for Monk's problems
• Grid Search in Python from scratch
• itertools — Functions creating iterators for efficient looping¶
• Joblib: running Python functions as pipeline jobs
• The MONK's Problems A Performance Comparison

For parameter tuning:

• How to Tune Hyperparameters for Machine Learning
• Hyper-parameter selection and tuning
• What are Hyperparameters? and How to tune the Hyperparameters
• L1 and L2 Regularization Methods
• A beginner guide to bias and variance in ML

Comparison with pre-built models:

• Scikit MLP Classifier
• Keras Classifier