Lighthouse.jl is a Julia package that standardizes and automates performance evaluation for multiclass, multirater classification models. By implementing a minimal interface, your classifier automagically gains a thoroughly instrumented training/testing harness (Lighthouse.learn!) that computes and logs tons of meaningful performance metrics to TensorBoard in real-time, including:
- test set loss
- inter-rater agreement (e.g. Cohen's Kappa)
- PR curves
- ROC curves
- calibration curves
Lighthouse itself is framework-agnostic; end-users should use whichever extension package matches their desired framework (e.g. https://github.com/beacon-biosignals/LighthouseFlux.jl).
This package follows the YASGuide.
To install Lighthouse for development, run:
julia -e 'using Pkg; Pkg.develop(PackageSpec(url="https://github.com/beacon-biosignals/Lighthouse.jl"))'
This will install Lighthouse to the default package development directory, ~/.julia/dev/Lighthouse.
Note that Lighthouse's LearnLogger logs metrics to a user-specified path in TensorBoard's logdir format. TensorBoard can be installed via python3 -m pip install tensorboard (note: if you have tensorflow>=1.14, you should already have tensorboard). Once TensorBoard is installed, you can view Lighthouse-generated metrics via tensorboard --logdir path where path is the path specified by Lighthouse.LearnLogger. From there, TensorBoard itself can be used/configured however you like; see https://github.com/tensorflow/tensorboard for more information.
You can use alternative loggers, as long as they comply with the logging interface.