getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1f14c07e",
   "metadata": {},
   "source": [
    "# Getting Started Examples for the Parangonar Library"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c95d4705",
   "metadata": {},
   "source": [
    "This notebooks gives an overview of parangonars main functionality:\n",
    "- offline alignment\n",
    "- online alignment\n",
    "- visualization and evaluation\n",
    "- file I/O\n",
    "- aligned Data\n",
    "\n",
    "To start we load the contents of a performance and score alignment file (encoded in the [match file format](https://cpjku.github.io/matchfile/)). This file contains a score, a performance, and a ground truth alignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c0bcd6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'\n",
    "%matplotlib inline\n",
    "import parangonar as pa\n",
    "import partitura as pt\n",
    "from pathlib import Path\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19a0acab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the example match file included in the library\n",
    "perf_match, groundtruth_alignment, score_match = pt.load_match(\n",
    "    filename= pa.EXAMPLE, # \n",
    "    create_score=True\n",
    ")\n",
    "\n",
    "# compute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "sna_match = score_match[0].note_array()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68762c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the ground truth alignment\n",
    "pa.plot_alignment(pna_match, sna_match, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc28a4e2",
   "metadata": {},
   "source": [
    "The above alignment shows a performance of Mozart KV 265 var 1. The bottom piano roll is extracted from the score, all notes have quantized lengths and coinciding chord onsets. The top piano roll is extracted from the performance MIDI, note onsets are played expressively and the note offsets are sometimes influenced by pedalling: all notes are held while the sustain pedal is pressed.\n",
    "\n",
    "\n",
    "# Offline Note Matching: \n",
    "\n",
    "Different note matchers in parangonar compute offline alignments:\n",
    "- `AutomaticNoteMatcher`: \n",
    "    piano roll-based, hierarchical DTW and combinatorial optimization for pitch-wise note distribution.\n",
    "    requires scores and performances in the current implementation, but not necessarily.\n",
    "- `DualDTWNoteMatcher`: \n",
    "    symbolic note set-based DTW, pitch-wise onsetDTW, separate handling of ornamentations possible.\n",
    "    requires scores and performances for sequence representation.\n",
    "    **Default and SOTA** for standard score to performance matching.\n",
    "- `TheGlueNoteMatcher`:\n",
    "    pre-trained neural network for note similarity, useful for large mismatches between versions.\n",
    "    works on any two MIDI files.\n",
    "- `AnchorPointNoteMatcher`: \n",
    "    semi-automatic version of the `AutomaticNoteMatcher`, useful if annotations can be leveraged as anchor points. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb71a84f",
   "metadata": {},
   "source": [
    "### `AutomaticNoteMatcher`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b13851d",
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher = pa.AutomaticNoteMatcher()\n",
    "pred_alignment = matcher(sna_match, \n",
    "                        pna_match,\n",
    "                        verbose_time=True)\n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fc8bf1a",
   "metadata": {},
   "source": [
    "### `DualDTWNoteMatcher`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20d3cdec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# recompute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "# because this matcher requires grace note info\n",
    "sna_match = score_match[0].note_array(include_grace_notes=True)\n",
    "matcher = pa.DualDTWNoteMatcher()\n",
    "pred_alignment = matcher(sna_match, \n",
    "                        pna_match,\n",
    "                        process_ornaments=True,\n",
    "                        score_part=score_match[0]) # if a score part is passed, ornaments can be handled seperately\n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c01173",
   "metadata": {},
   "source": [
    "### `TheGlueNoteMatcher`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "122cebaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# recompute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "sna_match = score_match[0].note_array()\n",
    "matcher = pa.TheGlueNoteMatcher()\n",
    "pred_alignment = matcher(sna_match, \n",
    "                         pna_match) \n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081e9280",
   "metadata": {},
   "source": [
    "`TheGlueNoteMatcher` made a mistake. We can plot an alignment comparison against the ground truth to find it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60c911e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pa.plot_alignment_comparison(pna_match, sna_match, pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dccd24a2",
   "metadata": {},
   "source": [
    "### `AnchorPointNoteMatcher` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0524727",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "sna_match = score_match[0].note_array()\n",
    "\n",
    "# compute synthetic anchor points every 4 beats\n",
    "nodes = pa.match.node_array(score_match[0], \n",
    "                   perf_match[0], \n",
    "                   groundtruth_alignment,\n",
    "                   node_interval=4)\n",
    "\n",
    "# match the notes in the note arrays\n",
    "matcher = pa.AnchorPointNoteMatcher()\n",
    "pred_alignment = matcher(sna_match, \n",
    "                        pna_match,\n",
    "                        nodes)\n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "925c30a4",
   "metadata": {},
   "source": [
    "# Online / Real-time Alignment\n",
    "\n",
    "Different note matchers in parangonar compute offline alignments:\n",
    "- `OnlineTransformerMatcher`::\n",
    "    pre-trained neural network for local alignment decisions.\n",
    "    post-processing by a tempo model.\n",
    "- `OnlinePureTransformerMatcher` \n",
    "    pre-trained neural network for local alignment decisions.\n",
    "    no post-processing.\n",
    "    \n",
    "For testing convenience, they all have a `offline` method that loops over all performed notes in a `performance_note_array` and calls the `online` method.\n",
    "\n",
    "\n",
    "### `OnlineTransformerMatcher` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82bbbc51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "# this matcher requires grace note info\n",
    "sna_match = score_match[0].note_array(include_grace_notes=True)\n",
    "\n",
    "matcher = pa.OnlineTransformerMatcher(sna_match)\n",
    "\n",
    "# the \"offline\" method loops over all notes in the performance and calls the \"online\" method for each one.\n",
    "pred_alignment = matcher.offline(pna_match)\n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c22a9c20",
   "metadata": {},
   "source": [
    "### `OnlinePureTransformerMatcher` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abaa836",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute note arrays from the loaded score and performance\n",
    "pna_match = perf_match[0].note_array()\n",
    "# this matcher requires grace note info\n",
    "sna_match = score_match[0].note_array(include_grace_notes=True)\n",
    "\n",
    "matcher = pa.OnlinePureTransformerMatcher(sna_match)\n",
    "\n",
    "# the \"offline\" method loops over all notes in the performance and calls the \"online\" method for each one.\n",
    "pred_alignment = matcher.offline(pna_match)\n",
    "\n",
    "# compute f-score and print the results\n",
    "pa.print_fscore_alignments(pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "160c2878",
   "metadata": {},
   "source": [
    "# Visualize and Evaluate Alignments\n",
    "\n",
    "We have already seen the plotter and printer in action, here they are again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1a66a06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this matcher creates an error for the default file, so we can use for visualization\n",
    "matcher = pa.OnlineTransformerMatcher(sna_match)\n",
    "pred_alignment = matcher.offline(pna_match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "638265ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# show or save plot of note alignment\n",
    "pa.plot_alignment(pna_match,\n",
    "                sna_match,\n",
    "                pred_alignment,\n",
    "                save_file = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cc9e99d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# or plot the performance and score as piano rolls given a reference: \n",
    "# we can encode errors if given ground truth\n",
    "# Blue lines indicate correct matches, red lines incorrect ones.\n",
    "pa.plot_alignment_comparison(pna_match, sna_match, \n",
    "                         pred_alignment, groundtruth_alignment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2fa939f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute precision, recall, and f-score of a type in \"insertion\", \"deletion\", or \"match\"\n",
    "precision, recall, fscore = pa.fscore_alignments(pred_alignment, groundtruth_alignment, types = [\"match\"]) \n",
    "print(fscore)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb2e8302",
   "metadata": {},
   "source": [
    "# File I/O for Note Alignments\n",
    "\n",
    "Most I/O functions are handled by partitura. \n",
    "For [Parangonada](https://sildater.github.io/parangonada/):\n",
    "- pt.io.importparangonada.load_parangonada_alignment\n",
    "- pt.io.importparangonada.load_parangonada_csv\n",
    "- pt.io.exportparangonada.save_parangonada_alignment\n",
    "- pt.io.exportparangonada.save_parangonada_csv\n",
    "\n",
    "For [(n)ASAP alignments](https://github.com/CPJKU/asap-dataset)\n",
    "- pt.io.importparangonada.load_alignment_from_ASAP\n",
    "- pt.io.exportparangonada.save_alignment_for_ASAP\n",
    "\n",
    "For [match files](https://cpjku.github.io/matchfile/)\n",
    "- pt.io.importmatch.load_match\n",
    "- pt.io.exportmatch.save_match\n",
    "\n",
    "and a basic interface for saving parangonada-ready csv files is also available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cf517a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# export a note alignment for visualization with parangonada:\n",
    "# https://sildater.github.io/parangonada/\n",
    "# pa.match.save_parangonada_csv(alignment, performance_data, score_data, outdir=\"path/to/dir\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9351b07b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import a corrected note alignment from parangonada:\n",
    "# https://sildater.github.io/parangonada/\n",
    "# alignment = pt.io.importparangonada.load_parangonada_alignment(filename= 'path/to/note_alignment.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2caca556",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load note alignments of the asap dataset: \n",
    "# https://github.com/CPJKU/asap-dataset\n",
    "# alignment = pt.io.importparangonada.load_alignment_from_ASAP(filename= 'path/to/note_alignment.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ebb13d1",
   "metadata": {},
   "source": [
    "# Aligned Data\n",
    "\n",
    "These note-aligned datasets are publically available:\n",
    "- [Vienna 4x22](https://github.com/CPJKU/vienna4x22)\n",
    "- [(n)ASAP note alignments](https://github.com/CPJKU/asap-dataset)\n",
    "- [Batik Dataset](https://github.com/huispaty/batik_plays_mozart)\n",
    "\n",
    "Here's how you get started with note alignments on the (n)ASAP Dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dab83883",
   "metadata": {},
   "outputs": [],
   "source": [
    "BASE_PATH = Path(\"path/to/nasap-dataset\")\n",
    "EXAMPLE_PATH = Path(\"Bach/Fugue/bwv_846\")\n",
    "PERFORMANCE_NAME = \"Shi05M\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "802b8c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some alignments in (n)ASAP are not great, get a list of non-robust note alignments\n",
    "df = pd.read_csv(Path(BASE_PATH,\"metadata.csv\"))\n",
    "not_robust = df[df[\"robust_note_alignment\"] == 0]\n",
    "not_robust_list = not_robust[\"midi_performance\"].tolist()\n",
    "not_robust_list[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60474b45",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load tsv note alignments of the (n)ASAP dataset: \n",
    "alignment = pt.io.importparangonada.load_alignment_from_ASAP(filename= BASE_PATH / \n",
    "                                                             EXAMPLE_PATH / \n",
    "                                                             Path(PERFORMANCE_NAME + \"_note_alignments/note_alignment.tsv\") )\n",
    "# load scores of the (n)ASAP dataset: \n",
    "score = pt.load_score(filename=  BASE_PATH / \n",
    "                      EXAMPLE_PATH / \n",
    "                      'xml_score.musicxml')\n",
    "# load performance of the (n)ASAP dataset: \n",
    "performance = pt.load_performance_midi(filename=  BASE_PATH / \n",
    "                                       EXAMPLE_PATH / \n",
    "                                       Path(PERFORMANCE_NAME +\".mid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c7c4fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load match note alignments of the (n)ASAP dataset: \n",
    "# performance, alignment, score = pt.load_match(filename= BASE_PATH / \n",
    "#                                              EXAMPLE_PATH / \n",
    "#                                              Path(PERFORMANCE_NAME + \".match\"), create_score = True )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b92bd85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sometimes the scores contain multiple parts which are best merged for easier processing\n",
    "part = pt.score.merge_parts(score)\n",
    "# sometimes scores contain repeats that need to unfolded to make the alignment make sense\n",
    "unfolded_part = pt.score.unfold_part_maximal(part)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d018803",
   "metadata": {},
   "source": [
    "Beware! MusicXML scores need to be unfolded for (n)ASAP, match files are already unfolded!\n",
    "\n",
    "Beware 2! unfolding will change the note IDs: they get a suffix -n for the nth repeat of that note. If the folding state is not clear, it's easy to check the note IDs in the note array and the alignment list for the suffixes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cef2cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to get numpy arrays of the score and performance for downstream processing without partitura:\n",
    "score_array = unfolded_part.note_array()\n",
    "performance_array = performance.note_array()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d271e631",
   "metadata": {},
   "outputs": [],
   "source": [
    "pa.plot_alignment(performance_array, score_array, alignment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "865f39ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# have a look at the score note array\n",
    "score_array[:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15269d77",
   "metadata": {},
   "outputs": [],
   "source": [
    "# have a look at the performance note array\n",
    "performance_array[:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b314c286",
   "metadata": {},
   "outputs": [],
   "source": [
    "# have a look at the alignment list\n",
    "alignment[:8]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}