improve documentation (#912)

* improve documentation - Create an index page for all documentation pages - Clean up README.md to reference documentation overview, from where people can find all docs - Harmonize Markdown headings in all individual pages - Add dedicated page for core metrics - Add NEG to model page - Harmonize naming of Markdown documents - Move PDFs to separate directories - Fix a few spelling errors, broken HTTP(S) links etc. * Update documentation. Co-authored-by: Zhi Li <[email protected]>
Netflix · Oct 16, 2021 · 400fe79 · 400fe79
1 parent c8f54f5
commit 400fe79
Show file tree

Hide file tree

Showing 20 changed files with 144 additions and 82 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,5 +1,7 @@
 # Contributing to VMAF
 
+Please refer to this [slide deck](https://docs.google.com/presentation/d/1Gr4-MvOXu9HUiH4nnqLGWupJYMeh6nl2MNz6Qy9153c/edit#slide=id.gc20398b4b7_0_132) for an overview contribution guide.
+
 If you would like to contribute code to the VMAF repository, you can do so through GitHub by forking the repository and sending a pull request. When submitting code, please make every effort to follow existing conventions and style in order to keep the code as readable as possible.
 
 ## License
@@ -49,7 +51,7 @@ For both cases, one can follow the procedure below:
     - creating a new `QualityRunner` class as a thin wrapper around the new `FeatureExtractor` created, or
     - using the established `VmafQualityRunner` class but training a custom VMAF model.
 
-For the concepts of `FeatureExtractor`, `QualityRunner` and `VmafQualityRunner`, please refer to the [Core Classes](resource/doc/VMAF_Python_library.md#core-classes) section of the VMAF Python library documentation.
+For the concepts of `FeatureExtractor`, `QualityRunner` and `VmafQualityRunner`, please refer to the [Core Classes](resource/doc/python.md#core-classes) section of the VMAF Python library documentation.
 
 For algorithmic contribution, for a clean organization of the repo, it is advised to submit new files under directory prefixed with `third_party/[orginization]`. For example, for a new model trained, it should go under `model/third_party/[organization]/`. As another example, the [PSNR-HVS feature extractor](https://github.com/Netflix/vmaf/commit/ce2ad1af0b1ba8dd1fbae3e03da0329f078e6bc6) code sits under `libvmaf/src/feature/third_party/xiph/`.
 
@@ -118,6 +120,6 @@ The default VMAF model has been using the `LibsvmNusvrTrainTestModel` class for
   - Optionally override housekeeping functions such as `_to_file()`, `_delete()`, `_from_info_loaded()` when needed.
 
 #### Calling the `run_vmaf_training` Script
-Once the `FeatureExtractor` and `TrainTestModel` classes are ready, the actually training of a VMAF model against a dataset of subjective scores can be initiated by calling the `run_vmaf_training` script. Detailed description of how to use the script can be found in the [Train a New Model](resource/doc/VMAF_Python_library.md#train-a-new-model) section of VMAF Python Library documentation.
+Once the `FeatureExtractor` and `TrainTestModel` classes are ready, the actually training of a VMAF model against a dataset of subjective scores can be initiated by calling the `run_vmaf_training` script. Detailed description of how to use the script can be found in the [Train a New Model](resource/doc/python.md#train-a-new-model) section of VMAF Python Library documentation.
 
 Notice that the current `run_vmaf_training` implementation does not work with `FeatureExtractors` with a custom input parameter (e.g. the `max_db` of `PypsnrFeatureExtractor`). A workaround of this limitation is to create a subclass of the feature extractor with the hard-coded parameter (by overriding the `_custom_init()` method). Refer to [this code](https://github.com/Netflix/vmaf/commit/e698b4d788fb3dcabdc4df2fd1bffe88dc0d3ecd#diff-c5c651eeebd2949a31e059575be35f2018ec57ebdc09c0706cfc1187b9b32dbcR536) and [this test](https://github.com/Netflix/vmaf/commit/e698b4d788fb3dcabdc4df2fd1bffe88dc0d3ecd#diff-5b58c2457df7e9b30b0a678d6f79b1caaad6c3f036edfadb6ca9fb0955bede33R751) for an example.
diff --git a/README.md b/README.md
@@ -6,63 +6,38 @@
 [![ffmpeg](https://github.com/Netflix/vmaf/workflows/ffmpeg/badge.svg)](https://github.com/Netflix/vmaf/actions?query=workflow%3Affmpeg)
 [![Docker](https://github.com/Netflix/vmaf/workflows/Docker/badge.svg)](https://github.com/Netflix/vmaf/actions?query=workflow%3ADocker)
 
-VMAF is a perceptual video quality assessment algorithm developed by Netflix. This software package includes a stand-alone C library `libvmaf` and its wrapping Python library. The Python library also provides a set of tools that allows a user to train and test a custom VMAF model. Read [this](https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652) tech blog post for an overview, [this](https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12) post for the tips of best practices, and [this](https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30) post for our latest efforts on speed optimization, new API design and the introduction of a codec evaluation-friendly NEG mode.
+VMAF is a perceptual video quality assessment algorithm developed by Netflix. This software package includes a stand-alone C library `libvmaf` and its wrapping Python library. The Python library also provides a set of tools that allows a user to train and test a custom VMAF model.
+
+Read [this](https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652) tech blog post for an overview, [this](https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12) post for the tips of best practices, and [this](https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30) post for our latest efforts on speed optimization, new API design and the introduction of a codec evaluation-friendly NEG mode.
 
 Also included in `libvmaf` are implementations of several other metrics: PSNR, PSNR-HVS, SSIM, MS-SSIM and CIEDE2000.
 
 ![vmaf logo](resource/images/vmaf_logo.jpg)
 
 ## News
 
-- (2021-10-7) We are open-sourcing CAMBI (Contrast Aware Multiscale Banding Index) - Netflix's detector for banding (aka contouring) artifacts. Check out the [tech blog](https://netflixtechblog.medium.com/cambi-a-banding-artifact-detector-96777ae12fe2) for an overview and the [technical paper](resource/doc/CAMBI_PCS2021.pdf) published in PCS 2021 (note that the paper describes an initial version of CAMBI that no longer matches the code exactly, but it is still a good introduction). Also check out the [usage](resource/doc/cambi.md) page.
+- (2021-10-7) We are open-sourcing CAMBI (Contrast Aware Multiscale Banding Index) - Netflix's detector for banding (aka contouring) artifacts. Check out the [tech blog](https://netflixtechblog.medium.com/cambi-a-banding-artifact-detector-96777ae12fe2) for an overview and the [technical paper](resource/doc/papers/CAMBI_PCS2021.pdf) published in PCS 2021 (note that the paper describes an initial version of CAMBI that no longer matches the code exactly, but it is still a good introduction). Also check out the [usage](resource/doc/cambi.md) page.
 - (2020-12-7) Check out our [latest tech blog](https://netflixtechblog.com/toward-a-better-quality-metric-for-the-video-community-7ed94e752a30) on speed optimization, new API design and the introduction of a codec evaluation-friendly NEG mode.
 - (2020-12-3) We are releasing `libvmaf v2.0.0`. It has a new fixed-point and x86 SIMD-optimized (AVX2, AVX-512) implementation that achieves 2x speed up compared to the previous floating-point version. It also has a [new API](libvmaf/README.md) that is more flexible and extensible.
 - (2020-7-13) We have created a [memo](https://docs.google.com/document/d/1dJczEhXO0MZjBSNyKmd3ARiCTdFVMNPBykH4_HMPoyY/edit?usp=sharing) to share our thoughts on VMAF's property in the presence of image enhancement operations, its impact on codec evaluation, and our solutions.
 - (2020-2-27) We have changed VMAF's license from Apache 2.0 to [BSD+Patent](https://opensource.org/licenses/BSDplusPatent), a more permissive license compared to Apache that also includes an express patent grant.
 
-## Frequently Asked Questions
+## Documentation
 
-Refer to the [FAQ](FAQ.md) page.
+There is an [overview of the documentation](resource/doc/index.md) with links to specific pages, covering FAQs, available models and metrics, software usage guides, and a list of resources.
 
-## Usages
+## Usage
 
 The software package offers a number of ways to interact with the VMAF implementation.
 
   - The command-line tool [`vmaf`](libvmaf/tools/README.md) provides a complete algorithm implementation, such that one can easily deploy VMAF in a production environment. Additionally, the `vmaf` tool provides a number of auxillary metrics such as PSNR, SSIM and MS-SSIM.
   - The [C library `libvmaf`](libvmaf/README.md) provides an interface to incorporate VMAF into your code, and tools to integrate other feature extractors into the library.
-  - The [Python library](resource/doc/VMAF_Python_library.md) offers a full array of wrapper classes and scripts for software testing, VMAF model training and validation, dataset processing, data visualization, etc.
+  - The [Python library](resource/doc/python.md) offers a full array of wrapper classes and scripts for software testing, VMAF model training and validation, dataset processing, data visualization, etc.
   - VMAF is now included as a filter in FFmpeg, and can be configured using: `./configure --enable-libvmaf`. Refer to the [Using VMAF with FFmpeg](resource/doc/ffmpeg.md) page.
-  - [VMAF Dockerfile](Dockerfile) generates a docker image from the [Python library](resource/doc/VMAF_Python_library.md). Refer to [this](resource/doc/docker.md) document for detailed usages.
-  - To build VMAF on Windows, follow [this](resource/doc/BuildForWindows.md) instruction.
-
-## Datasets
-
-We also provide [two sample datasets](resource/doc/datasets.md) including the video files and the properly formatted dataset files in Python. They can be used as sample datasets to train and test custom VMAF models.
-
-## Models
-
-Besides the default VMAF model which predicts the quality of videos displayed on a 1080p HDTV in a living-room-like environment, VDK also includes a number of additional models, covering phone and 4KTV viewing conditions. Refer to the [models](resource/doc/models.md) page for more details.
-
-## Confidence Interval
-
-Since June 2018, we have introduced a way to quantify the level of confidence that a VMAF prediction entails. Each VMAF prediction score now can come with a 95% confidence interval, which quantifies the level of confidence of the prediction. Refer to the [VMAF Confidence Interval](resource/doc/conf_interval.md) page for more details.
-
-## Matlab Functionality
-
-Besides the C and Python part of the software package, we also introduced a number of algorithms that are implemented in Matlab. For example, users can calculate ST-RRED, ST-MAD, SpEED-QA, and BRISQUE. For more details, see the [Matlab Usage](resource/doc/matlab_usage.md) page for more details.
-
-## Report of VMAF Bad Cases
-
-Over time, we have received feedbacks on when VMAF's prediction does not reflect the expected perceptual quality of videos, either they are corner cases where VMAF fails to cover, or new application scenarios which VMAF was not initially intended for. In response to that, we have created a Google form to allow users to upload their video samples and describe their cases. The bad cases are valuable for improving future versions of VMAF. Users can opt in or out for sharing their sample videos publicly. Please submit the bad cases through [this link](https://docs.google.com/forms/d/e/1FAIpQLSdJntNoBuucMSiYoK3SDWoY1QN0yiFAi5LyEXuOyXEWJbQBtQ/viewform?usp=sf_link).
+  - [VMAF Dockerfile](Dockerfile) generates a docker image from the [Python library](resource/doc/python.md). Refer to [this](resource/doc/docker.md) document for detailed usage.
+  - To build VMAF on Windows, follow [these](resource/doc/windows.md) instructions.
+  - AOM CTC: [AOM]((http://aomedia.org/)) has specified vmaf to be the standard implementation metrics tool according to the AOM common test conditions (CTC). Refer to [this page](resource/doc/aom_ctc.md) for usage compliant with AOM CTC.
 
 ## Contribution Guide
 
-Refer to the [contribution](CONTRIBUTING.md) page.
-
-## References
-
-Refer to the [references](resource/doc/references.md) page.
-
-## External Resources
-
-Refer to [this page](resource/doc/external_resource.md) for a list of 3rd-party tools built on top of VMAF.
+Refer to the [contribution](CONTRIBUTING.md) page. Also refer to this [slide deck](https://docs.google.com/presentation/d/1Gr4-MvOXu9HUiH4nnqLGWupJYMeh6nl2MNz6Qy9153c/edit#slide=id.gc20398b4b7_0_132) for an overview contribution guide.
diff --git a/resource/doc/aom_ctc.md b/resource/doc/aom_ctc.md
@@ -1,9 +1,8 @@
-AOM CTC
-=======
+# AOM CTC
 
 [AOM](http://aomedia.org/) has specified  [`vmaf`](../../libvmaf/tools/README.md) to be the standard implementation metrics tool according to the AOM common test conditions (CTC).
 
-# Metrics
+## Metrics
 Using the versioned `--aom_ctc` preset, the following metrics will be computed and logged. These metrics are specified according to the AOM CTC.
 
 * PSNR
@@ -15,7 +14,7 @@ Using the versioned `--aom_ctc` preset, the following metrics will be computed a
 * VMAF
 * VMAF NEG
 
-# Usage
+## Usage
 Basic usage of the tool is described in the [`vmaf` README](../../libvmaf/tools/README.md). Use the versioned `--aom_ctc` presets to register and configure all metrics according to the AOM CTC. Basic AOM CTC usage is as follows:
 
 ```
@@ -30,10 +29,10 @@ There are also a few optional command-line settings you may find useful.
 * Use `--threads` to set the thread count to be used for multi-threaded computation. This will decrease the overall latency.
 * If you prefer a JSON log over the default XML log, use the `--json` flag.
 
-# Output
+## Output
 `XML` and `JSON` logging formats provide per-frame metrics, pooled metrics, and aggregate metrics.
 
-# AOM CTC Version History
+## AOM CTC Version History
 * v1.0: `--aom_ctc v1.0`
   * 2021-01-13
   * Fix for lossless comparisons, dB clipping for PSNR/APSNR/SSIM/MS-SSIM according to the AOM CTC.

diff --git a/resource/doc/bad_cases.md b/resource/doc/bad_cases.md
@@ -0,0 +1,5 @@
+# Reporting Bad Cases
+
+Over time, we have received feedbacks on when VMAF's prediction does not reflect the expected perceptual quality of videos, either they are corner cases where VMAF fails to cover, or new application scenarios which VMAF was not initially intended for. In response to that, we have created a Google form to allow users to upload their video samples and describe their cases. The bad cases are valuable for improving future versions of VMAF. Users can opt in or out for sharing their sample videos publicly.
+
+Please submit the bad cases through [this link](https://docs.google.com/forms/d/e/1FAIpQLSdJntNoBuucMSiYoK3SDWoY1QN0yiFAi5LyEXuOyXEWJbQBtQ/viewform?usp=sf_link).
diff --git a/resource/doc/cambi.md b/resource/doc/cambi.md
@@ -2,12 +2,18 @@
 
 CAMBI (Contrast Aware Multiscale Banding Index) is Netflix's detector for banding (aka contouring) artifacts.
 
-For an introduction to CAMBI, please refer to this [tech blog](https://netflixtechblog.medium.com/cambi-a-banding-artifact-detector-96777ae12fe2). For a detailed technical description, please refer to the [technical paper](CAMBI_PCS2021.pdf) published at PCS 2021. Note that the paper describes an initial version of CAMBI that no longer matches the code exactly, but it is still a good introduction.
+## Background
+
+For an introduction to CAMBI, please refer to the [tech blog](https://netflixtechblog.medium.com/cambi-a-banding-artifact-detector-96777ae12fe2). For a detailed technical description, please refer to the [technical paper](papers/CAMBI_PCS2021.pdf) published at PCS 2021. Note that the paper describes an initial version of CAMBI that no longer matches the code exactly, but it is still a good introduction.
 
 The current version of CAMBI is a [no-reference metric](https://en.wikipedia.org/wiki/Video_quality#Classification_of_objective_video_quality_models), and operates on a frame-by-frame basis (no temporal information is leveraged). To integrate it as part of the VMAF framework, which employs a [full-reference metric](https://en.wikipedia.org/wiki/Video_quality#Classification_of_objective_video_quality_models) API, CAMBI takes both a reference and a distorted video as its input. For simplicity, one can point the input arguments `--reference` and `--distorted` to the same video path.
 
+## Scores
+
 The CAMBI score starts at 0, meaning no banding is detected. A higher CAMBI score means more visible banding artifacts are identified. The maximum CAMBI observed in a sequence is 24 (unwatchable). As a rule of thumb, a CAMBI score around 5 is where banding starts to become slightly annoying (also note that banding is highly dependent on the viewing environment - the brigher the display, and the dimmer the ambient light, the more visible banding is).
 
+## How to run CAMBI
+
 To invoke CAMBI using the VMAF command line, follow the [instruction](../../libvmaf/tools/README.md) and use `cambi` as the feature name. For example, after downloading the input video [`src01_hrc01_576x324.yuv`](https://github.com/Netflix/vmaf_resource/blob/master/python/test/resource/yuv/src01_hrc01_576x324.yuv), invoke CAMBI via:
 
 ```shell script
@@ -37,6 +43,8 @@ This will yield the output:
 </VMAF>
 ```
 
+## Options
+
 The CAMBI feature extractor also supports additional optional parameters as listed below:
 
 - `window_size` (min: 15, max: 127, default: 63): Window size to compute CAMBI (default: 63 corresponds to ~1 degree at 4K resolution and 1.5H)
@@ -69,7 +77,7 @@ This will yield the output:
 </VMAF>
 ```
 
-If, no encoding width and height parameters are specified:
+If no encoding width and height parameters are specified:
 
 ```shell script
 libvmaf/build/tools/vmaf \
@@ -94,4 +102,6 @@ The output will be:
 </VMAF>
 ```
 
-CAMBI can also be invoked in the [Python library](VMAF_Python_library.md). Use `CambiFeatureExtractor` as the feature exatractor, and `CambiQualityRunner` as the quality runner.
+## Python Library
+
+CAMBI can also be invoked in the [Python library](python.md). Use `CambiFeatureExtractor` as the feature exatractor, and `CambiQualityRunner` as the quality runner.