Skip to content

DFL Train Data Tracker

Jump to bottom
Wyverex-GR6 edited this page Jul 21, 2022 · 14 revisions

The DFL Train Data Tracker is intended to help you track the changes in your DeepFaceLab trained model between various iterations.
You can export a snapshot of your current model, train some more, export a new snapshot, and then use this tool to compare and easily see the changes between different model iterations.


Please note: This feature requires the use of the community DFL fork, and is only available in MVE 0.9.0 or later.

currently this is only available in this branch: https://github.com/MachineEditor/DeepFaceLab/tree/training-state but it should be merged into main soon


Preparing model state history

Before you can make use of the Train Data Tracker, you first need to export model state history by using --gen-snapshot flag.
For example, if using the .bat file for SAEHD training, it could look something like this:
model_state_history_bat
As you can see in the picture, only lines 10 and 11 are different compared to 6) train SAEHD with auto-gen-conf.bat

For more information, visit the article on MVE-DFL fork wiki:
Exporting training data snapshots


Accessing Train Data Tracker

To access this feature go to Tools > DFL Train Data Tracker from the main MVE menu (on top).

You will be presented with a split screen interface:
ts_split_interface

And you start using the feature by loading the model state history (which you prepared following the steps mentioned earlier in this article or via instructions from community DFL wiki article).

You load the model state history by following these 3 simple steps:
ts_model_load

  1. be sure you're on Configuration panel
  2. show MVE where your Model State is on your disk

it is the directory with *_state_history in its name and that has config.json in it, by default it is located within model directory

  1. click Load (and wait)

known issue: progress bar will not update until you left click anywhere within MVE, but that is just a visual issue - loading happens normally with or without the click.


Interface explained

Once your model is loaded, you will be presented with an interface like this:
interface

Immediately it becomes obvious that the interface is split:

  • on the left side you can see the loaded set, and some of the options for selecting how you can see it (options 1-6)
  • on the right side (if option for Set Graphs (marked with number 9 in the picture) is selected) you can see the option for choosing which set is loaded, and two Face Graphs - one for distribution of faces based on their distribution per angle (same as regular Heatmap), and one for distribution of average loss per angle

The six marked options on the left side in the picture are:

  1. select size of loaded faces below
  2. option to define custom image scales
  3. you can use several of the sort options on the faces
  4. dropdown for selecting which snapshot is previewed below
  5. checkmark for disabling timeline
  6. which pictures you want loaded in Timeline (output, swap, both, or none) in addition to aligned face

The three marked options on the right side in the picture are: 7. dropdown menu for selecting which set should be loaded: dst or src 8. Unselect button - to reset your selection if you used any of the Set Graphs to limit set to only pictures that conform to your selected angles 9. the option to display Set Graphs and Set selection in the first place

The Face Graphs function exactly the same like you'd expect if you ever used Face Graph in MVE.

For example, by selecting Yaw-60, Pitch 5 option on the graph, where it says 7 Faces, it will also show 7 faces (plus their model-generated images for current snapshot) on the left side of the screen
example_face_graph


Timeline

When you click on any of the faces in the bottom part of the interface, if Timeline is enabled (option 5 in the Interface Explained picture), you will see the Aligned Face + all the model-generated pictures that are exported in your snapshots. You can also customize which set samples will be shown (output and/or swap)

An example pic:
timeline

Please note that there is a horizontal scroll-bar at the bottom of Timeline, which you can use to see the pictures from all of your snapshots.

Clone this wiki locally