Skip to content

DFL Train Data Tracker

Jump to bottom
Wyverex-GR6 edited this page Jul 21, 2022 · 14 revisions

The DFL Train Data Tracker is intended to help you track the changes in your DeepFaceLab trained model between various iterations.
You can export a snapshot of your current model, train some more, export a new snapshot, and then use this tool to compare and easily see the changes between different model iterations.


Please note:
This feature requires the use of the MVE-DFL fork, and is only available in MVE 0.9.0 or later.


Preparing model state history

Before you can make use of the Train Data Tracker, you first need to export model state history by using --gen-snapshot flag.
For example, you can edit the .bat file for SAEHD training to look like the picture below:
model_state_history_bat
As you can see in the picture, only lines 10 and 11 are different compared to 6) train SAEHD with auto-gen-conf.bat

For more information, visit the article on MVE-DFL fork wiki:
Exporting training data snapshots


Accessing Train Data Tracker

To access this feature go to Tools > DFL Train Data Tracker from the main MVE menu (on top).

You will be presented with, what looks like a split screen interface (but it's just open Side Panel):
ts_split_interface

And you start using the feature by loading the model state history (which you prepared following the steps mentioned earlier in this article or via instructions from community DFL wiki article).

You load the model state history by following these 3 simple steps:
ts_model_load

  1. be sure you're on Configuration panel
  2. show MVE where your Model State is on your disk

It is the directory with *_state_history in its name, and that has config.json in it. By default it is located within model directory

  1. click Load (and wait)

known issue: progress bar will not update until you left click anywhere within MVE, but that is just a visual issue - loading happens normally with or without the click.


Interface explained

Once your model is loaded, you will be presented with an interface like in the picture below:
interface

Initially it looks like the interface is split:

  • on the left side you can see the loaded set, and some of the options for selecting how you can see it (options 1-6)
  • on the right side (if option for Set Graphs (marked with number 9 in the picture) is selected) you can see the option for choosing which set is loaded, and two Face Graphs - one for distribution of faces based on their distribution per angle (same as regular Heatmap), and one for distribution of average loss per angle

But in reality, left side is the main side, and the right side is just side panel while it is open. You can close the side panel if you don't need it.

The marked options in the picture are:

  1. select size of loaded faces below
  2. option to define custom image scales
  3. you can use several of the sort options on the faces
  4. dropdown for selecting which snapshot is previewed below
  5. checkmark for disabling timeline
  6. which pictures you want loaded in Timeline (output, swap, both, or none) in addition to aligned face
  7. dropdown menu for selecting which set should be loaded: dst or src
  8. Unselect button - to reset your selection if you used any of the Set Graphs to limit set to only pictures that conform to your selected angles
  9. the option to display Set Graphs and Set selection in the first place

The Face Graphs function exactly the same like you'd expect if you ever used Face Graph in MVE.

For example, by selecting Yaw-60, Pitch 5 option on the graph, where it says 7 Faces, it will also show 7 faces (plus their model-generated images for current snapshot) on the left side of the screen
example_face_graph


Timeline

When you click on any of the faces in the bottom part of the interface, if Timeline is enabled (option 5 in the Interface Explained picture), you will see the Aligned Face + all the model-generated pictures that are exported in your snapshots. You can also customize which set samples will be shown (output and/or swap)

An example picture:
timeline_example

Please note that there is a horizontal scroll-bar at the bottom of Timeline, which you can use to see the pictures from all of your snapshots.


Loading Logic and Manual Changes

By default, exporting snapshot does not copy input files (your original Aligned Faces) as the disk size would grow very quickly.

When you have multiple snapshots, the logic in which input files will be loaded is as follows:

  • your highest iteration snapshot checks if it there are input pictures in it - if there are, load them, if not - check previous snapshot
  • the snapshot before that checks if there are input pictures in it - load if there are, if not check previous snapshot
  • it continues like that until the snapshot with the lowest iteration
  • if an input picture wasn't found, check config.json for the location of the original set, and load from there

You can see the exact location where it will search for input files if you open config.json file in a text editor.

If you need to change the files in the original directories that the config.json points to (ie starting on another project), you can manually move/copy the input files in your oldest snapshot (and oldest in this context means the one with the lowest iteration number).

you can see the iteration number of a snapshot within the model_summary.json file (which you can read in any text editor).

For example, you would move/copy SRC input from \workspace\data_src\aligned into \workspace\model\<model_name>_state_history\<oldest_snapshot>\src


Another use for the manual changes is if you changed the files during your training.
In such a case, you can put a copy of initial files in the oldest snapshot's directory, and then the new version of the files in the snapshot after the file change during training.

Clone this wiki locally