So you want to develop a new metric?: Read this first!

[npatki]

The SDMetrics library provides a set of tools for evaluating synthetic data. We're excited to foster an open source community of users for contributing metrics for the various different uses of synthetic data.

We welcome new ideas for metrics! If you want to contribute your metric to the library, read through this discussion and then reach out to the SDV maintainers by filing an issue or messaging us on Slack.

Defining a metric

All metrics in this library are model-agnostic. Anyone who wants to use your metric should already have:

1. A real dataset
2. A synthetic dataset, which could be created using any model (not just SDV)

Base Usage

The base version of your metric takes in real and synthetic data with the smallest possible unit of data. This could be a column, a pair of columns, a table, etc. For example, if your metric is computing pairwise correlations, the base unit is a pair of columns.

The base metric is a class with a compute method. The method takes in the unit of real data, synthetic data and any other keyword args you want to add. It returns a score, represented as floating point value.

YourMetricName.compute(
  real_data[['column_1', 'column_2']],
  synthetic_data[['column_1', 'column_2']],
  kwarg1= ...
)
0.1234

Aggregate Usage

In many cases, you may want to iterate through the entire dataset to apply the base metric to different columns, pairs of columns, tables, etc. You can write a convenience method called compute_breakdown that performs this iteration.

This method takes in the full real data, synthetic data, keyword args and metadata. According to the metadata, you can determine when to apply the base metric. The metric returns a dictionary of results, keyed by the base unit.

YourMetricName.compute_breakdown(
  real_data,
  synthetic_data,
  metadata,
  kwarg1=...
)

{
  ('column_1', 'column_2'): 0.1234,
  ('column_1', 'column_3'): 0.5678,
  ('column_2', 'column_3'): 0.9012,
  ...
}

Think through this before implementing a metric

We are always aiming to improve the feature set and usability of SDMetrics. Before implementing your metric, think through the questions below.

1. Description

• What aspect of synthetic data does your metric evaluate? This could be privacy, quality or something else
• What is a high-level, non-technical description you can give to a user?
• What is a detailed, technical explanation you can give to a user who wants to dig deeper into the math or algorithms? Cite any relevant papers, wikipedia articles, formulas etc.

2. Basic Information

• Dataset compatibility: Can it run on single table, sequential or multi-table datasets?
• Data type compatibility: Can it run on numerical, categorical, boolean, datetime or ID columns?
• Is it acceptable to run your metric on data with missing values?
• If your metric introduces a new dependency, why is it needed? We want to avoid adding unnecessary dependencies, especially if the library is small and not frequently updated.

3. Usage

• When would you recommend using this metric vs. other, similar metrics in SDMetrics? If you are implementing multiple variants of the metric, what are the differences between them? If different variants are highly correlated, consider creating a single metric with different parameters.
• If you have additional parameters in your metric, what do they do? What are the defaults and when should a user consider changing them?
• Is the score deterministic? If it isn't, how much will it generally vary?

4. Interpretation

• What is the range of possible scores? Which one is best and which is the worst?
• Is there anything the user can do to better contextualize the score? For example, when using an MLEfficacy metric, users can get a baseline by running it on a train/test split of the real data. You can also consider providing benchmarks for well-known examples.

Next Steps

The questions in this discussion should give you a clearer idea of how your metric should work. You can file an issue or reach out to the SDV team on Slack to discuss the next steps of implementing your metric.