tsfresh to generate features for test data

[manjrekarom]

Greetings,

I am using tsfresh for generating features which I then want to use for clustering the data. The way I am doing that is by using extract_features with default arguments (as shown here) to get the features, imputing and standard scaling them, and using PCA to get a subset of the features since I don't have any labels for the data. I can't use select_features because I do not have labels.

Now the issue is when it comes to do the same on test data, extract_features returns the features of different dimension (3935 in test vs 3184 in train). I tried using settings using from_columns API as shown here but that makes the output of extract_features (0, 0). I am guessing that is because I didn't apply feature selection on the features.

So my question is how do I extract those same 3184 features obtained from extract_features on test set?

[nils-braun]

Hi!

Are you applying any inputing already during the feature extraction?

In principle, the number of extracted features does only depend on the settings you use (as you said default, I assume you did not change those) and the number of unique kinds you have.
If you have both dataframes, could you just do a diff of the column sets to see, which columns are additional?

[manjrekarom]

Thanks for your reply.

Do you mean imputing?

After feature extraction I impute using the tsfresh impute utility. The feature size before and after imputing is the same.

I am using extract_features by providing only the dataframe and column_id, no settings explicitly. But as the feature size differed, I used the from_columns utility to fetch and provide the settings. But that gave me features with size (0, 0).

I should mention that first I extract features for about 400 time-series (put in a single dataframe) and then while testing I do it for 1 time-series (again in a single dataframe). The procedure is identical but the output features aren't the same.

I will try what you suggested and reply shortly. Thanks again.

[manjrekarom]

Test sample features

>>> test_sample.columns
Index(['0__variance_larger_than_standard_deviation', '0__has_duplicate_max',
       '0__has_duplicate_min', '0__has_duplicate', '0__sum_values',
       '0__abs_energy', '0__mean_abs_change', '0__mean_change',
       '0__mean_second_derivative_central', '0__median',
       ...
       '4__permutation_entropy__dimension_5__tau_1',
       '4__permutation_entropy__dimension_6__tau_1',
       '4__permutation_entropy__dimension_7__tau_1',
       '4__query_similarity_count__query_None__threshold_0.0',
       '4__matrix_profile__feature_"min"__threshold_0.98',
       '4__matrix_profile__feature_"max"__threshold_0.98',
       '4__matrix_profile__feature_"mean"__threshold_0.98',
       '4__matrix_profile__feature_"median"__threshold_0.98',
       '4__matrix_profile__feature_"25"__threshold_0.98',
       '4__matrix_profile__feature_"75"__threshold_0.98'],
      dtype='object', length=3935)

Train set features

>>> train_set.columns
Index(['1__variance_larger_than_standard_deviation', '1__has_duplicate_max',
       '1__has_duplicate_min', '1__has_duplicate', '1__sum_values',
       '1__abs_energy', '1__mean_abs_change', '1__mean_change',
       '1__mean_second_derivative_central', '1__median',
       ...
       '0__permutation_entropy__dimension_5__tau_1',
       '0__permutation_entropy__dimension_6__tau_1',
       '0__permutation_entropy__dimension_7__tau_1',
       '0__query_similarity_count__query_None__threshold_0.0',
       '0__matrix_profile__feature_"min"__threshold_0.98',
       '0__matrix_profile__feature_"max"__threshold_0.98',
       '0__matrix_profile__feature_"mean"__threshold_0.98',
       '0__matrix_profile__feature_"median"__threshold_0.98',
       '0__matrix_profile__feature_"25"__threshold_0.98',
       '0__matrix_profile__feature_"75"__threshold_0.98'],
      dtype='object', length=3148)

The difference

>>> test_sample[test_sample.columns.difference(train_set.columns)].columns
Index(['4__abs_energy', '4__absolute_sum_of_changes',
       '4__agg_autocorrelation__f_agg_"mean"__maxlag_40',
       '4__agg_autocorrelation__f_agg_"median"__maxlag_40',
       '4__agg_autocorrelation__f_agg_"var"__maxlag_40',
       '4__agg_linear_trend__attr_"intercept"__chunk_len_10__f_agg_"max"',
       '4__agg_linear_trend__attr_"intercept"__chunk_len_10__f_agg_"mean"',
       '4__agg_linear_trend__attr_"intercept"__chunk_len_10__f_agg_"min"',
       '4__agg_linear_trend__attr_"intercept"__chunk_len_10__f_agg_"var"',
       '4__agg_linear_trend__attr_"intercept"__chunk_len_50__f_agg_"max"',
       ...
       '4__symmetry_looking__r_0.9500000000000001',
       '4__time_reversal_asymmetry_statistic__lag_1',
       '4__time_reversal_asymmetry_statistic__lag_2',
       '4__time_reversal_asymmetry_statistic__lag_3',
       '4__value_count__value_-1', '4__value_count__value_0',
       '4__value_count__value_1', '4__variance',
       '4__variance_larger_than_standard_deviation',
       '4__variation_coefficient'],
      dtype='object', length=787)

[nils-braun]

That looks as if you have an additional "kind" in your test data sample. The "4" at the beginning of each feature indicates the kind/column it came from.

As you are not passing in any "column_kind" (as you wrote), I think that the input data is already different in the columns. Could you show the head of both the test and train set?
tsfresh extracts features separately for every "kind" (= column) it gets. It extracts 787 per column. As your test data set is exactly 3935 (= 5 * 787) and your train set is 3148 (= 4 * 787) I think you have a different number of columns in your data sets.

Just our of curiosity: did you name your columns 0, 1, 2, 3, 4 just for this example or could it maybe be the case, that you try to represent your time series row-wise?

[manjrekarom]

Thank you so much. I accidentally updated the dataframe and thus it had an extra column which was not present in the train set.

Sorry for the trouble and thanks a lot for helping out.

Oh I see those 0, 1, 2 ... in the features names represented the column on which feature is calculated.

The columns are features for the time-series. Specifically they are x and y of corner of objects at each frame. The time-series is multiple frames for each object and like that I have multiple time-series. No I didn't name columns that way. I had to convert from numpy array to dataframes and pandas used the default names. I added an id column to the dataframes for each of the individual time-series which is required by tsfresh to work.