Increase efficiency gait #80

Erikpostt · 2024-12-03T09:51:58Z

Het doel van deze PR is om de pipeline een stuk efficiënter te maken, met name bij preprocessing en feature extraction. Ik heb de volgende aanpassingen gemaakt:

Quantification weggehaald voor nu (deze komt in een volgende PR terug)
Feature names gestandaardizeerd met _sensor_name__X
Cepstral coefficients (CCs) aangepast naar Mel Frequency Cepstral coefficients (MFFCs) na overleg met Nienke
IMU preprocessing werkt nu met numpy, waardoor o.a. e.e.a geparalleliseerd wordt
Tabulate windows werkt nu met numpy, waardoor de tremor and HR pipeline nu tijdelijk gebruik maken van tabulate_windows_legacy wat nog gebruik maakt van pandas.
Classifiers, scalers en thresholds zijn geupdate met aanpassingen o.b.v pdathome

…increase_efficiency_gait

Copilot reviewed 20 out of 33 changed files in this pull request and generated no suggestions.

Files not reviewed (13)

tests/data/0.classification/gait/scalers/gait_detection_scaler_params.json: Language not supported
tests/data/0.classification/gait/scalers/gait_filtering_scaler_params.json: Language not supported
tests/data/0.classification/gait/thresholds/gait_detection_threshold.txt: Language not supported
tests/data/0.classification/gait/thresholds/gait_filtering_threshold.txt: Language not supported
tests/data/2.preprocessed_data/imu/accelerometer_meta.json: Language not supported
tests/data/3.extracted_features/gait/arm_activity_meta.json: Language not supported
tests/data/3.extracted_features/gait/gait_meta.json: Language not supported
tests/data/5.quantification/gait/arm_swing_meta.json: Language not supported
docs/notebooks/gait/gait_analysis.ipynb: Evaluated as low risk
src/paradigma/heart_rate/heart_rate_analysis.py: Evaluated as low risk
src/paradigma/tremor/tremor_analysis.py: Evaluated as low risk
src/paradigma/constants.py: Evaluated as low risk
src/paradigma/preprocessing_config.py: Evaluated as low risk

Comments skipped due to low confidence (7)

src/paradigma/imu_preprocessing.py:6

The term interp1d should be capitalized as interp1D to match the convention used by the scipy.interpolate module.

from scipy.interpolate import interp1d

src/paradigma/imu_preprocessing.py:43

The condition config.acceleration_units == 'm/s^2' should be compared using DataUnits.ACCELERATION for consistency.

if config.acceleration_units == 'm/s^2':

src/paradigma/imu_preprocessing.py:48

[nitpick] The variable name filter_configs is ambiguous and does not clearly convey its purpose.

filter_configs = {

src/paradigma/imu_preprocessing.py:199

The check if not np.all(np.diff(time_abs_array) > 0) should be performed before creating t_resampled to avoid unnecessary computation.

if not np.all(np.diff(time_abs_array) > 0):

src/paradigma/imu_preprocessing.py:217

The parameter single_sensor_col in the docstring should be updated to data.

single_sensor_col: np.ndarray,

src/paradigma/imu_preprocessing.py:243

The return type in the docstring should be updated to np.ndarray.

sensor_column_filtered: pd.Series

tests/test_gait_analysis.py:116

The test case for arm swing quantification has been commented out. This should be addressed to ensure that the functionality is properly tested.

# def test_6_arm_swing_quantification_output(shared_datadir: Path):

…increase_efficiency_gait

…nson/paradigma into increase_efficiency_gait

Copilot reviewed 21 out of 34 changed files in this pull request and generated 1 suggestion.

Files not reviewed (13)

tests/data/0.classification/gait/scalers/gait_detection_scaler_params.json: Language not supported
tests/data/0.classification/gait/scalers/gait_filtering_scaler_params.json: Language not supported
tests/data/0.classification/gait/thresholds/gait_detection_threshold.txt: Language not supported
tests/data/0.classification/gait/thresholds/gait_filtering_threshold.txt: Language not supported
tests/data/2.preprocessed_data/imu/accelerometer_meta.json: Language not supported
tests/data/3.extracted_features/gait/arm_activity_meta.json: Language not supported
tests/data/3.extracted_features/gait/gait_meta.json: Language not supported
tests/data/5.quantification/gait/arm_swing_meta.json: Language not supported
docs/notebooks/gait/gait_analysis.ipynb: Evaluated as low risk
src/paradigma/heart_rate/heart_rate_analysis.py: Evaluated as low risk
src/paradigma/tremor/tremor_analysis.py: Evaluated as low risk
src/paradigma/constants.py: Evaluated as low risk
src/paradigma/ppg_preprocessing.py: Evaluated as low risk

Comments skipped due to low confidence (6)

src/paradigma/imu_preprocessing.py:214

The parameter 'single_sensor_col' was renamed to 'data', but the docstring was not updated to reflect this change. Update the docstring to match the new parameter name.

single_sensor_col: np.ndarray,

src/paradigma/segmenting.py:10

[nitpick] The parameter 'columns' in 'tabulate_windows' should be clearly documented to explain its purpose.

def tabulate_windows(config, df, columns):

src/paradigma/segmenting.py:109

The function 'create_segments' documentation should be updated to reflect the use of 'config.time_colname' instead of 'time_column_name'.

def create_segments(config, df):

src/paradigma/gait/gait_analysis_config.py:16

[nitpick] The variable name 'l_axes' is ambiguous. It should be renamed to 'axes_list' for clarity.

self.l_axes = ["x", "y", "z"]

src/paradigma/gait/gait_analysis_config.py:78

[nitpick] The variable name 'window_step_length_s' should be consistent with 'window_length_s'. Consider renaming it to 'window_step_size_s'.

self.window_step_length_s: int = 1

src/paradigma/gait/gait_analysis_config.py:102

The variable 'self.sensor' is used before being initialized. Ensure 'self.sensor' is correctly set before this line.

f"{self.sensor}_std_norm": DataUnits.GRAVITY,

2024-12-03T10:22:28Z

src/paradigma/segmenting.py

        time_start='min',  # Start time (min time in each segment)
        time_end='max'     # End time (max time in each segment)
    ).reset_index()

    return df_segment_times


-def discard_segments(df, segment_nr_colname, min_length_segment_s, sampling_frequency):
+def discard_segments(config, df):


The function 'discard_segments' documentation should be updated to reflect the use of 'config.min_segment_length_s' and 'config.sampling_frequency' instead of 'min_length_segment_s' and 'sampling_frequency'.

Erikpostt · 2024-12-03T10:47:29Z

@KarsVeldkamp ready for review. Ik heb nog een aantal aanpassingen gemaakt in de docstrings en variabelnamen voor consistentie, die had ik eigenlijk in een volgende PR moeten doen maar mijn perfectionisme zat me dwars. Excuses voor deze toevoeging!

KarsVeldkamp · 2024-12-03T11:59:59Z

src/paradigma/gait/feature_extraction.py

-    list
-        The aggregated statistics
-    """
+def compute_statistics(data: np.ndarray, statistic: str) -> np.ndarray:


Ziet er goed uit en lijkt idd een stuk efficiënter zo, maar wil je hier ook nog doc strings toevoegen?

KarsVeldkamp · 2024-12-03T12:03:32Z

src/paradigma/gait/feature_extraction.py

-        sampling_frequency: int = 100,
-    ) -> tuple:
-    """Compute the Fast Fourier Transform (FFT) of a signal per window (can probably be combined with compute_fft and simplified).
+def compute_power_in_bandwidth(config, psd, freqs):


Idem en wat bedoel je inde functie met band_mask (miss kun je nog wat comments plaatsen want vond deze niet self-explaining)

KarsVeldkamp · 2024-12-03T12:42:04Z

src/paradigma/gait/feature_extraction.py

-                        if df.loc[index, angle_colname][df.loc[index, f'{angle_colname}_new_minima'][i_pks+1]] < df.loc[index, angle_colname][df.loc[index, f'{angle_colname}_new_minima'][i_pks]]:
-                            df.at[index, f'{angle_colname}_new_minima'] = np.delete(df.loc[index, f'{angle_colname}_new_minima'], i_pks)
-                        # otherwise, keep the current minimum and discard the next minimum
+    distances = config.sampling_frequency * 0.6 / dominant_frequencies


Zijn deze 2 values nog iets voor in de config?

Deze line plus de volgende van prominence

prominence mogelijk wel, maar de dominant_frequencies is een array met een dominant frequency per window. Ik denk dat ik 'm voor nu binnen deze functie houd, omdat ik prominence niet ergens anders gebruik nog.

KarsVeldkamp · 2024-12-03T12:58:15Z

src/paradigma/gait/gait_analysis.py

+    scaler.scale_ = scaler_params['scale']
+    scaler.feature_names_in_ = scaler_params['features']
+
+    df[scaler_params['features']] = scaler.transform(df[scaler_params['features']])


Dit bedoelde je dus ook bij mijn stukje, zal dit aanpassen ;)

KarsVeldkamp · 2024-12-03T13:00:08Z

src/paradigma/gait/gait_analysis.py

+#         window_step_size_s=config.window_step_length_s,
+#         metrics=['range_of_motion', f'peak_{config.velocity_colname}'],
+#         aggregates=['median'],
+#         quantiles=[0.95]


wil je de quantiles nog naar je config zetten? Weet dat hij nu uitgecomment is dus miss was je dat straks ook al van plan

Goeie, had ik nog niet over nagedacht maar ga ik zeker doen!

KarsVeldkamp · 2024-12-03T13:07:27Z

tests/data/0.classification/gait/thresholds/gait_detection_threshold.txt

Eea opnieuw getraind?

Zeker, een aantal spectral features zijn aangepast (o.a. MFCCs die nu mel-scale gebruiken)

KarsVeldkamp

Hi Erik,

Ziet er goed uit, paar kleine dingetjes in de comments maar het lijkt idd een stuk efficiënter te zijn! Goed werk!

Erikpostt added 14 commits November 28, 2024 15:28

Nothing

90b6875

merged

124567b

Vectorize functions using numpy

6fbeacb

Fix bugs

5a3062d

Add case for single window

2d3ea7a

Add bandwidth of first harmonic arm swing

7c68868

Consistency

c6d2b73

Merge branch 'main' of github.com:biomarkersParkinson/paradigma into …

d511680

…increase_efficiency_gait

Update poetry and merge into main

0c0506c

Remove quantification from notebook

499d3e8

Remove quantification temporarily

94eb5c2

Adjust to windowing changes

e3be843

Adjust testing to new numpy processing format

75c03e4

Merge branch 'main' of github.com:biomarkersParkinson/paradigma into …

d682753

…increase_efficiency_gait

Erikpostt requested a review from Copilot December 3, 2024 09:52

KarsVeldkamp added 2 commits December 3, 2024 10:53

Adding signal quality classification code

fa04bf1

adding classifier

4d5403d

Copilot AI reviewed Dec 3, 2024

View reviewed changes

Erikpostt added 5 commits December 3, 2024 11:09

Merge branch 'main' of github.com:biomarkersParkinson/paradigma into …

4d5dd19

…increase_efficiency_gait

Merge branch 'increase_efficiency_gait' of github.com:biomarkersParki…

6a2d761

…nson/paradigma into increase_efficiency_gait

Adjust param specifications

6709f21

Update docstrings and improve chaining

aaa7296

Include option for 1D sensors

03516e0

Erikpostt requested a review from Copilot December 3, 2024 10:20

Copilot AI reviewed Dec 3, 2024

View reviewed changes

Erikpostt added 3 commits December 3, 2024 11:26

Add local pytest

3320d93

Remove list declaration in name

cc3b561

Update docs segmentation

cd07d9c

Erikpostt requested a review from KarsVeldkamp December 3, 2024 10:46

Erikpostt assigned KarsVeldkamp Dec 3, 2024

KarsVeldkamp reviewed Dec 3, 2024

View reviewed changes

KarsVeldkamp marked this pull request as ready for review December 3, 2024 13:07

KarsVeldkamp approved these changes Dec 3, 2024

View reviewed changes

KarsVeldkamp assigned Erikpostt and unassigned KarsVeldkamp Dec 3, 2024

Erikpostt added 3 commits December 3, 2024 15:10

Added docstrings and improved inline documentation

d992aec

Solve bug

2616767

Change docstrings for data types

9f8420c

Erikpostt merged commit 5ed7fd4 into main Dec 3, 2024
1 check passed

Erikpostt deleted the increase_efficiency_gait branch December 4, 2024 15:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase efficiency gait #80

Increase efficiency gait #80

Erikpostt commented Dec 3, 2024

Copilot AI left a comment

Copilot AI Dec 3, 2024

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Erikpostt commented Dec 3, 2024

KarsVeldkamp Dec 3, 2024

KarsVeldkamp Dec 3, 2024

KarsVeldkamp Dec 3, 2024

KarsVeldkamp Dec 3, 2024

Erikpostt Dec 3, 2024

KarsVeldkamp Dec 3, 2024

KarsVeldkamp Dec 3, 2024

Erikpostt Dec 3, 2024

KarsVeldkamp Dec 3, 2024

Erikpostt Dec 3, 2024

KarsVeldkamp left a comment

Increase efficiency gait #80

Increase efficiency gait #80

Conversation

Erikpostt commented Dec 3, 2024

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot AI Dec 3, 2024

Choose a reason for hiding this comment

Erikpostt commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KarsVeldkamp left a comment

Choose a reason for hiding this comment