Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

downsampling causes errors in calibration #153

Open
mshallow opened this issue Jun 13, 2024 · 45 comments
Open

downsampling causes errors in calibration #153

mshallow opened this issue Jun 13, 2024 · 45 comments

Comments

@mshallow
Copy link

I have been trying to use the downsampling built into your code database, and it seems successful at downsampling the data (the initial lines for temporal downsampling don't error) but then when I try to run the calibration step, it loads ~30-50% of the frames needed and then errors. The error message is attached here.
I am currently trying to downsample from 200 fps to 40 fps, so I set the downsampling factor to 5, I also tried with the example downsampling factor of 2 and ran into the same error. It seems to be indexing something incorrectly after the downsampling occurs.

Code:
`# load data (e.g. from DeepLabCut)
keypoint_data_path = '/Users/mollyshallow/Desktop/new_demo_project/data' # can be a file, a directory, or a list of files
coordinates, confidences, bodyparts = kpms.load_keypoints(keypoint_data_path, 'deeplabcut', exclude_individuals='single')

#downsample data
downsample_rate = 5 # keep every 2nd frame
coordinates = kpms.downsample_timepoints(coordinates, downsample_rate)
confidences = kpms.downsample_timepoints(confidences, downsample_rate)

format data for modeling

data, metadata = kpms.format_data(coordinates, confidences, **config())

kpms.noise_calibration(project_dir, coordinates, confidences, **config(), downsample_rate=downsample_rate)`

Error Message:
`Loading sample frames: 49%|█████▊ | 40/82 [00:02<00:02, 16.49it/s]

ValueError Traceback (most recent call last)
Cell In[11], line 1
----> 1 kpms.noise_calibration(project_dir, coordinates, confidences, **config(), downsample_rate=downsample_rate)

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/keypoint_moseq/calibration.py:528, in noise_calibration(project_dir, coordinates, confidences, bodyparts, use_bodyparts, video_dir, video_extension, conf_pseudocount, downsample_rate, **kwargs)
525 annotations = load_annotations(project_dir)
526 sample_keys.extend(annotations.keys())
--> 528 sample_images = load_sampled_frames(
529 sample_keys, video_dir, video_extension, downsample_rate
530 )
532 return _noise_calibration_widget(
533 project_dir,
534 coordinates,
(...)
540 **kwargs,
541 )

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/keypoint_moseq/calibration.py:110, in load_sampled_frames(sample_keys, video_dir, video_extension, downsample_rate)
102 readers = {key: OpenCVReader(video) for key, video in zip(keys, videos)}
103 pbar = tqdm.tqdm(
104 sample_keys,
105 desc="Loading sample frames",
(...)
108 ncols=72,
109 )
--> 110 return {
111 (key, frame, bodypart): readers[key][frame * downsample_rate]
112 for key, frame, bodypart in pbar
113 }

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/keypoint_moseq/calibration.py:111, in (.0)
102 readers = {key: OpenCVReader(video) for key, video in zip(keys, videos)}
103 pbar = tqdm.tqdm(
104 sample_keys,
105 desc="Loading sample frames",
(...)
108 ncols=72,
109 )
110 return {
--> 111 (key, frame, bodypart): readers[key][frame * downsample_rate]
112 for key, frame, bodypart in pbar
113 }

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/vidio/read.py:70, in BaseReader.getitem(self, *args, **kwargs)
62 def getitem(self, *args, **kwargs) -> Union[np.ndarray, list]:
63 """Wrapper around read
64
65 Args:
(...)
68 frame = reader[10]
69 """
---> 70 return self.read(*args, **kwargs)

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/vidio/read.py:107, in OpenCVReader.read(self, framenum)
102 """Read the frame indicated in framenum from disk
103
104 Uses sequential reads where possible if using OpenCV to read
105 """
106 # does checks. if framenum is a slice, calls read recursively. In that case, just return
--> 107 output = super().read(framenum)
108 if output is not None:
109 return output

File /opt/anaconda3/envs/keypoint_moseq/lib/python3.9/site-packages/vidio/read.py:39, in BaseReader.read(self, framenum)
37 return [self.read(i) for i in self.slice_to_list(framenum)]
38 if framenum < 0 or framenum > self.nframes:
---> 39 raise ValueError('frame number requested outside video bounds: {}'.format(framenum))

ValueError: frame number requested outside video bounds: 270590
Loading sample frames: 49%|█████▊ | 40/82 [00:19<00:02, 16.49it/s]`
@calebweinreb
Copy link
Contributor

Hmm weird. To diagnose, could you pick a recording and then tell me the shape the corresponding array in coordinates before and after downsampling, and also the number of frames in the corresponding video?

@mshallow
Copy link
Author

mshallow commented Jun 14, 2024 •
edited
Loading

Yea I can do that! Just using the first video in my list of videos as a test, here are the results of testing that out.

full_array=coordinates['Sky_mouse-0897_2022-05-06T08_16_08DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_filtered_mouse'] np.shape(full_array)
(7687, 6, 2)

downsample_rate = 5 # keep every 2nd frame coordinates_down = kpms.downsample_timepoints(coordinates, downsample_rate) confidences_down = kpms.downsample_timepoints(confidences, downsample_rate)
downsamp_array=coordinates_down['Sky_mouse-0897_2022-05-06T08_16_08DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_filtered_mouse'] np.shape(downsamp_array)
(308, 6, 2)

The length of the original coordinates array should be the full length of frames of the video. The video is ~38s long at 200fps which is around 7600 frames.

@calebweinreb
Copy link
Contributor

can you check this systematically?

from vidio.read import OpenCVReader

keys = sorted(coordinates.keys())
videos = kpms.find_matching_videos(keys, video_dir)
for key,video in zip(keys,videos):
    print(len(coordinates[key]), len(OpenCVReader(video)), key)

@mshallow
Copy link
Author

Using that method, I get these values:
1538 7687 Sky_mouse-0897_2022-05-06T08_16_08DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_filtered_mouse
The first seems like it's the downsampled frames, and the second the full length, which doesn't really match the output that I got using the other method, so not quite sure what is going on there.

@calebweinreb
Copy link
Contributor

"which doesn't really match the output that I got using the other method".. can you elaborate on that?

also can you do this for all your data? Given the frame number in the error "270590" it seems like the short video you've been testing isn't the one that caused the error.

@mshallow
Copy link
Author

Here is the output from all the videos:
731 3653 Sky_mouse-0893_2022-07-11T08_36_51DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1216 6077 Sky_mouse-0893_2022-07-12T08_47_30DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1404 7017 Sky_mouse-0893_2022-07-19T09_45_45DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1844 9220 Sky_mouse-0893_2022-07-21T11_59_16DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1368 6838 Sky_mouse-0893_2022-07-31T11_31_56DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1198 5988 Sky_mouse-0893_2022-08-22T10_55_14DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1080 5397 Sky_mouse-0893_2022-08-22T10_56_56DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 3350 16746 Sky_mouse-0895_2022-07-12T10_42_42DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2272 11360 Sky_mouse-0895_2022-07-26T10_45_21DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1972 9859 Sky_mouse-0895_2022-07-29T10_27_36DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 3626 18126 Sky_mouse-0895_2022-07-31T10_26_26DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2195 10974 Sky_mouse-0895_2022-08-15T11_55_18DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1820 9098 Sky_mouse-0895_2022-08-22T10_16_09DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1359 6794 Sky_mouse-0895_2022-08-23T10_49_34DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 12870 64347 Sky_mouse-0896_2022-04-06T10_01_10DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2074 10367 Sky_mouse-0896_2022-04-13T09_03_40DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2255 11271 Sky_mouse-0896_2022-04-27T08_35_14DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1518 7590 Sky_mouse-0896_2022-05-04T09_33_27DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2437 12182 Sky_mouse-0897_2022-04-04T09_15_51DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1404 7017 Sky_mouse-0897_2022-04-08T09_03_27DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 344 1720 Sky_mouse-0897_2022-04-14T08_39_05DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 1538 7687 Sky_mouse-0897_2022-05-06T08_16_08DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_filtered_mouse 1589 7943 Sky_mouse-0898_2022-04-12T12_42_28DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 2168 10838 Sky_mouse-0898_2022-04-29T09_16_40DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 897 4485 Sky_mouse-0898_2022-05-03T09_52_50DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 11883 59412 Sky_mouse-1337_2023-01-05T11_37_41DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 2656 13277 Sky_mouse-1337_2023-01-13T12_25_28DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 12309 61545 Sky_mouse-1337_2023-01-15T16_46_51DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 13194 65969 Sky_mouse-1337_2023-01-20T11_02_26DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1518 7588 Sky_mouse-1429_2023-01-04T15_51_08DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1572 7858 Sky_mouse-1429_2023-01-15T16_15_26DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1441 7204 Sky_mouse-1429_2023-01-16T18_00_32DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 2027 10131 Sky_mouse-1429_2023-01-24T11_32_56DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1929 9642 Sky_mouse-1429_2023-01-29T12_54_54DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 2496 12479 Sky_mouse-1430_2023-01-16T16_46_28DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1190 5949 Sky_mouse-1430_2023-01-17T10_39_22DLC_dlcrnetms5_optopreycapFeb16shuffle1_150000_el_mouse 5683 28411 Sky_mouse-1430_2023-01-23T10_37_18DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 2122 10607 Sky_mouse-1430_2023-01-24T12_15_10DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1472 7360 Sky_mouse-1430_2023-01-24T12_23_54DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse 1699 8494 Sky_mouse-1430_2023-01-29T13_25_16DLC_resnet50_optopreycapFeb16shuffle1_300000_el_mouse

And previously to try to figure out the length of the downsampled array I had just looked at one entry in the dictionary of coordinates, and after downsampling, it was (308,6,2) which doesn't match up with taking every 5th frame as the downsampling factor should be telling it to do, the value of 1538 is the correct downsampled length for the coordinates.

@calebweinreb
Copy link
Contributor

Hmm it's also confusing where "270590" came from since none of the coodinates are even 1/5 that long...

My guess is that there is some accidental failure to downsample or double-downsampling happening here. Can run the code from a clean starting point and do the following?

  1. Load coordinates fresh
  2. Run this code block before downsampling
from vidio.read import OpenCVReader

keys = sorted(coordinates.keys())
videos = kpms.find_matching_videos(keys, video_dir)
for key,video in zip(keys,videos):
    if len(coordinates[key]) != len(OpenCVReader(video)):
        print(len(coordinates[key]), len(OpenCVReader(video)), key)

assuming nothing prints, try running calibration.

BTW calibration itself is kind of buggy lately but this troubleshooting will also be useful for the subsequent viz steps

@mshallow
Copy link
Author

There are no steps in this that downsample at all, were you suggesting just to try this to see if it was a bug with calibration even without the downsampling?
When I tried this, there was no output from that code block, and everything for the calibration loaded completely fine.
Is there a possibility that something is being concatenated to load the frames for the calibration and that is where the 270590 comes from? I've messed around with a couple of different downsampling rates, and if I set the downsampling rate to 2, the same error occurs, but the value error is then "ValueError: frame number requested outside video bounds: 108236"

@calebweinreb
Copy link
Contributor

O sorry yeah I forgot the downsampling. Can you try the following?

keypoint_data_path = '/Users/mollyshallow/Desktop/new_demo_project/data' # can be a file, a directory, or a list of files
coordinates, confidences, bodyparts = kpms.load_keypoints(keypoint_data_path, 'deeplabcut', exclude_individuals='single')

downsample_rate = 5 # keep every 2nd frame
coordinates = kpms.downsample_timepoints(coordinates, downsample_rate)
confidences = kpms.downsample_timepoints(confidences, downsample_rate)

from vidio.read import OpenCVReader
keys = sorted(coordinates.keys())
videos = kpms.find_matching_videos(keys, video_dir)
for key,video in zip(keys,videos):
    if (len(coordinates[key])-1)*downsample_rate >= len(OpenCVReader(video)):
        print(len(coordinates[key]), len(OpenCVReader(video)), key)

kpms.noise_calibration(project_dir, coordinates, confidences, **config(), downsample_rate=downsample_rate)

@mshallow
Copy link
Author

That still gave the same error.

@calebweinreb
Copy link
Contributor

Have you run calibration previously? If so it may looking for frames from a non-downsampled instance of calibration. Look for a file called error_annotations.csv in the project directory, delete if present and try again?

@mshallow
Copy link
Author

Ok that seems to be what it was doing, deleting the file fixed the error! Thanks!

@mshallow
Copy link
Author

mshallow commented Jun 14, 2024 •
edited
Loading

I also don't know if anyone else has encountered general bugginess with the calibration, but I feel like in more recent times I've tried to use it, the loading of the frames seems really glitchy. It'll load the first couple for me to click through smoothly and then after that you have to advance two or three at a time to get it to change frames or load the image and not just the skeleton, unless I wait a long time (around 10+ seconds), before I try to advance.

@calebweinreb
Copy link
Contributor

That's different from other bug reports but glitchiness is the consensus. We're planning to change the backend for calibration in the next release/

@mshallow
Copy link
Author

Following up on downsampling bugs: initial calibration step is no longer erroring, but after downsampling, there appear to be strange issues with the trajectory plots and grid videos. From what I can tell, it seems like something with the scaling for generating these plots and videos is off.
The trajectory plots that are generated after the PCA to initialize the model look like the correct scale (see attached photo) but after training the model, all of the points are overlayed on top of each other (second attached photo). The same issue seems to apply to the grid movies where everything is super zoomed out rather than cropping and zooming in on the mouse like it did previously. None of these lines of code throw actual errors, just user warnings, but I was curious if solving these warnings would change the output or if there is something else going on.
Before training model plots from PCA:

Screenshot 2024-06-18 at 3 09 48 PM

Trajectory plots after training:

Screenshot 2024-06-18 at 3 10 13 PM

Grid movies after training:

Screenshot 2024-06-18 at 3 10 40 PM

Screenshot 2024-06-18 at 3 11 35 PM

I tried running this with two different downsampling rates as well as two different kappa values and encountered the same issues.

@calebweinreb
Copy link
Contributor

HI,

Hmm that's weird! But it doesn't strike me as related to downsampling per se. Have you ever run kpms without downsampling? Did it work in that case?

@mshallow
Copy link
Author

Yea it has always worked without downsampling. I just tried it again on the same dataset without downsampling since previous attempts without downsampling were run on a different computer, and did not encounter this issue.

@mshallow
Copy link
Author

mshallow commented Jun 19, 2024 •
edited
Loading

This is the same set of outputs from a run without downsampling:
Screenshot 2024-06-19 at 2 41 44 PM

Screenshot 2024-06-19 at 2 39 59 PM
Screenshot 2024-06-19 at 2 40 08 PM
Screenshot 2024-06-19 at 2 40 28 PM

@calebweinreb
Copy link
Contributor

Hmm maybe the fitting got wonky for some reason. Can you try exporting the inferred coordinates and see if they're weird?

https://keypoint-moseq.readthedocs.io/en/latest/advanced.html#exporting-pose-estimates

@mshallow
Copy link
Author

The first half of that code runs fine, and then when it gets to making the video and overlaying the coordinates, it gives me a similar issue to what I was encountering with the calibration earlier.
Error message:
IndexError: index 1495 is out of bounds for axis 0 with size 1495
Just by eye, there are a few coordinates that look a little weird ( a bunch of negative values or really high values) but not quite sure how to systematically check that.
These are the coordinate outputs from the first video in the dictionary for the downsampled data and not downsampled data.
Downsampled:
Screenshot 2024-06-19 at 2 51 13 PM

Not downsampled:
Screenshot 2024-06-19 at 2 51 26 PM

@mshallow
Copy link
Author

mshallow commented Aug 8, 2024

I just wanted to check back in and see if you had any more insight as to what could be going wrong with the downsampling here. I have tried this a couple more times with different kappa values and clearing all of the outputs from the directory before starting with the downsampling and the same issue arises every time where it appears to have trained fine and have decent median syllable lengths, but it only finds about 2 syllables and the trajectory plots look very wrong.

@calebweinreb
Copy link
Contributor

So just to be confirm:

  • without downsampling, everything runs fine and the syllables look normal
  • with downsampling by 5X, only 2 syllables are found (based on trajectory plots)
  • also there's an additional issue of an IndexError when making grid movies?

I'm totally sure what the problem is... how much total data do you have? Are there still very few syllables (+ weird trajectory plots) when you only downsample 2X?

@mshallow
Copy link
Author

Yes correct. Without downsampling, everything looks completely fine and the syllables look normal, but with downsampling it finds very few states and the plotting/ grid movies look really weird.
I have been running this on a test dataset of ~40 videos that are 1-5min in length at 200fps so should contain a couple hundred thousand frames as was suggested in your documentation. I just tried running the downsampling with 2x downsampling factor and it found ~10 syllables, but the same issue still occurs with the trajectory plots and grid movies.

@calebweinreb
Copy link
Contributor

Hmm maybe at this point the easiest thing would be to send me the dataset and the code you have been running

@mshallow
Copy link
Author

mshallow commented Aug 13, 2024 •
edited
Loading

I'd be happy to do that! What is the easiest format to get the data to you? I've just been working in your demo jupyter notebooks for the majority of the code base, but I can share the versions I'm using for any of the changes I've made over time.

@calebweinreb
Copy link
Contributor

Google drive or dropbox to [email protected]

@mshallow
Copy link
Author

mshallow commented Aug 13, 2024 via email

@mshallow
Copy link
Author

Any insights into what might be happening with the downsampling?

@calebweinreb
Copy link
Contributor

I haven't confirmed it yet but I think the problem is coming from two recordings where the keypoint tracking went screwy. You can see which ones in the y-axis labels of the attached screenshot
Screenshot 2024-08-26 at 3 27 10 PM

Can you try modeling with those recordings excluded and let me know how it goes?

@mshallow
Copy link
Author

mshallow commented Aug 26, 2024 via email

@mshallow
Copy link
Author

That doesn't seem to have changed it. The training of the model looked a little better, longer and more states, but the trajectory plots still are weirdly small.

@calebweinreb
Copy link
Contributor

OK I'll take another look.

@mshallow
Copy link
Author

I also tried running the modeling with the data at its full frame rate (no downsampling) and now the trajectory plots for that run also look the same.
Screenshot 2024-08-27 at 11 32 35 AM

@calebweinreb
Copy link
Contributor

Hi,

So I tried modeling your data in a bunch of different configurations:

  • latent_dim=2 vs. latent_dim=4
  • with or without excluding the bad session mentioned above
  • with or without location-aware modeling
  • with 5X downsampling or no downsampling

In every case, the trajectory plots looked very reasonable and didn't look like the screenshot you posted above. I'm not entirely sure what's going on. You can see all the notebooks I used and their full output including checkpoints and everything here.

Separate from the trajectory plot issue, I had some suggestions based on your modeling notebook:

  • Only using the right ear for anterior_keypoints leads to the mouse always being slightly tilted. It would be better to set anterior_keypoints=["Rear", "Lear"]
  • latent_dim=2 is very low (even if it does explain 90% of variance). I would suggest latent_dim=3 or 4 instead.
  • In the kappa scan, kappa is decreased by a factor of 10 between the ar_only and full model steps. You should recapitulate this when you run those steps yourself.
  • If you plan to set "location_aware=True" in your final modeling then you should also do so for the kappa scan.

@mshallow
Copy link
Author

mshallow commented Aug 28, 2024 via email

@calebweinreb
Copy link
Contributor

The maximum of 1e7 is just because that's typically high enough. But higher values could definitely be necessary, especially for high speed video where the syllable durations are a higher number of frames.

@mshallow
Copy link
Author

mshallow commented Aug 28, 2024 via email

@calebweinreb
Copy link
Contributor

The only way error calibration would affect things is via the "slope" and "intercept" parameters under "error_estimator" in the config. It's possible you were using a different values of those parameters which could have caused the problem.

@mshallow
Copy link
Author

mshallow commented Aug 28, 2024 via email

@mshallow
Copy link
Author

Ok as far as I can tell, the only difference between how you had the notebooks you created running and how I had been working with the data is the jitter value. Due to NaNs in the dataset, previously you had suggested that I set that value to the maximum value of 1e-1. When I do this, that is when I get the strange coordinates and the trajectory plots don't look correct. If I leave it at the default value, the training of the model terminates early after finding too many NaNs in the data, but the trajectory plots are not messed up. Without jitter=1e-1, generally it gets through about 50% of the iterations, sometimes 60-65% before it terminates. would you say that is enough iterations, or should I figure out a way to get it to run for the full 500?

@calebweinreb
Copy link
Contributor

I'm going to test a few things to get rid of the NaNs. Also I have an idea to make your trajectory plots render more clearly but I'd like to test it on a real example. Would you be willing to send me the results.h5 file from which the weird looking trajectory plots were generated?

@mshallow
Copy link
Author

mshallow commented Aug 30, 2024 via email

@mshallow
Copy link
Author

mshallow commented Aug 30, 2024 •
edited
Loading

I uploaded several folders from different times that I tried to train the model, two with the weird trajectory plots, and one without. The two models that were trained today (8/30) are identical except for the jitter value; the one with normal trajectory plots (11_51_33) was run with a jitter=1e-2 and the one with weird plots (12_35_27) was run with 1e-1.
I uploaded everything including checkpoints, training progress and the plots and grid videos as well as the results.h5 just in case you needed anything else.
Neither of these trajectory plots are the full size that the not downsampled data give, but without jitter=1e-1 is definitely better.

@mshallow
Copy link
Author

Did you ever figure out a way to eliminate some of the NaNs or get the trajectory plots to render more clearly? I tested a couple more things and without the jitter=1E-1 the plots are definitely better but still not the full size that they were without downsampling. Does that have something to do with the amount of data or is it just a visualization change? I'm trying to figure out how important the size of those plots is or if the syllable data are fine and could be used for further data analysis.

@calebweinreb
Copy link
Contributor

Hey! So sorry for the late response. A bunch of stuff came up and I lost the thread on this. Overall, regarding NaNs, I don't have a full solution but in general would recommend cleaning up the keypoint data as much as possible and removing any sessions that seem like they contain a lot of errors. But it seems like you can get pretty decent results before the NaNs happen so it might be fine to just go with that.

Regarding the trajectory plots, I was hoping to implement a more general solution but for now you can use the lims parameter of generate_trajectory_plots. Basically what's currently happening is that the limits are being set too large so al the keypoints get smooshed together. You can avoid that by setting the lims manually. Here's the relevant part of the docstring:

lims: ndarray of shape (2,2), default=None
        Axis limits used for all the trajectory plots with format
        `[[xmin,ymin],[xmax,ymax]]`. If None, the limits are determined
        automatically based on the coordinates of the keypoints using
        :py:func:`keypoint_moseq.viz.get_limits`.

I would recommend experimenting until you get a good size where the keypoints arent getting cropped but also you can see them clearly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@calebweinreb @mshallow