Add data and restructure file handling #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
umertens
commented
Oct 16, 2024
|
|
||
| def __len__(self): | ||
| return len(self.file_names) | ||
| return len(self.file_names) * self.n_targets |
There was a problem hiding this comment.
we increase the len to account for multiple targets
umertens
commented
Oct 16, 2024
|
|
||
| def __getitem__(self, idx): | ||
| file_path = os.path.join(self.folder_path, self.sorted_files[idx]) | ||
| actual_idx = idx // self.n_targets |
There was a problem hiding this comment.
we need to map back to original idx, so if there are k rows and 2 targets, idx goes to k*2
umertens
commented
Oct 16, 2024
|
|
||
| def __getitem__(self, idx): | ||
| file_path = os.path.join(self.folder_path, self.sorted_files[idx]) | ||
| actual_idx = idx // self.n_targets | ||
| augment_idx = idx % self.n_targets |
There was a problem hiding this comment.
augment_idx determines which target is used
umertens
commented
Oct 16, 2024
| c, target_embedding = self.task_dict[keys[augment_idx]] | ||
| target_embedding = torch.tensor(target_embedding,dtype=data.dtype) | ||
| data = add_positional_info(target_embedding=target_embedding, feature_embeddings=data) | ||
| return torch.cat((data[:c, :], data[(c+1):, :]), dim=0) |
There was a problem hiding this comment.
remove target embedding
umertens
commented
Oct 16, 2024
| dataset_name, target, _ = filename.split(";") | ||
| return dataset_name, torch.tensor(float(target), dtype=torch.float32) | ||
| dataset_name, target_list, _ = filename.split(";") | ||
| target_list = eval(target_list) |
There was a problem hiding this comment.
the list is encapsulated in a string, hence eval
umertens
commented
Oct 16, 2024
| ds_dir = os.path.join(join_paths(d.current_dir, 'files', mode)) | ||
| datasets.append(TabularDataset(folder_path=ds_dir, | ||
| task_description_path=os.path.join(d.current_dir, 'task_description', mode), | ||
| n_targets=len(d.target_column) if not only_main else 1)) |
There was a problem hiding this comment.
The 1 works since the main target is always the first in the list
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR
What has changed:
We now treat every! column as feature, compute embeddings for each row and store as npy-file. The tensor that is stored has dimension (
n_cols, emb_dim). Within the file name, we store for each target its value and position in the sequence. So ifn_colsis 10 and there are two targets at position 6 and 9, the file name contains[6: 0.3, 9: 0.7]with 0.3 and 0.7 being the respective target values for that particular row. In addition, we store a dict containing the embedding for the task/target description of each target.In the torch dataset, we then:
Why this approach?
When training on multiple targets for a particular dataset, we stored one file per row per target, so with nrows = 100 and n_targts = 3 we stored 300 files. However, with the new approach, we only need to store 100 rows (plus the small dict)