[WIP] #107
[WIP] #107
Conversation
| for pid in range(pid_offset, pid_offset + metadata["num_passages"]): | ||
| pid_to_chunk_idx[pid] = chunk_idx | ||
| pid_doclen = chunk_doclens[pid - pid_offset] | ||
| pid_to_chunk_metadata[pid] = [chunk_idx, pid_doclen, pid_offset_in_chunk] |
There was a problem hiding this comment.
Can we make this a NamedTuple?
| offset = 0 | ||
| for chunk_idx in sorted(chunks.keys()): | ||
| for chunk_idx in sorted(pids_per_chunk.keys()): |
There was a problem hiding this comment.
This is a problem in the code I wrote, but one concern I have is that it may be possible for the pids passed in to lookup_pids to not be in sorted order - if that is the case, then iterating over the chunks in sorted order would produce an output that's inconsistent with what the calling function expects
There was a problem hiding this comment.
I guess the solution would be to iterate over the pids in the original order, then find the chunk each pid belongs to and extract the relevant data from that chunk
Or alternatively, we could sort the pids upstream somewhere (assuming this is not done already)
|
@santhnm2 this is outdated right? Do we close it? |
|
Or is it WIP? |
No description provided.