Map chunk query #277

ekg · 2024-09-25T17:00:13Z

Restructuring parallelism in mapping, with the goal of implementing parallel processing across several aspects of mapping.

….hpp

…queries with improved memory management

…tomic_queue_t

…unction

…mputeMap.hpp

…segment mapping completion

…g in computeMap module

…ment_queue

…liseconds

…regatedMappings

…_parameters.hpp

ekg added 30 commits September 24, 2024 21:46

feat: replace ThreadPool with atomic queues and threads in computeMap…

57e2169

….hpp

fix: Declare allowed_query_names in reader_thread function

08773c0

build: Improve AddressSanitizer configuration for detailed reports

490edec

refactor: restructure mapModule and threading logic to handle entire …

d6c8d5e

…queries with improved memory management

fix: resolve type mismatch by changing output_queue to query_output_a…

bea4c1a

…tomic_queue_t

clean up useless allocation

4267b9c

correct synchronization issues (race) leading to incomplete processing

6fb2fa9

feat: Parallelize the split read mapping phase within the mapModule f…

0dfd04e

…unction

fix: Address narrowing conversion and invalid conversion issues in co…

6075e16

…mputeMap.hpp

fix: Resolve narrowing conversion warnings in computeMap.hpp

dd7d5d0

fix: Resolve infinite loop issue in fragment mapping

297107f

refactor: implement thread pool for fragment workers and synchronize …

5851502

…segment mapping completion

feat: implement processFragment function to handle fragment processin…

6cdd1a8

…g in computeMap module

fix: Resolve variable scoping and declaration issues in computeMap.hpp

2d9889a

feat: Add new member variables to Map class

4f90de8

fix: Declare and use fragment_atomic_queue_t consistently

4da8e68

fix: Remove redundant declaration of fragment_atomic_queue_t and frag…

ff7efb3

…ment_queue

fix: Declare fragment_queue as a member variable of the Map class

93ff8e3

fix: Move delete statement to the end of processFragment function

2fa12c5

feat: Refactor shared variables in multi-threaded context

dcd8b02

chore: remove "processed fragment" debugging logging

fe8ff1a

fix: Update progress meter handling in InputSeqProgContainer

77c6a97

feat: Increment progress without locking mutex

f5d8147

feat: Restore granular progress updates in mapping module

da7a7ac

fix: Remove duplicate definition of InputSeqProgContainer

0f9120d

improvements to progress meter

9051cb1

avoid racing too hard

f715238

fix: Respect split parameter in mergeMappingsInRange and mapModule

aec3faf

use the correct increment for an atomic int

c569b46

feat: Reduce fragment queue wait time from 100 microseconds to 10 mil…

62e9047

…liseconds

ekg added 29 commits October 10, 2024 14:44

feat: Index combinedMappings by query sequence id

2e86c81

refactor: update aggregator_thread to use seqno_t as key type for agg…

655d9c3

…regatedMappings

fix: Update types in computeMap.hpp to resolve compilation errors

5f59fd1

fix: Update computeMap.hpp to use correct querySeqId variable

95150e3

compiles, self-mapping is broken

a46018f

feat: Implement flexible sequence grouping and self-mapping checks

a0e2f57

fix: Correct indentation of query_list and query_prefix fields in map…

5a8fd01

…_parameters.hpp

fix: Update SequenceIdManager constructor and populateFromFiles method

406f6ee

fix: correct SequenceIdManager constructor call in computeMap.hpp

88df62d

fix: update SequenceIdManager constructor arguments in computeMap.hpp

c6cc6ee

fix: Correct constructor call for SequenceIdManager

ca20a85

fix: Update SequenceIdManager constructor call in computeMap.hpp

4da8a12

compiles but still equivalently broken

0f4517d

clean up how we get ref group ids

5a87f3f

bad race conditions in winsketch, unfixed

9851b2f

unset new index build code due to race issues

2edce4d

feat: Implement sequence grouping logic based on prefixes and delimiters

64a2b47

corrects mapping sensitivity error

60b1f03

unbork the mappings

87e0bb1

remove unused aggregation data structure

12014e3

log which subset we in

4611dc5

rename sequenceidmanager file

95b3bb2

refactor: Extract target subset creation into a function

634860c

feat: Implement index-only mode for creating subset-specific index files

849adee

fix: Convert std::filesystem::path to string before concatenation

e9ab49f

refactor: Extract condition into its own function

d602127

create all the indexes

6fe2be0

feat: Add index file loading to Sketch constructor

f11f911

load subindexes from disk

abda8f3

ekg merged commit 1460a98 into main Oct 13, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Map chunk query #277

Map chunk query #277

ekg commented Sep 25, 2024

Map chunk query #277

Map chunk query #277

Conversation

ekg commented Sep 25, 2024