Skip to content

peteonrails/threadedcollections

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Threaded Collections is a package for iterating through collections 
over multiple threads. With large collections, sometimes it can be 
more efficient to process a collection in parallel, provided that the 
collected items don't have a interdependencies, or need to be processed
in a specific order. 

Usage: 

require "threaded_collections"

threadcount = 2
arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
tps = ThreadedCollectionProcessor.new(arr)
tps.process(2) { |thread_id, item| puts "Thread #{thread_id} processed item: #{item}" }

I abstracted this pattern from a web services client that posted items from a collection, but
each request took a second to process. The remote service had plenty of threads available, so
I parallelized the task with this pattern. 

THIS IS AN ALPHA RELEASE. 

I have no plans to break the interface but I do plan to make two major enhancements: 

1. Make it possible to mix this functionality in to the Ruby iterators, so you don't have create the 
ThreadedCollectionProcessor.
2. Make it work with fibers and processes in addition to threads. 

Note: Don't implement the Observer pattern inside the block! 

Submit bugs, enhancement requests, and other issues through Lighthouse: 

http://peteonrails.lighthouseapp.com/projects/11058-threaded-collections/overview

About

Gem for processing collections over a number of threads.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages