[workstealer] at most once execution semantics

[aavarghese]

Right now, we have at least once task execution semantics. I.e. it is possible that the same task may be processed twice. This can happen due to the lockless implementation of the workstealer. There are race windows between worker deciding to work on a task and work stealer deciding to steal that task.

We can probably add at most once semantics without massive reengineering.

Prototype Idea

• worker observes a file in inbox. Today it will start working on it immediately. Proposal: instead, touch a {task}.trylock file. Wait until a {task}.trylock.ack file exists. If that object says "yes", then proceed; if it says "no", then skip
• workstealer observes {task}.trylock. if it hasn't yet stolen the work from that worker, it will ack with yes; otherwise ack with no
• workstealer only steals non-trylocked tasks