Prepping for first pilot of semantic parts experiment

[judithfan]

• Validate that the spline extraction works well. Benchmark the latency for doing this on the fly with sketches containing a large number of strokes. If low, let's do it client-side. If higher, let's precompute.
• Add "basic-level" class label in large font above the canvas, all caps.
• Convert the part menu color scheme to interpolate between two RGB endpoints, where every intervening hue is highly distinct against white background.
• Gray--> Dark Gray --> Black progression when selecting splines in the canvas, reserving color for semantic labels only.
• Add consent form and instructions (actually, just port over and edit the forms directory in related experiment: https://github.com/judithfan/graphical_conventions/tree/master/experiments/draw_chairs/forms). Include a "practice trial" where the participant has to correctly annotate the smiley face in order to proceed to the full task. This is our comprehension check quiz.
• Have a way of handling the "straggler" strokes that seem to not be easily selectable in some sketches. One way is to bump down the z-level of each svg element once it is selected.
• Include an "Do not know" option in the part menu for strokes that are unidentifiable.
• Extend the current trial data JSON object that is being sent to the database to include everything relevant that we are saving in other similar experiments: HITId, WorkerId, AssignmentId, timestamp, various metadata from the source sketches to be able to match them back up again, color of each spline, trialNum, dbName, colName, iterationName from source dataset.
• Clear the text field in the free-response window between each free-response annotation.
• When re-labeling a stroke, only highlight the previously selected label (and not all labels so far).
• Advance to next sketch only upon clicking "Next Sketch" button.
• Task layout adjustments: Shift the canvas and part menu down about 50% of the height of the canvas. Place class label in black font above the canvas. Move the "Next Sketch" button to just below the canvas.
• Display "Congrats! You're all done." message to participant when the task is over.
• Make sure that the data we're excluding from annotation really the come from the trials where we had the shift-key artifact. Right now, we are excluding sketches with numStrokes > mu + 3*sd, which is an imperfect proxy for that.
• Fix issue with browser window being much larger than necessary when in the annotation environment (so the images don't show up in the center).
• Make sure that element positioning/scaling (maybe with the exception of the sketch canvas) be defined in terms of % of the browser dimensions, rather than in raw pixels, which can vary a lot across different display resolutions.
• We want the part menu to be right next to the sketch canvas to minimize the motor cost of going back and forth.
• Let us always display the class label in lower case.
• Let us use a "modern cool looking font." One possibility (but there may be others out there) is Lato: https://fonts.google.com/specimen/Lato
• Only display the "Are you sure you want to continue?" popup iff not all the splines have been tagged. If they completed tagging all of them, just let them continue.
• What does "Unknown" mean? Maybe the text in the part menu can be "I can't tell!" but it maps to --> "Unknown" when sent to the database.
• We need to add a cumulative bonus meter (in numerical form) in one of the corners of the display to tell people exactly how much bonus they've earned so far. We're planning to pay them for each tagged spline AND a separate bonus for completing each sketch (100% of the splines).
• A symbolic progress meter that displays: "Sketch 5 of 10."
• A cheesy animation (shooting stars) when the participant finishes annotating 100% of the splines in a given sketch so that it is maximally visually + financially rewarding to complete a sketch.
• Instructions: allow to go backwards as well as forwards.
• They shouldn't be able to advance to the game if they don't complete the practice trial. They can maybe get some "wrong" but they can't skip it.
• Instructions, page 1:

PAGE 1:
In this HIT you will play a fun game where you will see some sketches and tell us what you see!

Each sketch was made by somebody who was playing a Pictionary-style game, in which they had to make a sketch of a target object (outlined in red) so that someone else could tell which object in the lineup they were trying to draw. 

Your goal is to label each of the strokes in these sketches by selecting parts from a menu. 

[Include the sketch+context image below, without the annotation bells/whistles.]

PAGE 2:
Let's walk through an example! See the diagram below:
[Annotated version of the task interface.]

PAGE 3: 
Here's how you label a sketch! 

1. You can click on a stroke or "paint over" it by dragging your cursor over several strokes. Once it's selected, it will turn gray.
2. You then click on one of the labels in the righthand menu to label it. Once it's labeled, it will turn into the same color as the word in the menu.
3. If the label you think is most appropriate for the stroke is not already in the menu, please click "Other" to provide your own label. 
4. If you change your mind about a stroke you already labeled, just click that stroke again, and select a new label from the menu. 
5. Repeat steps 1–4 until you have labeled all the strokes in the sketch.

[Embed one modestly-sized graphic for each step above.]

Note: To make your life easier, as you label more strokes in each sketch, the remaining ones will get thicker so it is easier to see them. 

PAGE 4: 

Okay, let's practice!

[embed one practice trial]

PAGE 5:

You will receive a "stroke bonus" for every stroke you label AND a special "completion bonus" for every sketch that you finish labeling. 

For each sketch, you are allowed to continue to the next sketch if you haven't finished labeling every stroke, but then you will miss out on the completion bonus. So please do your best to label every stroke you see!

We expect the average amount of time required to complete the HIT to be around 10 minutes, including the time it takes to read the instructions.

If you encounter a problem or error, send us an email ([email protected]) and we will make sure you're compensated for your time! Please pay attention and do your best! Thank you!

Note: We recommend using Chrome. We have not tested this HIT in other browsers.

• Finalize the HIT length and payment scheme so that it is feasible... Judy needs to think more about this.