Program repair

[Felienne]

In Hedy there are many opportunities to repair programs instead of giving error messages. In general, program repair is really hard and might not be feasible, but because Hedy programs are small it might be. Also, because kids use Hedy to learn, program repair can be extra useful not to fix the program but to show the kids how to fix it.

Some ideas for program repair are:

1. We now sometimes have suggestions (like: "did you mean print" or "remove that space" or "you forgot quotes") but we could envision actually fixing the program with the suggestion and running it.
2. When printing an unquoted string, we could check if there is a variable that is similar. This alone is a valuable better error message in itself, but could also be used to fix a program.
3. When detecting half quoted text (beginning or end), try to add quotes and run again
4. When detecting code at the wrong level, "upgrade"
5. When detecting two variable names with different casing fix the program and let the kids know they would not do that.

Any other ideas?

[Felienne]

Let's dive into direction 1 a bit more here. Some Hedy exceptions are already really close to being repaired, these are "low hanging fruit".

Invalid Space

For invalid space, we even already have a (very simple!) repair function:

https://github.com/Felienne/hedy/blob/731331ccfd0b810fa3934df8ad07637f0daf9b67/hedy.py#L1270-L1275

So the easiest first step is toch take this repaired program to the frond end and present it to the user.

[Felienne]

Invalid

For the Invalid Exception, we have a suggested keyword that we use in the error message, that could also be repaired:

https://github.com/Felienne/hedy/blob/731331ccfd0b810fa3934df8ad07637f0daf9b67/hedy.py#L1280-L1285

[Felienne]

This idea, also low-hanging fruit, is a first implementation of direction 2.

Unquoted Text

For the unquoted text exception in levels 3 and up, we do not know if the text missed quotes or that it is a misspelled variable name (both are named in the error message). A bit harder than the two above, but we could do some string distance algorithm to see if we have a variable nam in the lookup list that is close and fix that. A simpler version of this idea could first improve the error message using that information.

[Felienne]

Thinking about this a bit more, a proper implementation of the string distance function would also address direction 5 about the casing.

[MarleenGilsing]

Not about the programming, but about the lay-out:
I don't know what this is going to look like yet, but I think we should really keep in mind that we want the students to learn from their mistakes. So we shouldn't make a button that 'magically' fixes their mistakes without them having to look at it first, but one that takes them to their own mistake (e.g. highlights it) and then shows the students how they could fix the mistakes themselves first. Like the spelling check in word for example.

[Felienne]

Not about the programming, but about the lay-out: I don't know what this is going to look like yet, but I think we should really keep in mind that we want the students to learn from their mistakes. So we shouldn't make a button that 'magically' fixes their mistakes without them having to look at it first, but one that takes them to their own mistake (e.g. highlights it)

I think as a design principle this is what we are aiming for! A system that shows the mistake and how to fix it, rather than a magic fix.
Although we might want to fall back to the magic fix after a few consecutive errors to not discourage the kids (this could become a separate discussion when program repair is more ready)

Like the spelling check in Word for example.

I like the spell check in word idea! This is also how programming editors how issues in some cases, with a "squiggly underline" so kids also get used to grown-up IDEs that way.

I am not sure how hard this would be in Ace, but I am would guess that they support something like that!

[bjorn3]

I am not sure how hard this would be in Ace, but I am would guess that they support something like that!

The .setAnnotations() method can do this I think.

[bjorn3]

One question would be how to handle error squiggles when editing the code. IDEs normally run the parser locally and often use incrementallity to reduce the latency between typing and presenting errors. In case of hedy the parser runs on the server, which has a higher latency. We also may not want to submit the code after every keystroke to save server resources. There are three ways I can think of to handle this:

• Rewrite the parser in javascript and run it in the browser (hard)
• Move errors around when typing and clear when typing in the middle of the part of the code containing the error. (medium, outdated error messages may be confusing)
• Clear all errors as soon as the user types anything. (easy, hides context that the user may need to fix the actual error. maybe show the error message below the error together with the original line containing the error?)

[Felienne]

• Rewrite the parser in javascript and run it in the browser (hard)
• Move errors around when typing and clear when typing in the middle of the part of the code containing the error. (medium, outdated error messages may be confusing)
• Clear all errors as soon as the user types anything. (easy, hides context that the user may need to fix the actual error. maybe show the error message below the error together with the original line containing the error?)

Good analysis of options @bjorn3! I would say number 1 is not so much hard as it is much work. I think for now a good baseline would just be to ad the squiggles when we transpile as a kid hits the button and indeed remove them when the kid types again. I like the idea of keeping the message in view somehow so it can still be used as input/context.

[bjorn3]

#831 (comment) (by @Jasper1408):

How will the programme repair work exactly? What would we like the user to do?

Some suggestions made were the creation of a pop-up. Would this pop-up allow the user to copy the suggestion made, or will they have to retype it? Would the differences be highlighted between the suggestion and the faulty programme?

[Felienne]

#831 (comment) (by @Jasper1408):

Some suggestions made were the creation of a pop-up. Would this pop-up allow the user to copy the suggestion made, or will they have to retype it? Would the differences be highlighted between the suggestion and the faulty programme?

I think @MarleenGilsing had some great thoughts about this earlier: #772 (comment)

So definitely not a copy-pasteable suggestion but something the kid has to change or at the very least accept actively (like in Word). We can also try different versions and a/b test (we have infra ready to run a/b tests on the site)

[fRedelaar]

As a team, we have brainstormed and came up with some ideas for how the program repair could look like. We would like to present 5 ideas. 4 ideas came from our team members, and the 5th idea is a combination of the 4 ideas. We as a group think idea 5 might be the best. We have created videos so the explanation becomes more visual. Please note that this is just meant as a visual aid, and not 100% the final concept. Please pause or replay the video if it goes too quickly.

If you are in a hurry, please scroll down and only watch/read UX idea 5 since it is our combined idea. We would like to propose this idea and would love get feedback.

Hedy Program Repair UX idea 1:
Video 1: 27 seconds
Whenever a child makes a mistake, a hint button appears. It does not appear if the code runs successfully. Once the "Hint" button has appeared, a child can click on it. A modal pops up asking whether someone is sure he/she/they want to get a hint. If "No" is clicked, the modal disappears. If "Yes" is clicked, a new modal appears. The new modal is an animation of the original code and shows a cursor slowly going to the line and location where the code needs to be fixed. Then, the animation shows the fix. In the video, the fix was a single "'" before "hallo wereld". After the animation, the modal disappears by itself. The child has to remember what the fix was, and type the fix in its own code editor.

Hedy Program Repair UX idea 2:
Video 2: 23 seconds

Whenever a child makes a mistake, a hint button appears. It does not appear if the code runs successfully. Once the Hint button has appeared, a child can click on it. A new modal pops-up, showing the original code made by the user on the left and the repaired code on the right. There are 3 ways to highlight the mistake on the repaired side. A line could be highlighted, a word could be underlined with a squiggly line, or there could be no highlighting at all. In the third scenario a child has to make more effort to find the mistake. It could also be an idea to do the third scenario at higher levels. Afterwards the modal disappears by itself and the child has to remember what the fix was, and type the fix in its own code editor.

Hedy Program Repair UX idea 3:
Video 3: 33 seconds

This idea is more focused on gradual feedback, and giving positive feedback to children. The reason behind this is because children get a big red notification box when things go wrong, but don't get a green positive box if things went well.

Whenever a child makes a mistake, the error message says "We couldn't read your code well, do you understand why?" A big Yes" and "No" button appear. If the user fixes the code successfully and clicks on either "Yes" or "Voer de code uit", confetti will pop up the screen to celebrate the child solving its own mistakes. A credit system could be introduced here, but we as a group think it might overcomplicate the program repair aspect.

Scenario 2 is when the child did not know how to fix their mistake. In this scenario the child clicks "No", an error message comes up giving a hint such as "you made a spelling mistake". If the child still doesn't know how to fix their mistake and clicks on either "Run" or "Hint", another error comes up with the regular error that would pop up normally such as "You wrote Prin, did you mean Print?". If the child still doesn't know, the "Hint" button will automatically repair the code for the child in the editor.

Hedy Program Repair UX idea 4:
Video 4: 24 seconds

This idea is also more focused on gradual feedback. Whenever a child makes a mistake, the error message says "We couldn't read your code well". Here there are two scenarios. The first: A child could type in its editor, and the error will fade away automatically to increase the UX. If after typing and rerunning the code the program fails, then another error comes up with a more direct hint (in blue).

The second scenario is that the child doesn't know how to fix it, and clicks on the lightbulb icon for a "Hint". The direct hint in blue will appear. If the child still doesn't know, clicking on the "Hint" button for a second time will repair its code automatically.

Combined Hedy Program UX idea 5
Video 5: 34 seconds
Scenario 1: A child programs code and runs it successfully. Confetti will show up, and a green notification with positive text such as "Great job!", "Awesome!", "Well done!".

Scenario 2: A child made a mistake. The light bulb icon will appear if Hedy is able to repair the code. Once this is clicked, a modal pops up asking whether someone is sure he/she/they want to get a hint. If "No" is clicked, the modal disappears. If "Yes" is clicked, a new modal appears. The new modal is an animation of the original code and shows a cursor slowly going to the line and location where the code needs to be fixed. Then, the animation shows the fix. In the video, the fix was a single "'" before "hallo wereld". After the animation, the modal disappears by itself. The child has to remember what the fix was, and type the fix in its own code editor.

I am not an educational expert at all(!), but I do think it would be good to let kids fix the mistakes themselves instead of just having a magic button like @MarleenGilsing said too.

[Felienne]

Wow what a lovely mock-ups!!! I love the little clouds with the appearing button (not sure you want to build those in the UI too but esp. on the first appearance I think that make a lot of sense) and the confetti!!

I think my favorite is 1 (or 5) with the popping-up modal that shows the change. But of course, I want to know what @MarleenGilsing thinks!

[MarleenGilsing]

That looks great! I love how you can clearly and slowly see how the mistake should be fixed in idea 5.
I wonder what happens if you make multiple mistakes in a code. Does it just show you to fix one mistake at a time? Or does it show you all mistakes at once?

I really like the confetti too and i bet the kids would love it. But i do think it might become annoying after a while, so maybe we can change it to something clickable instead. For example, if you program a correct code a party popper emoji shows up and if you want to you can click it to fire the confetti, but you could also leave it.
Also, maybe we can use it to prompt kids to make their own code. So if you just run the example code you don't get the confetti, but if you run your own you do.

[fRedelaar]

Thank you both for the feedback! We as a group will soon start with working on idea 1 (or 5).

@MarleenGilsing 's suggestion about confetti becoming annoying after a while makes perfect sense. The solution to getting confetti by clicking on a party popper seems to be great. I have made a Proof of Concept for this idea in the draft pull request #1175. Here a green error pops up if the code runs successfully, with a text saying "Good job!" and a party popper emoji. If the party popper emoji is clicked, some confetti will show up.

Hedy_POC_Confetti.mov

Is this concept what you meant @MarleenGilsing ? (The confetti can be made smaller/prettier/slower or so. We could even edit the colors, or make emojis as confetti! See this demo for example: demo.)