Highlighting OCaml attributed expressions?

[kenichi-asai]

I want to highlight OCaml code that contains attbirutes in a specific way. Given a program like this:

let a = 1 + ((if true then (2 + 3) * 4 else 5)[@x ])

I want:

let a = 1 + (if true then (2 + 3) * 4 else 5)

where the attribute [@x ] as well as the parentheses around it are removed and the attributed expression (if true then (2 + 3) * 4 else 5) is highlighted in a specific color. Since an attributed expression can contain arbitrarily large expressions with possibly matching parentheses, and it is impossible to detect matching parentheses using regular expressions, I assume it is impossible to do such highlighting using prism.js. Is this correct? (If there is any way, please let me know.) Thanks in advance.

[RunDevelopment]

I assume it is impossible to do such highlighting using prism.js. Is this correct?

Yes, and no. It's impossible to count arbitrarily nested expressions with JS regexes, that's correct. However, it is possible to count until a finite maximum depth (e.g. 4 to 8), which is typically good enough for syntax highlighting in practice. We do this trick in a few languages (e.g. C#).

However, this trick as a cost: large regular expressions. This can be a problem because:

• Browsers will just not execute (or not at least not correctly execute) regexes that are too big. I found that some regexes just stop working once you go 16-32 levels deep.
• Large and deeply nested regular expressions also make it easy for exponential backtracking to slip in, so be sure to keep an eye on that.

---

I also want to point out another you could do. Prism doesn't actually require regexes to do that text matching, it only requires something that behaves like a regex (see #2595). So you could write an OCamlAttribute class that implements the parentheses counting. The class just has to follow this interface:

interface RegExpLike {
  global: true
  lastIndex: number;
  exec(value: string): RegExpExecArray | null;
}

You can then use the class like a regex:

Prism.languages.ocaml = {
  // ...
  'attribute': {
    pattern: new OCamlAttribute(),
    inside: { /* ... */ }
  }
};

You can also implement this as a single function using the toRegExpLike function I suggested here.

However, there are downsides to this approach as well:

• You have to implement the text processing yourself. This isn't trivial.
• We don't currently use this in Prism. It's theoretically supported in all present and future versions of Prism, but it's not tested/used yet.

---

I hope this helps.

[kenichi-asai]

Thank you. That's helpful. Now that I know there is a way, I would try the second approach. Is there any introductory tutorial on how to write my own language definition, inheriting a pre-existing language definition? (I am completely new to prism.js.)

I found https://prismjs.com/extending.html, but it assumes a lot of basics, e.g., where is 'comment' defined, what color is assigned to it, what categories other than 'comment' available, how can I introduce a new category, how can I assign a regular expression to it. I found a link to API at the end of the above page, but it is hard for a beginner to obtain necessary information from there. If there is a simple worked example, that would also be great.

I also welcome a simple example on how to use the toRegExpLike function. Thanks in advance.

[RunDevelopment]

Before I start answering your questions: It's generally a good to understand one or two existing language definitions before writing your own. I recommend starting with JSON. It's a small and simple language. After that, C might be a good next step to see Prism.extend in action.

I also want to say: you're right. There's too little docs on how language definitions work. I'll add some more today.

Is there any introductory tutorial on how to write my own language definition, inheriting a pre-existing language definition?

Not really. Prism.extend just copies an existing language one-for-one, so it's not any different from defining those tokens yourself.

That being said, especially as a beginner, you don't have to use Prism.extend. You can also just literally copy another language definition and modify it as needed.

where is 'comment' defined
what color is assigned to it
what categories other than 'comment' available
how can I introduce a new category

comment, function, string, etc. are token names, but we typically just call them tokens when talking about the names specifically. Token names describe the semantic meaning of a piece of code. E.g. is this a comment, string, numeric literal, boolean, function?

The rules (token name + regex. e.g. 'comment': /\/\/.*/) of a language definition can have arbitrary token names. Those rules then generate tokens with the token name using the regex(es) of the rules.

So how does Prism assign colors to tokens then? Answer: it doesn't. Colors and other styles are assigned to tokens by themes, CSS stylesheets that define the looks of everything Prism produces. So how do themes assign colors to tokens then? Answer: using token name. Themes support a limited number of known token names and assign based on those names. Different themes might support different token names, but all themes are guaranteed to support Prism's standard tokens.

However, sometimes you might want to choose a more specific token name, one that isn't a standard token. You can still get colors for non-standard tokens by using a standard token as an alias (example).

a simple example on how to use the toRegExpLike function

Sorry, but everything regarding #2595 and toRegExpLike is still in its early stages. As I said, we currently don't use it anywhere, so I can't give you any real-world example. The only example I can point to is the XML example (right below the definition of toRegExpLike) I implemented to showcase toRegExpLike.

As I said, doing the text processing yourself isn't easy. toRegExpLike only makes sure you get the interface right, but it doesn't help you with the actual text processing. We will likely add simplified functionality on top of toRegExpLike in the future, but none is implemented yet.

Unfortunately, I can't give you an easy solution here. Matching non-regular patterns in code isn't an easy problem. We rarely have to deal with it for the purposes of Prism, so we don't have a good tool to handle it yet.

You might want to try out nested regexes (approach 1) before using toRegExpLike. They aren't prefect, but they are comparatively easy to implement, and we have examples of their usage in existing languages (C#, CSHTML, Q#; or even simpler: JS string interpolation).