Some questions about Unicode aliases

[rljacobson]

I am taking a look at named-characters.yml at @rocky's request, and it occurs to me that there are some philosophical questions that should be answered about which Unicode symbols should be included and how. It appears Mathics has been relatively conservative with respect to adding Unicode aliases, so I think this discussion is really about making additions to what you already have from here on, not really about removing existing symbols.

Broadly speaking, I propose the following heuristics:

1. Unicode symbols used by Mathematica should be used in the same way by Mathics for the sake of compatibility.
2. Unicode symbols that correspond semantically with existing mathematical symbols should be included. Example: − (U+2212, "Minus Sign") should be an alias for ASCII - even though Mathematica does not consider it so.
3. Unicode symbols outside of the Mathematical Operators Block (and the ASCII block) should be excluded unless one of the previous heuristics includes it. Example: ✕ (U+2715, "Multiplication X") can be used for Times but is in the Dingbats Block and is thus excluded.
4. All typographical variants of "plain"/"regular" symbols should be excluded unless included by a previous heuristic. For example, all Full Width variants, bold variants, italic variants, and so forth are excluded.
5. Unicode symbols should not be overloaded, i.e. should not be used for more than one underlying function, unless required for Mathematica compatibility. For example, ≫ (U+226B, "Much Greater-Than") is already used for GreaterGreater and therefore should not be an alias for >> for Put. Likewise, ≪ (U+226A, "Much Less-Than") for Get, ∷ (U+2237, "Proportion") for MessageName, etc.

The general idea is to continue to be conservative while also covering compatibility and use cases we are reasonably likely to encounter. I also argue that having these heuristics written down somewhere is helpful for future contributors, whether future us or someone else, for a variety of reasons.

These heuristics do not cover all cases worthy of discussion. Here are two cases where it's not clear whether the Unicode aliases should be included:

• ≔ (U+2254, "Colon Equals") for SetDelayed. This feels to me to be, while not exact, a close enough semantic correspondence to include it under heuristic 2.
• ⋙ (U+22D9, "Very Much Greater-Than") for PutAppend. This would be awkward to include considering ≫ (U+226B, "Much Greater-Than") cannot be an alias for Put (by heuristic 5).

This issue is to solicit:

1. Discussion on the heuristics themselves?
2. Discussion on recording them somewhere?
3. Opinions about my two specific symbols ≔ and ⋙?

[rljacobson]

An additional third symbol to consider:

• ‼ (U+203C, "Double Exclamation Mark") for Factorial2. It is in the General Punctuation, not the Mathematical Operators block, but ! is also not in the Mathematical Operators block.

[mmatera]

@rljacobson Thanks for the comments and the information!

In principle, I agree with the heuristic that you are proposing for assigning meaning to Unicode characters.
I also agree that Put (>>) and PutAppend(>>>) should not be represented by the characters ≫ and ⋙ mainly because the semantics is different.
In any case, I think that we need to check if these changes/inclusions do not affect the usability of the front-ends, and the interoperability with WMA and other languages (let's say, the ability to translate an expression into LaTeX or a Python expression, or to copy and paste expressions from the CLI).

Another thing we should consider is that we still do not have Input and Output Forms properly implemented in Mathics. The plan is to do that in a not too far future, and at that point, we have to see which representation of these symbols we should choose for these Forms.

Finally, regarding where to record the discussion, assuming @rocky also agrees, I think it would be enough to add an entry in the documentation stating these criteria. If you are up for it, please rewrite the issue in the format of a documentation entry and make a pull request adding that file into mathics-scanner/doc.

[rljacobson]

What kind of test suites do you have that I should pay particular attention to?

For that matter, what test suites does Mathics have in general?

Cory Walker has a nice corpus of tests and documentation for his Wolfram Language implementation expreduce. His documentation has good coverage but is spartan, but I think it would nonetheless be a good starting point for language documentation that does not yet exist.

I have advocated for independent documentation and test suite projects that Wolfram Language projects can pull in as desired, but that hasn't really materialized.

[rocky]

What kind of test suites do you have that I should pay particular attention to?

Look in https://github.com/Mathics3/mathics-scanner/tree/master/test

For that matter, what test suites does Mathics have in general?

A broad and basic question like this gives me an impression that you haven't spent much time looking at the code or trying to answer this question first. Generally, I try to do a little investigation before I post a general question. (Also, this really is not about mathics-scanner.)

However in other projects, as with mathics-scanner, you may find a test directory which has tests. The GNU makefile has "check" targets that run tests. Most of the time we use github workflow actions so looking at the actions tab shows the tests that were run and the corresponding .github.workflows directory. Also please see the Mathics3 Developer Guide and the section on running tests.

Cory Walker has a nice corpus of tests and documentation for his Wolfram Language implementation expreduce. His documentation has good coverage but is spartan, but I think it would nonetheless be a good starting point for language documentation that does not yet exist.

I was not aware of expreduce, it seems interesting — Thanks! I am not sure exactly what language documentation or coverage you mean specifically. Feel free to put in Pull Requests from this as you see fit.

I have advocated for independent documentation and test suite projects that Wolfram Language projects can pull in as desired, but that hasn't really materialized.

Why don't you can just start such github repository yourself? If after you have something concrete, we can try to hook it into Mathics3.

[rljacobson]

A broad and basic question like this gives me an impression that you haven't spent much time looking at the code or trying to answer this question first. Generally, I try to do a little investigation before I post a general question. (Also, this really is not about mathics-scanner.)

No, I haven't looked at Mathics's codebase in any significant detail. Given that you had only asked me to look at operators.yml and named-characters.yml, I'd say I have gone above and beyond, wouldn't you?

[rocky]

I was not suggesting that you look at the code base in significant detail, but just that when you have a question of a very general nature, to spend a little bit of time to see if you can answer the question on your own. If then it isn't answered, by all means, ask.

I just googled "Does mathics3 have any tests" and got back https://github.com/Mathics3/mathics-core/wiki/Documentation-and-tests.

Thanks.

[rocky]

When it comes to Unicode characters used in Mathics3 and the specification in named-characters.yml, some things at the outset to keep in mind...

We have separated and isolated character symbols (to a large extent) from the rest of Mathics3. Therefore, front ends, (mathicssscript, the mathics script that comes with Mathics-core and Django) control what is acceptable.

In the case of mathicsscript, mma-tables.json, gives a list of the operators, and their translation to Mathics3 operator names. The escape sequences for inserting a Unicode character are in files inputrc-unicode and inputrc-no-unicode.
Users can control and customize what translations they want to use.

And this is in line with the open-source philosophy: if there is something you don't like about the code, you can always go and change it. Good open-source software is transparent in its operation and tries to make it easier to alter it, especially in ways that might have controversy.

That said, sure, we should try to provide something that is reasonable and does not require a lot of tweaking.

But we don't have to get everything exactly right initially. The community will complain when something is egregiously wrong. And in contrast to Wolfram, we are not the ultimate gatekeepers of truth.

Broadly speaking, I propose the following heuristics:
...
I also argue that having these heuristics written down somewhere is helpful for future contributors, whether future us or someone else, for a variety of reasons.

As best as I can tell, the existing tables follow these. I will put in a PR to adapt the heuristics, changing the "should" to "are" ; you can review the PR.

There are tests we have over the tables which may cover some of these. And if there are more complete tests, by all means add them.

≔ (U+2254, "Colon Equals") for SetDelayed. This feels to me to be, while not exact, a close enough semantic correspondence to include it under heuristic 2.
⋙ (U+22D9, "Very Much Greater-Than") for PutAppend. This would be awkward to include considering ≫ (U+226B, "Much Greater-Than") cannot be an alias for Put (by heuristic 5).

As far as I can tell, neither of these is dictated by Mathematica so it is not wrong to omit these. However, if a user wants to allow the mappings, the user is free to do so, with the understanding that it might also mess up others try to use the code but don't have such mappings.