Template strings support in JS lexer

[everyonesdesign]

It seems that ECMAScript2015's Tempate strings breaks Babel's Javascript parsing.

That's sad, because it's supported quite well in latest browsers.

It would be great if Babel supported this feature.

[akx]

Entirely possible to implement!

There are PRs in the pipeline to improve JSX and dotted name parsing (#310 and #298 respectively).

However, a question:

Are you planning to use template strings in conjunction with translation, like

gettext(`Hi, ${name}!`)
// or to use the tag syntax,
i18n `Hi, ${name}!` 

or would you just like to see parser support for plain template strings?

[everyonesdesign]

Well, I planned to use this way:

new ScrollUp({
    text: `▲<br><span> ${ _("back to top") } </span>`
});

But, actually, it's not the best idea to put markup directly to strings.
After checking out your examples I see that the issue is not so easy to implement due to multiple usage ways.
Anyway, it would be great if there was a best practice or a recommendation on its usage and we could stick to it.

[akx]

Ah, that (parsing the ${ } expressions within templates) should also be possible to implement. I didn't think about that :)

(Mental note for implementation: Namely, before emitting a template string, find ${..} sections within it, "recursively" re-parse them as source and emit the tokens from within too.)

But sure, this definitely looks like something to implement along with the other JavaScript extraction improvements! :)

[akx]

@everyonesdesign If you're around, could you do code review on #332? :)

[everyonesdesign]

@akx
I'm not so good at Python, so I looked mostly at the tests changes in the MR. I don't see tests for the recursive template strings parse trick you mentioned in your comment. Is it planned to be implemented?

Anyway, new usecases of the translations in tests look really good, thanks for that great job.

[akx]

@everyonesdesign Yes, that support is not (yet) implemented; it could be done as further development . I just wanted to get some template string support in. :)

[johbo]

Just ran into this problem as well, taking a look if I can find out how to improve the situation.

[johbo]

Hmm, found out that my issues were caused by not recognizing the jsx tags themselves, the change in #396 gets it good enough for my need.

While looking at this topic, I noticed that template strings can be nested:

`${condition ? `${varA}` : `${varB}`}`

Although this seems to produce odd results, it's probably not going to harm since the backticks come in pairs.

[petterm]

I found this issue while troubleshooting extracting strings like the one below. Any updates on this?

const text = `${variable} - ${gettext('some translated text')}`;

[fdanielsen]

Any updates on this? PR #332 references this as something included in that merged improvement, but this issue is still open and in my testing extraction from template strings does not work.

Would be really nice to have support for this. We ended up with an untranslated string in production due to unawareness of this issue with extraction, and had to somewhat awkwardly restructure some code to ensure the gettext calls were outside template strings to make sure they got extracted.

[akx]

@fdanielsen Sorry to hear about the trouble! If I may inquire, how are you using template strings for translation at the moment?

[fdanielsen]

@akx Thanks for the quick response. In this case I was building up a string with some minor HTML formatting in addition to a translatable string:

const description = `<em>${gettext('…')}</em>`;

I've now changed it to simply extract the gettext call to a separate variable declaration that then gets interpolated in the template string. But would be nice to avoid that. And since we use Prettier which tends to reformat these string concatenations it would be nice to avoid having to remember this "gotcha" with gettext calls.

[gitaarik]

I've had issues with this for a while, so I decided to hack around. I got the JS template expressions working. These:

`${gettext('my text')}`

And it also works with nested expressions like:

${`some text ${gettext('my text')} more text`}

I created a fork. You can try it out like this:

pip uninstall babel
pip install setuptools
pip install https://github.com/gitaarik/babel/archive/javascript-improvements.zip

You have to enable template string extraction with an option in the option_map:

extract_from_dir(
    # ...
    options_map={
        '**': {'parse_template_string': True}
    }
)

Then it should work.

On this branch I also fixed another issue that I was facing. In a JS library that I use, there was regex with a / inside a character class [a-z]. The babel JS lexer thought this closed the regex, but JavaScript allows this as a normal match character within a character class, and does not close the regex. I updated the regex so that it will cover this exception.

Here you can see the diff of my fork:

https://github.com/python-babel/babel/compare/master...gitaarik:javascript-improvements?expand=1

It works for me, and I'm very happy about it. Because now I can extract the messages from my es6 files instead of es5. That goes a lot faster, because of less clutter from polyfills in es5 files.

But I'm not sure what needs to be done to merge this in the babel master. I don't know how to run the tests and if there needs to be unit tests added and how to etc. Are there people that can help me with this? Maybe @akx ?

Edit:

I created a PR now and all automatic checks seem to pass: #792