Parsing expressions inside of strings

[RossComputerGuy]

From my grammar definition:

  Parser<List<dynamic>> stringLexicalToken() =>
      (ref0(multiLineStringLexicalToken) | ref0(singleLineStringLexicalToken))
          .map((value) {
        print(value);
        return [value];
      });

  // TODO: escape for inner expression
  Parser<List<List<dynamic>>> multiLineStringLexicalToken() =>
      (string("''") & any().starLazy(string("''")).flatten() & string("''"))
          .map((value) => value[1]
              .split('\n')
              .map((v) => v.trim())
              .where((v) => v.isNotEmpty as bool)
              .map((v) => [v])
              .toList()
              .cast<List<dynamic>>());

  // TODO: escape for inner expression
  Parser<List<dynamic>> singleLineStringLexicalToken() => (char('"') &
                  ref0(stringContentDoubleQuotedLexicalToken).star().flatten() &
                  char('"') |
              pattern('^"\n\r').star() & char('"'))
          .map((value) => [value[1]]);

  Parser identifierExpressionLexicalToken() =>
      (ref1(token, '\${') & ref0(expression) & ref1(token, '}'))
          .labeled('identifierExpressionLexicalToken');

I need to get it to where the identifierExpressionLexicalToken is matched inside the strings. The idea is the resulting parser type would have a list of string or expression. An example of how this looks would be:

# Multi line
''
Hello, world
${variable}
``

# Single line
"Hello, world ${variable}"

However I also need to escape when ${ is prefixed by '' and reject it from parsing that. Reference from the language manual.

I've tried using ref0(identifierExpressionLexicalToken) | any() in multiLineStringLexicalToken but that doesn't seem to do the trick.

[renggli]

What you tried in the last paragraph should work.

The only catch I see is that you won't be able to use starLazy(string("''")), if you also want to sometimes consume '' inside the repeat. In this case I would suggest to use the normal star() repeat, and replace the last/catch-all any() with string("''").not() && any().

[RossComputerGuy]

Hmm, I'm trying that but I get parse failures:

  Parser<List<List<dynamic>>> multiLineStringLexicalToken() =>
      (string("''") & (ref0(identifierExpressionLexicalToken) | any()).star() & string("''"))
          .map((value) {
            print(value);
            return [value];
          });

''
      Hello, world
      ${1}
    ''
Failure[1:3]: "\"" expected

[RossComputerGuy]

I got the mutli line string to mostly compute to how I want it to now:

  Parser<List<List<dynamic>>> multiLineStringLexicalToken() => (string("''") &
          (ref0(identifierExpressionLexicalToken) | any())
              .starLazy(string("''")) &
          string("''"))
      .map((value) => value[1].fold(<List<dynamic>>[], (prev, item) {
            var value = prev;
            if (item is String) {
              final i = item.indexOf('\n');
              if (i > 0) {
                if (value.length == 0) {
                  value.add(<Object?>[item.substring(0, i)]);
                } else if (value.length > 0) {
                  final last = value[value.length - 1];
                  if (last[last.length - 1] is String) {
                    last[last.length - 1] += item.substring(0, i);
                  } else {
                    last.add(<Object?>[item.substring(0, i)]);
                  }
                }

                value.add([item.substring(i + 1)]);
              } else {
                if (value.length == 0) {
                  value.add(<Object?>[item]);
                } else {
                  final last = value[value.length - 1];
                  if (last[last.length - 1] is String) {
                    last[last.length - 1] += item;
                  } else {
                    last.add(item);
                  }
                }
              }
            } else {
              if (value.length == 0) {
                value.add([item]);
              } else {
                value[value.length - 1].add(item);
              }
            }
            return value;
          }));

''
      Hello, world
      ${1}
    ''
Success[4:7]: [[
      Hello, world, 1]]

It just seems to not retain the next stuff for newline.

[RossComputerGuy]

Still working on this problem:

Failure[1:1]: "_" expected

  // FIXME: handle ''${expr}
  Parser<List<List<dynamic>>> multiLineStringLexicalToken() =>
      (string("''").token() &
              ((string("''").not() & ref0(identifierExpressionLexicalToken)) | any())
                  .token()
                  .star() &
              string("''").token())
          .map((value) {
        final column = (value[0].column - value[2].column).abs();
        final line = value[0].line;
        print(value);
        return value[1].fold(<List<dynamic>>[], (prev, token) {
          var value = prev;

          final reline = token.line - line;
          final recol = token.column - column;

          while (value.length < reline) {
            value.add(<Object?>[]);
          }

          if (recol > 0) {
            if (token.value is String) {
              final newline = token.value.indexOf('\n');
              if (newline == -1) {
                final curr = value[reline - 1];
                if (curr.isNotEmpty && curr[curr.length - 1] is String) {
                  curr[curr.length - 1] += token.value;
                } else {
                  curr.add(token.value);
                }
              }
            } else {
              value[reline - 1].add(token.value);
            }
          }
          return value;
        });
      });

I'm not sure what I'm doing wrong.

[RossComputerGuy]

This ended up being the solution:

diff --git a/packages/nixietube_parser/lib/src/parser.dart b/packages/nixietube_parser/lib/src/parser.dart
index 5a033db..7220096 100644
--- a/packages/nixietube_parser/lib/src/parser.dart
+++ b/packages/nixietube_parser/lib/src/parser.dart
@@ -79,12 +79,13 @@ class NixParser extends GrammarDefinition {
           .labeled('stringLexicalToken')
           .map((value) => value);

-  // FIXME: handle ''${expr}
   Parser<List<List<dynamic>>> multiLineStringLexicalToken() =>
       (string("''").token() &
-              (ref0(identifierExpressionLexicalToken) | any())
+              ((string("''\${") & any()).map((value) => "\${${value[1]}") |
+                      ref0(identifierExpressionLexicalToken) |
+                      any())
                   .token()
-                  .starLazy(string("''")) &
+                  .starLazy(string("''\${").not() & string("''")) &
               string("''").token())
           .map((value) => value[1].fold(<List<dynamic>>[], (prev, token) {
                 var list = prev;