fix: token propagation when using a custom scanner (#51)

Hi.

I just noticed that the Token.template file in the Std directory uses
an old representation of how token names get propagated into a
Token.java file when using plcc.py with the '!pattern=' flag in the
input file.

The '!pattern=' flag is typically used when writing a compiler and not
a PLCC-generated interpreter. In this case, the user must supply a
stand-alone scanner (Scan.java) instead of using the Scan.java found in
the Std library directory. Such a Scan.java file must still implement
the IScan interface, so its 'cur' method must still return a Token
object.

Token.template (instead of Token.pattern) in the Std library is used to
generate Token.java when PLCC is flagged to use a hand-crafted scanner
other than the Scan.java in the Std directory. Note that this only
applies in the case where PLCC is used to create a compiler instead of
an interpreter.

I have uploaded Token.template into the plcc.pithon.net directory. This
will *not* have any effect on any of the examples in the Code
directory, since none of these examples use the '!pattern=' flag.

Tim

Co-authored-by: Timothy Fossum <[email protected]>

src/Std/Token.template

@@ -1,43 +1,44 @@
 import java.util.*;
 
-// Token class without patterns
+// bare Token class for use with hand-crafted scanners
 public class Token {
 
-    public enum Val {
-%%Vals%%
-    }
+    public enum Match {
+%%Match%%
+        }
 
-    public Val val;          // the token value
-    public String str;       // the token string matched
-    public int lno;          // the line number where the token occurred
-    public StringBuffer buf; // used to collect the token's lexeme
+    public Match match;      // token match
+    public String str;       // this token's lexeme (never empty!)
+    // the following two fields may be unused
+    public int lno;          // the line number where this token was found
+    public String line;      // the text line where this token appears
 
     public Token() {
-        val = null;
+        match = null;
         str = null;
         lno = 0;
-        buf = new StringBuffer();
+        line = null;
     }
 
-    public Token(Val val, String str, int lno) {
-        this.val = val;
+    public Token(Match match, String str, int lno, String line) {
+        this.match = match;
         this.str = str;
         this.lno = lno;
+        this.line = line;
     }
 
-    public Token(Val val, String str) {
-        this(val, str, 0);
+    public Token(Match match, String str) {
+        this(match, str, 0, null);
     }
 
     public String toString() {
-        if (str == null)
-            str = buf.toString();
         return str;
     }
 
     public static void main(String [] args) {
-        for (Val v : Val.values())
-            System.out.println(v);
+        for (Match match: Match.values()) {
+            System.out.println( String.format("%s", match));
+        }
     }
 
 //Token//