Add support for keeping single line comments from typescript source code

[adamuso]

Add support for generating single line comments in gdscript from original typescript code. Multiline comments are still not supported and will be stripped out. The order and generation of comments depends on code in parse_node/ directory. If some comments are out of place or not showing up then we will have to adjust parsing in the future.

I think it closes #29 for now.

Example code:

// This is a test class
// abc
class Test extends Area2D {
    // This a constructor
    constructor() {
      // This is a super call
  
    }
  
    // this is a method
    method() {
      // this is a print
      print("a")
  
      // This is an if statement
      if (2 == 3) {
        // This is an empty block
      }
  
      print(/* this is block comment */"l")
  
      print(
        // this is a call with parameters in multiple lines
        "hello",
        // this is second parameter
        "world"
      )
  
      // this is a print
      // this is a print
      // and this is third comment
      print("b")
  
      print("x") // trailing comment
    }
  }

Generates:

# This file has been autogenerated by ts2gd. DO NOT EDIT!

extends Area2D
class_name Test

# This is a test class
# abc
# This a constructor
func _ready(): 
  pass

# this is a method
func method():
  # this is a print
  print("a")
  
  # This is an if statement
  if ((typeof(2) == typeof(3)) and (2 == 3)):
    pass
  
  print("l")
  print(
  # this is a call with parameters in multiple lines
  "hello", 
  # this is second parameter
  "world")
  
  # this is a print
  # this is a print
  # and this is third comment
  print("b")
  print("x") # trailing comment

[johnfn]

It's extremely cool that we can do inline comments without pulling them out into the previous statement... but does it work in the even more general case? I mean, TS can handle all sorts of weird stuff, like:

let foo = { bar: 1 }
console.log(foo. /* Hello */ bar)

That's actually valid and it does indeed print 1!

I feel like Godot would choke on that if we did it this way! Or would it?? Why don't we toss a few of these "weirder" cases into our test suite and test the codegen in Godot? If they all pass, that would be awesome, but if not, we could pull out inline comments into the previous line, e.g. the codegen for the above would be

var foo = { "bar" : 1 }
# Hello
console.log(foo.bar)

[johnfn]

If you do decide to go by the per-line route, I think the proper place to inject the comments would be here:

https://github.com/johnfn/ts2gd/blob/main/parse_node.ts#L195

Basically all code in ts2gd flows through combine so this is a pretty safe place to put it. We have this notion of beforeLines and afterLines that allow you to say "emit this code as a standalone line before/after the line currently being codegened" so we could use that for comments as well.

[adamuso]

It's extremely cool that we can do inline comments without pulling them out into the previous statement... but does it work in the even more general case? I mean, TS can handle all sorts of weird stuff, like:

I'm a bit confused because this example print(/* this is block comment */"l") shows that all block/multiline /* ... */ comments are currently strpped out. Did I misunderstand something?

I didn't implement block comments because I do not really know how to handle them and I think they can be separated into another PR. In this PR I wanted to create base algorithm for comment parsing and only focus on single line comments :)

[johnfn]

Oh - good point on the block comments, I glossed over that. However, what about something like this?

let foo = { bar: 1 }
console.log(foo. 
  // Hello
bar)

[johnfn]

I realize that specifically may seem like an obscure case, but I think the thing that I'm more generally worried about is that there are possibly a lot of cases like this: you can really put comments almost anywhere in JS/TS, but if you map that 1:1 to Godot, there might be a lot of weird cases that don't compile correctly.

[johnfn]

I believe you that we need to dedupe them, but why? Does TS give us back the same comment multiple times? When and why?

It might help to leave a comment about this.

[adamuso]

The problem is that multiple nodes can give the same comment when they are in the same line. For example:

// some comment
myFunc(1 + 2, someVar)

This code generates AST like: CallExpression (params: BinaryExpression (left: NumberLiteral (1), right: NumberLiteral (2)), Identifier (someVar)). All nodes in this AST will report // some comment that is why we need to handle only first node (CallExpression) that report this comments and ignore the rest otherwise we would generate # some comment in .gd four times. I will add some comments.

[johnfn]

Ah, I understand. Bit of a wild guess here, but maybe comments are mapped 1:1 with statements, and then copied into child nodes? Could you try just checking if node is a statement type and ONLY if it is appending the comments?

e.g. like this:
https://github.com/johnfn/ts2gd/blob/main/parse_node.ts#L169

[johnfn]

Because it seems if we could solve this then a lot of this code would be simplified - we wouldn't even need to track comments at all in the ParseState!

[adamuso]

I tried limiting it only to statements but then cases like one below are removed. I don't think comments are mapped 1:1 with statements because when single statement is divided to multiple lines and there are comments inside those lines then comments are bound to expression nodes and not to the statement.

someCall(
  // parameter comment
  "abcd",
  // next comment
 123
)

someCall("abcd", 123)

I also tested your example from other comment and indeed it generates invalid GDScript code, so I think we somehow handle these weird cases or we do as you said: limit comment generation only for statements and this way we won't need too worry about such cases.

// example from other comment
let foo = { bar: 1 }
console.log(foo. 
  // Hello
bar)

I don't really know which way to move forward so I'll wait for your opinion. Also there is not so good another way, just leave it as it is and if someone creates comment that generates invalid GDScript then Godot will actually tell him that something is wrong anyway.

[johnfn]

nit: ?? (maybe change the other line too :P)

[adamuso]

You mean why this is in Paths class or why this is called like that?

It is in Paths because I couldn't find another class which have access to ts2gd.json file and I wanted comment generation to be configurable from config.

[johnfn]

Oh I just mean instead of || we can use ?? e.g. this.removeComments = tsgdJson.removeComments ?? false

[adamuso]

Right, still getting used to newer TypeScript after working a lot in old JS code :P

[johnfn]

:D I know that feeling. Welcome to the future my friend!!

[johnfn]

This is really awesome, way to go!

Couple of medium sized issues but nothing major. I think the strategy may end up being to push this into combine since we have a lot of the machinery for doing this already there (as I mentioned in one of my comments). On the other hand, this could still be totally workable if it handles the inline issues I mentioned in my review!

[johnfn]

I'm actually OK with preserving only statement-level comments, and just disregarding all inline comments for now. Once you fix that up let me know and we should be good to go!