Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add type-checking to unplugin #689

Merged
merged 16 commits into from
Sep 14, 2023
Merged

Conversation

Mokshit06
Copy link
Contributor

Adds type-checking to the unplugin. This currently runs only when dts is enabled, though maybe this should be changed.
One of the issues with the current implementation though, which i'm not sure can be fixed either, is that the errors point to the location in the compiled ts/js source rather than the original civet source. Ideally tsc should be mapping them using the sourcemaps, but it doesn't do that here.

@Mokshit06 Mokshit06 temporarily deployed to build September 2, 2023 17:29 — with GitHub Actions Inactive
@STRd6
Copy link
Contributor

STRd6 commented Sep 2, 2023

TypeScript doesn't support incoming source maps and probably never will. We can do the same thing we do in the LSP and remap the diagnostics by using our own source maps like this: https://github.com/DanielXMoore/Civet/blob/main/lsp/source/lib/util.mts#L425

@Mokshit06
Copy link
Contributor Author

@STRd6 How can I get source map lines from compiled civet source?

@STRd6
Copy link
Contributor

STRd6 commented Sep 6, 2023

You can pass sourceMap: true to Civet.compile:

https://github.com/DanielXMoore/Civet/blob/main/lsp/source/lib/typescript-service.mts#L482
https://github.com/DanielXMoore/Civet/blob/main/source/main.civet#L106

Then it returns a {code, sourceMap} object instead of just the string of code.

@edemaine
Copy link
Collaborator

edemaine commented Sep 6, 2023

There's also some utility code for sourcemap mapping. I think in util.civet?

@Mokshit06
Copy link
Contributor Author

Thanks, yeah this should be working now :)

@edemaine
Copy link
Collaborator

edemaine commented Sep 7, 2023

Could you add some docs to the unplugin README about how this works?

@Mokshit06
Copy link
Contributor Author

Sure, will do that

@edemaine
Copy link
Collaborator

edemaine commented Sep 8, 2023

Another minor thing: I think typecheck should not be camel-cased. It's more common this way, and I think it can be considered a single word.

Copy link
Collaborator

@edemaine edemaine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good to me. I got stuck when trying to test it though, because CJS vs. ESM.

integration/unplugin/README.md Outdated Show resolved Hide resolved
integration/unplugin/src/index.ts Show resolved Hide resolved
lsp/source/lib/util.mts Outdated Show resolved Hide resolved
package.json Outdated Show resolved Hide resolved
@edemaine edemaine temporarily deployed to build September 12, 2023 18:48 — with GitHub Actions Inactive
@edemaine edemaine temporarily deployed to build September 12, 2023 18:50 — with GitHub Actions Inactive
@edemaine
Copy link
Collaborator

edemaine commented Sep 12, 2023

This seems like it should be done, but I'm still failing to run tests. Here's what I tried:

integration/unplugin/examples/vite

diff --git a/integration/unplugin/examples/vite/src/main.civet b/integration/unplugin/examples/vite/src/main.civet
--- a/integration/unplugin/examples/vite/src/main.civet
+++ b/integration/unplugin/examples/vite/src/main.civet
@@ -1,3 +1,6 @@
 {a} from "./module.civet"

-console.log a
+const b = 5
+console.log b()
+
+console.log a + 5
diff --git a/integration/unplugin/examples/vite/vite.config.js b/integration/unplugin/examples/vite/vite.config.js
index 7d9ffb5..bc51b4a 100644
--- a/integration/unplugin/examples/vite/vite.config.js
+++ b/integration/unplugin/examples/vite/vite.config.js
@@ -2,5 +2,5 @@ import civetVitePlugin from '@danielx/civet/vite';
 import { defineConfig } from 'vite';

 export default defineConfig({
-  plugins: [civetVitePlugin({})],
+  plugins: [civetVitePlugin({typecheck: true})],
 });

Then I run

npm install
npm link @danielx/civet
npm run build

This builds, but doesn't generate any diagnostics. Should it?

integration/unplugin/examples/astro

Just running:

npm install
npm link @danielx/civet
npm run build

crashes with error:


> [email protected] build
> astro build

2:46:58 PM [vite] Error when evaluating SSR module /@fs/C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs: failed to import "@danielx/civet/ts-diagnostic"
|- Error: Cannot find module '@danielx/civet/ts-diagnostic' imported from 'C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs'
    at nodeImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:56010:25)
    at ssrImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55912:30)
    at eval (C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs:5:37)
    at async instantiateModule (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55974:9)

2:46:58 PM [vite] Error when evaluating SSR module /@fs/C:/Users/edemaine/Projects/Civet/dist/vite.mjs: failed to import "/@fs/C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs"
|- Error: Cannot find module '@danielx/civet/ts-diagnostic' imported from 'C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs'
    at nodeImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:56010:25)
    at ssrImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55912:30)
    at eval (C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs:5:37)
    at async instantiateModule (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55974:9)

2:46:58 PM [vite] Error when evaluating SSR module C:\Users\edemaine\Projects\Civet\integration\unplugin\examples\astro\astro.config.mjs: failed to import "/@fs/C:/Users/edemaine/Projects/Civet/dist/vite.mjs"
|- Error: Cannot find module '@danielx/civet/ts-diagnostic' imported from 'C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs'
    at nodeImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:56010:25)
    at ssrImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55912:30)
    at eval (C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs:5:37)
    at async instantiateModule (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55974:9)

[astro] Unable to load your Astro config

 error   Cannot find module '@danielx/civet/ts-diagnostic' imported from 'C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs'
  File:
    C:\Users\edemaine\Projects\Civet\integration\unplugin\examples\astro\node_modules\vite\dist\node\chunks\dep-df561101.js:56010:25
  Code:
    56009 |         if (!resolved) {
    > 56010 |             const err = new Error(`Cannot find module '${id}' imported from '${importer}'`);
            |                         ^
      56011 |             err.code = 'ERR_MODULE_NOT_FOUND';
      56012 |             throw err;
      56013 |         }
  Stacktrace:
Error: Cannot find module '@danielx/civet/ts-diagnostic' imported from 'C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs'
    at nodeImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:56010:25)
    at ssrImport (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55912:30)
    at eval (C:/Users/edemaine/Projects/Civet/dist/unplugin-shared.mjs:5:37)
    at async instantiateModule (file:///C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/astro/node_modules/vite/dist/node/chunks/dep-df561101.js:55974:9)

I'm not sure why though; the export looks properly defined. (And I did build Civet, and I checked the link; in particular, node_modules/@danielx/civet/dist/ts-diagnostic.{js,mjs} exist.) Maybe Astro doesn't always respect exports? Or maybe it's an npm link issue?

@Mokshit06
Copy link
Contributor Author

Huh, so my first guess was that it might have something to do with exports not being supported with self-referencing in node (referencing @danielx/civet/ts-diagnostic inside unplugin) but that is indeed supported https://nodejs.org/api/packages.html#self-referencing-a-package-using-its-name. It looks like this error is occurring inside vite, so might be something specific to it. I'll have to dig deeper

@edemaine
Copy link
Collaborator

Do you have a setting where type checking successfully reports errors? (Should I try another example?) I'd be happy to merge this and leave more situations for the future — I'd just like to see it report type errors in at least one setting. 😸

@Mokshit06 Mokshit06 temporarily deployed to build September 13, 2023 17:40 — with GitHub Actions Inactive
@Mokshit06
Copy link
Contributor Author

Mokshit06 commented Sep 13, 2023

Vite should be working now 😃 When adding files to fsMap for typescript to walk over, I missed adding the case for options.typecheck. Just tried out adding esbuild as well, and that also works fine.

@edemaine
Copy link
Collaborator

edemaine commented Sep 13, 2023

In the Vite example I'm now getting this error:

[unplugin-civet] Cannot read properties of undefined (reading 'length')
error during build:
TypeError: Cannot read properties of undefined (reading 'length')
    at createSourceFile2 (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:29215:58)
    at parseSourceFileWorker (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:29089:30)
    at Object.parseSourceFile (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:28918:26)
    at Object.createSourceFile (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:28103:23)
    at Object.getSourceFile (C:\Users\edemaine\Projects\Civet\node_modules\@typescript\vfs\dist\vfs.cjs.production.min.js:1:4253)
    at findSourceFileWorker (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:120079:25)
    at findSourceFile (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:119999:22)
    at C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:119948:24
    at getSourceFileFromReferenceWorker (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:119917:28)
    at processSourceFile (C:\Users\edemaine\Projects\Civet\node_modules\typescript\lib\typescript.js:119946:7)

This seems like progress but also not correctly reporting an error. :-) Am I doing something wrong? I did build...

@Mokshit06
Copy link
Contributor Author

Mokshit06 commented Sep 13, 2023

Weird, this seems to working for me. What's the location in unplugin-shared.js for this error?


> build
> npx vite build

vite v4.4.9 building for production...
transforming (1) index.htmlsrc/main.civet.tsx:4:13 - error TS2349: This expression is not callable.
  Type 'Number' has no call signatures.

4 console.log(b())
              ~

✓ 4 modules transformed.
dist/index.html                0.38 kB │ gzip: 0.27 kB
dist/assets/index-733fc199.js  0.77 kB │ gzip: 0.43 kB
✓ built in 319ms

@edemaine
Copy link
Collaborator

Good news: Vite indeed works for me on Linux. So the error I get (still as above) is Windows-specific.

Annoyingly, it does not give a source line number, but with some instrumentation, I find that it's using dist/vite.js and it's failing during this call:

const program = import_typescript.default.createProgram({
          rootNames: [...fsMap.keys()],
          options: compilerOptions,
          host: host.compilerHost
        });

I checked fsMap.keys() and it seems reasonable, though the paths are in Windows format. ('C:\\Users\\edemaine\\Projects\\Civet\\integration\\unplugin\\examples\\vite\\src\\main.civet.tsx', 'C:\\Users\\edemaine\\Projects\\Civet\\integration\\unplugin\\examples\\vite\\src\\module.civet.tsx') Whereas on Linux, they are Unix paths ('/.../edemaine/Projects/Civet/integration/unplugin/examples/vite/src/main.civet.tsx', '/.../edemaine/Projects/Civet/integration/unplugin/examples/vite/src/module.civet.tsx').

Ah yes, the problem is that someone is normalizing filenames to use / instead of \. When processSourceFile gets called (by Vite I believe), it gives the key of C:/Users/edemaine/Projects/Civet/integration/unplugin/examples/vite/src/main.civet.tsx whereas we stored the version with \s.

I think this is the issue we're running into: https://vitejs.dev/guide/api-plugin.html#path-normalization
It sounds like we might need to call normalizePath helpers specifically for Vite and Rollup, and maybe other build systems that do similar path normalization. It doesn't seem like unplugin offers a helper that abstracts this, though perhaps it should...

I think one fix could be this: when we call fsMap.set(path.resolve(process.cwd(), id), code); also call fsMap.set(samePath.replace(/\\/g, '/'), code). I think this should only double the set size, which should be minimal (re-using the code object) and should be safe in all environments. Thoughts?

@edemaine edemaine temporarily deployed to build September 14, 2023 17:00 — with GitHub Actions Inactive
@edemaine
Copy link
Collaborator

I added my proposed fix and this now works for me on both Windows and Linux. So I think we're done!

@edemaine edemaine merged commit cfa1cc7 into DanielXMoore:main Sep 14, 2023
2 checks passed
@Mokshit06
Copy link
Contributor Author

Great find! I remember this is something I had faced issues with when working on Macaron as well, didn't realise we were facing the same issue here. I wonder if this also fixes the Astro issue  🤔 though it seems unrelated to the path formatting

@edemaine
Copy link
Collaborator

Amazingly, it did fix Astro too! Yay!!

By the way, do you think we should add typecheck: true to all the examples?

@Mokshit06
Copy link
Contributor Author

That's funny, Astro's error messages can be pretty weird sometimes I've seen. I was testing the plugin out again, and it looks like we need to move the fsMap.set part to be inside load itself as rollup expects the return value of transform to be valid JS.

we should add typecheck: true to all the examples

Hmm I don't think so, I like build tools to not do type checking in CLI during dev builds as it slows it down by alot. Maybe we can do it in the examples based on process.env.TS_CHECK?

@edemaine
Copy link
Collaborator

I'm not getting any errors with the Rollup example using typecheck: true. If I use dts: true, I do get errors:

main.civet  dist...
(!) The emitted file "main.civet.d.ts" overwrites a previously emitted file of the same name.
(!) The emitted file "utils\module.civet.d.ts" overwrites a previously emitted file of the same name.
created dist in 361ms

That's funny, Astro's error messages can be pretty weird sometimes I've seen. I was testing the plugin out again, and it looks like we need to move the fsMap.set part to be inside load itself as rollup expects the return value of transform to be valid JS.

we should add typecheck: true to all the examples

I like build tools to not do type checking in CLI during dev builds as it slows it down by alot. Maybe we can do it in the examples based on process.env.TS_CHECK?

It would be natural to have a test script to do the type checking. So yeah, maybe the script could be TYPECHECK=1 npx rollup --config rollup.config.js and then the config file checks for the environment variable TYPECHECK.

@Mokshit06
Copy link
Contributor Author

That is because currently the rollup example doesn't have any TS-specific syntax—all of it is inferred. But as soon I add a type declaration to rollup/main.civet

{a} from "./utils/module.civet"

export b = a

export * from "./utils/module.civet"

export type X = string

I get this error

$ npx rollup --config rollup.config.js

main.civet → dist...
[!] RollupError: Unexpected token (Note that you need plugins to import files that are not JavaScript)
main.civet.tsx (7:7)
5: export * from "./utils/module.civet"
6: 
7: export type X = string
          ^
8: 
9: //# sourceMappingURL=data:application/json;charset=utf-8;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoibWFpbi5jaXZldC50c3gudHN4I...
    at error (/Users/mokshitjain/dev/Civet/node_modules/rollup/dist/shared/rollup.js:353:30)
    at Module.error (/Users/mokshitjain/dev/Civet/node_modules/rollup/dist/shared/rollup.js:15206:16)
    at Module.tryParse (/Users/mokshitjain/dev/Civet/node_modules/rollup/dist/shared/rollup.js:15937:25)
    at Module.setSource (/Users/mokshitjain/dev/Civet/node_modules/rollup/dist/shared/rollup.js:15538:39)
    at ModuleLoader.addModuleSource (/Users/mokshitjain/dev/Civet/node_modules/rollup/dist/shared/rollup.js:25720:20)

@edemaine
Copy link
Collaborator

Oh, right, I recall you've mentioned that issue before: Rollup needs js: true. Could you put together another PR when you get a chance?

Somewhat related: Given that we have TypeScript integration in this plugin, I wonder whether it would actually make sense to offer an option of transpiling TS → JS via TypeScript instead of Civet's js: true mode, which lacks some features (most recently, using). Not sure the best option name — perhaps typescript: true? (ts: true feels weird given the meaning of js: true...?) It could be implied by typecheck: true or dts: true...

@Mokshit06
Copy link
Contributor Author

That would disable all the other bundler plugins from further processing the code. It could lead to confusing behaviour, so inferring it from typecheck: true or dts: true might not be suitable. Though maybe we can add it as a separate config option if enough people feel the need for it?

@edemaine
Copy link
Collaborator

edemaine commented Sep 14, 2023

Oops, indeed. Retroactively, I guess I meant "implied by typecheck: true or dts: true when we also have js: true". Alternatively, we could use something like js: 'tsc'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants