test: add integration testing

[sockmaster27]

Fixes #341

[sockmaster27]

TODO:

• Use TypeScript in tests
• Fix issue where tests always fail on first run
• Signal when extension is ready instead of waiting
• Fix Unable to resolve filesystem provider with relative file path 'flixcompiler:141183124/CHANGELOG'
• Write FlixRelease to somewhere persistent to avoid downloading compiler every time (unless it should just be part of the test?)
• GitHub action

[sockmaster27]

• Fix issue where tests always fail on first run

Seems to be entirely dependent on the existence of .vscode-test/user-data/Workspaces/<some number>/workspace.json.
If this does not exist the test will quit after encountering the first await.

[sockmaster27]

Got this far with GitHub action

- run: |
        apt-get update -y
        apt-get install libnss3-dev libgdk-pixbuf2.0-dev libgtk-3-dev libxss-dev -y
        apt-get install libasound2 -y
        # Run tests through Xvfb to give Electron a screen to draw to
        apt-get install xvfb -y
        xvfb-run -a npm test

But currently blocked by

| Warning: vkCreateInstance: Found no drivers!
| Warning: vkCreateInstance failed with VK_ERROR_INCOMPATIBLE_DRIVER

[magnus-madsen]

Does the internet say anything about that? You would think it would be possible to run on GitHub CI since VSCode and GitHub are both Microsoft.

[magnus-madsen]

Maybe this is useful: https://github.com/microsoft/vscode-test/blob/master/sample/azure-pipelines.yml

[magnus-madsen]

https://code.visualstudio.com/api/working-with-extensions/continuous-integration#github-actions

[sockmaster27]

https://code.visualstudio.com/api/working-with-extensions/continuous-integration#github-actions

This was the one I followed, but at least when running locally with act, it needs a lot of extra installations.
Perhaps it works when running it on the real thing. I can try pushing it.

The installations also take quite a lot of time each run, so either way, it might also be preferable to create a Docker image to run on.

[magnus-madsen]

Do you want to wrap this PR up and then we can try GitHub CI in a new PR? I think that would be best.

[magnus-madsen]

Ill give some feedback on the code.

[magnus-madsen]

I think testing the HoverProvider is a good start, if we have like 5-10 tests.

Then I think we should test inline diagnostics on a separate broken program.

But what I would also like is to check that:

• Adding a Flix file works.
• Removing a Flix file works.
• A project with a Flix toml and some dependencies works.

Essentially I am more worried about the "stuff around" than the providers, because if the HoverProvider works probably all the other providers work. What could break is the integration.

[sockmaster27]

I think we should just treat this PR as an introduction of the setup, then we can write some actual test cases in another one.

[magnus-madsen]

Ok,, we can wait with the file manipulation stuff, but lets get the rest done in this PR.

[magnus-madsen]

Write FlixRelease to somewhere persistent to avoid downloading compiler every time (unless it should just be part of the test?)

Lets let it be part of the test!

[sockmaster27]

Do you want to wrap this PR up and then we can try GitHub CI in a new PR? I think that would be best.

It should be working as is, so sounds good 👍

[magnus-madsen]

Do you want to wrap this PR up and then we can try GitHub CI in a new PR? I think that would be best.

It should be working as is, so sounds good 👍

Roger. So if you cleanup my comments then we merge it in. Then in a subsequent PR we can fiddle with the GitHub CI part.

[mlutze]

Can we also get a Readme about how to run the tests?

[magnus-madsen]

Can we also get a Readme about how to run the tests?

Yes please. @sockmaster27 Can you add it to README.md?

[sockmaster27]

• Fix Unable to resolve filesystem provider with relative file path 'flixcompiler:141183124/CHANGELOG'

Can't figure out what causes this but it seems harmless

[magnus-madsen]

Merged! What are the next steps?