Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add E2E tests #350

Merged
merged 30 commits into from
Sep 13, 2023
Merged

Add E2E tests #350

merged 30 commits into from
Sep 13, 2023

Conversation

andrii-i
Copy link
Collaborator

@andrii-i andrii-i commented Aug 18, 2023 •
edited
Loading

  • Updates and fixes integration / E2E testing CI pipeline
  • Adds snapshot-based E2E tests: sidebar with chat icon, chat welcome message
  • Adds test helper class
  • Updates docs at packages/jupyter-ai/ui-tests/README.md, docs/contributors/index.md with e2e testing instructions

Fixes #211

image

After this PR is merged, E2E workflow should be added to required CI checks

To test this PR and run e2e tests locally, please use instructions from https://github.com/andrii-i/jupyter-ai/blob/ui-tests/docs/source/contributors/index.md#integration--e2e-tests:

Install test dependencies (needed only once):

cd ./packages/jupyter-ai/ui-tests/
jlpm install
jlpm playwright install

Tests involve snapshot comparisons against a reference snapshots generated by the Github CI. If you are using an OS other than Linux, you will need to generate local snapshots before running the tests locally for the first time. To do this, execute the command:

cd ./packages/jupyter-ai/ui-tests/
jlpm test:update

To execute tests, run:

cd ./packages/jupyter-ai/ui-tests/
jlpm test

You can find more information in the ui-tests README.

@welcome
Copy link

welcome bot commented Aug 18, 2023

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.
welcome
You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

@andrii-i andrii-i added enhancement New feature or request testing Quality assurance tests maintenance Change related to maintenance of the repository labels Aug 18, 2023
@andrii-i andrii-i marked this pull request as ready for review August 23, 2023 17:53
@andrii-i andrii-i changed the title Add UI tests Add E2E tests Aug 23, 2023
@andrii-i andrii-i force-pushed the ui-tests branch 3 times, most recently from 4c54b8c to b4e6ef7 Compare August 25, 2023 21:01
@andrii-i andrii-i added this to the 2.2.0 Release milestone Aug 28, 2023
@andrii-i andrii-i added the project:ui In-editor dialog/popup UI, improve selction UI label Aug 28, 2023
@JasonWeill JasonWeill assigned andrii-i and unassigned andrii-i Aug 28, 2023
@JasonWeill JasonWeill linked an issue Aug 28, 2023 that may be closed by this pull request
Add tests to CI pipeline #211
Closed
@JasonWeill JasonWeill removed a link to an issue Aug 28, 2023
Add tests to CI pipeline #211
Closed
@JasonWeill JasonWeill linked an issue Aug 28, 2023 that may be closed by this pull request
Add tests to CI pipeline #211
Closed
@JasonWeill JasonWeill removed this from the 2.2.0 Release milestone Aug 28, 2023
@3coins
Copy link
Collaborator

3coins commented Aug 29, 2023

@andrii-i
I tested the change locally, and found a few issues:

  1. The port to run the server should be a setting that should be documented in the instructions. For instance, I had port 8888 already occupied, so had to change this at two places, but other contributors might not be aware of this.
  2. jlpm start blocks the terminal, and a new terminal session is needed to run jlpm playwright test, let's make this clear in the instructions.
  3. The tests are currently getting stuck and failing waiting for the token value to be input in the login window. How did we handle this in Jupyter Scheduler? Can we suppress this or have a default token that could avoid the login screen?

@andrii-i andrii-i force-pushed the ui-tests branch 2 times, most recently from 36173a1 to 5816369 Compare August 31, 2023 15:26
@andrii-i
Copy link
Collaborator Author

andrii-i commented Sep 6, 2023

Kicking CI

@andrii-i andrii-i closed this Sep 6, 2023
@andrii-i andrii-i reopened this Sep 6, 2023
3coins
3coins requested changes Sep 6, 2023
View reviewed changes
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
packages/jupyter-ai/ui-tests/README.md Outdated Show resolved Hide resolved
docs/source/contributors/index.md Outdated Show resolved Hide resolved
@andrii-i andrii-i requested a review from 3coins September 8, 2023 03:40
@andrii-i
Copy link
Collaborator Author

andrii-i commented Sep 8, 2023

@3coins PR is ready for re-review but it can wait until you are back from vacation

3coins
3coins approved these changes Sep 12, 2023
View reviewed changes
Copy link
Collaborator

@3coins 3coins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrii-i
One minor comment, looks good otherwise.

packages/jupyter-ai/ui-tests/README.md Show resolved Hide resolved
@andrii-i andrii-i merged commit 8f3bfe0 into jupyterlab:main Sep 13, 2023
5 checks passed
@andrii-i andrii-i deleted the ui-tests branch September 13, 2023 05:45
andrii-i added a commit to andrii-i/jupyter-ai that referenced this pull request Sep 13, 2023
@andrii-i @pre-commit-ci @3coins
Add E2E tests (jupyterlab#350)
3cb1235 
* add basic e2e testing setup

* adjust execute test step name

* test sidebar chat icon, add testing class

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sidebar snapshot

* test chat sidepanel, extend helper class

* adjust welcome message test, add snapshot

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adjust naming

* removeempty line

* move ui-tests to packages/jupyter-ai/

* update e2e ci workflow for ui-tests folder move

* update ui-tests folder location for yarn.lock hash

* run lint locally

* Add "Update Playwright Snapshots" CI workflow

* change if clause

* specify npm client

* remove report and artifact specifiers

* Update README.md to have correct commands and folders

* update e2e/integration test README

* Add Integration / E2E testing section to the docs

* update wording of docs on snapshots

* Ignore all non-linux snapshots

* Update packages/jupyter-ai/ui-tests/README.md

Co-authored-by: Piyush Jain <[email protected]>

* remove cd command that would return users back to root

* remove cd ../../../

* Remove repeating setup instructions

* Add suggestion to generate snapshots before the 1st run

* remove unnecessary link anchor

* remove rudimentary jlpm build

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>
@andrii-i andrii-i mentioned this pull request Sep 13, 2023
Merged
andrii-i added a commit that referenced this pull request Sep 18, 2023
@andrii-i @pre-commit-ci @3coins
[1.x] Add E2E tests (#388)
491490a 
* Add E2E tests (#350)

* add basic e2e testing setup

* adjust execute test step name

* test sidebar chat icon, add testing class

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sidebar snapshot

* test chat sidepanel, extend helper class

* adjust welcome message test, add snapshot

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adjust naming

* removeempty line

* move ui-tests to packages/jupyter-ai/

* update e2e ci workflow for ui-tests folder move

* update ui-tests folder location for yarn.lock hash

* run lint locally

* Add "Update Playwright Snapshots" CI workflow

* change if clause

* specify npm client

* remove report and artifact specifiers

* Update README.md to have correct commands and folders

* update e2e/integration test README

* Add Integration / E2E testing section to the docs

* update wording of docs on snapshots

* Ignore all non-linux snapshots

* Update packages/jupyter-ai/ui-tests/README.md

Co-authored-by: Piyush Jain <[email protected]>

* remove cd command that would return users back to root

* remove cd ../../../

* Remove repeating setup instructions

* Add suggestion to generate snapshots before the 1st run

* remove unnecessary link anchor

* remove rudimentary jlpm build

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>

* update server test config

* regenerate yarn.loc

* roll back package.json

* regenerate yarn.lock

* add install-chromium command

* update snapshots

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>
Wzixiao added a commit to Wzixiao/jupyter-ai-bigcode-code-completion that referenced this pull request Oct 31, 2023
@Wzixiao @andrii-i
@pre-commit-ci @3coins @dlqqq @chenJuZ @krassowski
Mock request (#27)
b5a0ada 
* Add E2E tests (jupyterlab#350)

* add basic e2e testing setup

* adjust execute test step name

* test sidebar chat icon, add testing class

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sidebar snapshot

* test chat sidepanel, extend helper class

* adjust welcome message test, add snapshot

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adjust naming

* removeempty line

* move ui-tests to packages/jupyter-ai/

* update e2e ci workflow for ui-tests folder move

* update ui-tests folder location for yarn.lock hash

* run lint locally

* Add "Update Playwright Snapshots" CI workflow

* change if clause

* specify npm client

* remove report and artifact specifiers

* Update README.md to have correct commands and folders

* update e2e/integration test README

* Add Integration / E2E testing section to the docs

* update wording of docs on snapshots

* Ignore all non-linux snapshots

* Update packages/jupyter-ai/ui-tests/README.md

Co-authored-by: Piyush Jain <[email protected]>

* remove cd command that would return users back to root

* remove cd ../../../

* Remove repeating setup instructions

* Add suggestion to generate snapshots before the 1st run

* remove unnecessary link anchor

* remove rudimentary jlpm build

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>

* Adds chat anthropic provider, new models (jupyterlab#391)

* Adds chat anthropic provider, new models

* Added docs for anthropic chat

* Upgraded LangChain, fixed prompts for Bedrock

* Updated docs for bedrock-chat

* Added bedrock embeddings, refactored chat vs reg models

* Fixed magics

* Removed unused import

* Updated provider list.

Added bedrock and bedrock-chat to provider list.

* Publish 2.3.0

SHA256 hashes:

jupyter-ai-core-2.3.0.tgz: 8f37fe0f15b6f09b2eeb649ac972d2749427ed3668a03ffae9cf5b5f8f37a8ce

jupyter_ai-2.3.0-py3-none-any.whl: 09e264c40f05ef34cd188dd5804d22afe43a91e4c82ed729428377bd5c581263

jupyter_ai-2.3.0.tar.gz: 8ce44b88528195e6de1f9086994d68731b1dbbc03f0e8709baf5a8819c254462

jupyter_ai_magics-2.3.0-py3-none-any.whl: 714cf33746c121ef2b5d5c45b4460e690d6815b306a3f5f7224008866e794602

jupyter_ai_magics-2.3.0.tar.gz: 37554e53d3576a6c8938e5812764efe7749dfeda2f47d5e551111d2f6d8a5c48

* [pre-commit.ci] pre-commit autoupdate (jupyterlab#344)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0)
- [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1)
- [github.com/asottile/pyupgrade: v3.10.1 → v3.15.0](asottile/pyupgrade@v3.10.1...v3.15.0)
- [github.com/sirosen/check-jsonschema: 0.23.3 → 0.27.0](python-jsonschema/check-jsonschema@0.23.3...0.27.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add key completion code completion

* Add developer readme

* Fix the bug of remote call not using stram

* Comment out useless methods and improve development documentation

* Updates for more stable generate feature

* Refactored generate for better stability with all providers/models.

* Upgraded LangChain to 0.0.318

* Updated to use memory instead of chat history, fix for Bedrock Anthropic

* Optimize code naming

* Fixed the issue where enter would intercept automatic requests

* Fixed the issue where enter fails when pressing the ghost text to start.

* Allow to define block and allow lists for providers (jupyterlab#415)

* Allow to block or allow-list providers by id

* Add tests for block/allow-lists

* Fix "No language model is associated with" issue

This was appearing because the models which are blocked were not
returned (correctly!) but the previous validation logic did not
know that sometimes models may be missing for a valid reason even
if there are existing settings for these.

* Add docs for allow listing and block listing providers

* Updated docs

* Added an intro block to docs

* Updated the docs

---------

Co-authored-by: Piyush Jain <[email protected]>

* Publish 2.4.0

SHA256 hashes:

jupyter-ai-core-2.4.0.tgz: 04773e2b888853cd1c27785ac3c8434226e9a279a2fd253962cb20e5e9f72c1d

jupyter_ai-2.4.0-py3-none-any.whl: a5880cc108a107c746935d7eaa2513dffa29d2812e6628fd22a972a97aba4e2a

jupyter_ai-2.4.0.tar.gz: 0d065b18f4985fb726010e76d9c6059932e21327ea2951ccaa18b6e7b5189240

jupyter_ai_magics-2.4.0-py3-none-any.whl: 585bd960ac5c254e28ea165db840276883155a0a720720aa850e3272edc2001e

jupyter_ai_magics-2.4.0.tar.gz: 2cdfb1e084aad46cdbbfb4eed64b4e7abc96ad7fde31da2ddb6899225dfa0684

* Complete autocomplete

* Animations now also exist on automatic requests

* Remove DEVELOP.md

* Added toggle for selecting mock tests

* Make the code neater

---------

Co-authored-by: Andrii Ieroshenko <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>
Co-authored-by: dlqqq <[email protected]>
Co-authored-by: cjq <[email protected]>
Co-authored-by: Michał Krassowski <[email protected]>
Co-authored-by: 3coins <[email protected]>
dbelgrod pushed a commit to dbelgrod/jupyter-ai that referenced this pull request Jun 10, 2024
@andrii-i @pre-commit-ci @3coins
Add E2E tests (jupyterlab#350)
abcc8ce 
* add basic e2e testing setup

* adjust execute test step name

* test sidebar chat icon, add testing class

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sidebar snapshot

* test chat sidepanel, extend helper class

* adjust welcome message test, add snapshot

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adjust naming

* removeempty line

* move ui-tests to packages/jupyter-ai/

* update e2e ci workflow for ui-tests folder move

* update ui-tests folder location for yarn.lock hash

* run lint locally

* Add "Update Playwright Snapshots" CI workflow

* change if clause

* specify npm client

* remove report and artifact specifiers

* Update README.md to have correct commands and folders

* update e2e/integration test README

* Add Integration / E2E testing section to the docs

* update wording of docs on snapshots

* Ignore all non-linux snapshots

* Update packages/jupyter-ai/ui-tests/README.md

Co-authored-by: Piyush Jain <[email protected]>

* remove cd command that would return users back to root

* remove cd ../../../

* Remove repeating setup instructions

* Add suggestion to generate snapshots before the 1st run

* remove unnecessary link anchor

* remove rudimentary jlpm build

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>
Marchlak pushed a commit to Marchlak/jupyter-ai that referenced this pull request Oct 28, 2024
@andrii-i @pre-commit-ci
@3coins @Marchlak
Add E2E tests (jupyterlab#350)
6f8f7ac 
* add basic e2e testing setup

* adjust execute test step name

* test sidebar chat icon, add testing class

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sidebar snapshot

* test chat sidepanel, extend helper class

* adjust welcome message test, add snapshot

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adjust naming

* removeempty line

* move ui-tests to packages/jupyter-ai/

* update e2e ci workflow for ui-tests folder move

* update ui-tests folder location for yarn.lock hash

* run lint locally

* Add "Update Playwright Snapshots" CI workflow

* change if clause

* specify npm client

* remove report and artifact specifiers

* Update README.md to have correct commands and folders

* update e2e/integration test README

* Add Integration / E2E testing section to the docs

* update wording of docs on snapshots

* Ignore all non-linux snapshots

* Update packages/jupyter-ai/ui-tests/README.md

Co-authored-by: Piyush Jain <[email protected]>

* remove cd command that would return users back to root

* remove cd ../../../

* Remove repeating setup instructions

* Add suggestion to generate snapshots before the 1st run

* remove unnecessary link anchor

* remove rudimentary jlpm build

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Piyush Jain <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3coins 3coins 3coins approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request maintenance Change related to maintenance of the repository project:ui In-editor dialog/popup UI, improve selction UI testing Quality assurance tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add tests to CI pipeline
3 participants
@andrii-i @3coins @JasonWeill