Add documentation for testing (#165)

_quarto.yml

@@ -100,6 +100,8 @@ website:
             href: api/express/index.qmd
           - text: "Shiny Core"
             href: api/core/index.qmd
+          - text: "Testing"
+            href: api/testing/index.qmd
     tools:
       - icon: discord
         href: https://discord.gg/yMGCamUMnS
@@ -255,6 +257,10 @@ website:
           contents:
             - docs/modules.qmd
             - docs/module-communication.qmd
+        - section: "Testing"
+          contents:
+            - docs/testing.qmd
+            - docs/playwright-testing.qmd
         - section: "Extending"
           contents:
             - docs/custom-component-one-off.qmd

docs/playwright-testing.qmd

@@ -0,0 +1,170 @@
+---
+title: End-to-end testing using Playwright
+editor:
+  markdown:
+    wrap: sentence
+---
+
+### What are End-to-end Tests?
+
+End-to-end testing for Shiny apps is like checking your app from start to finish, just as a user would.
+
+Imagine you're using your Shiny app. You click buttons, enter data, and see results on a graph or a dashboard. End-to-end tests mimic these actions.
+Instead of manually clicking around, we write code to do this for us. The code interacts with your app like a user, checking if everything works as expected.
+
+#### Benefits
+- End-to-end tests find issues early, like broken links or unexpected behavior.
+- As your app grows, it becomes harder to keep track of all parts. Tests help ensure nothing breaks.
+
+
+### Playwright
+
+***Playwright*** is an open-source library developed by Microsoft. It enables developers to automate browser interactions and perform end-to-end testing of web applications.
+
+Benefits of using Playwright for Shiny App testing
+
+- **End-to-End Testing**: Playwright allows you to simulate real user interactions with your Shiny app, ensuring that the reactive components and user flows work as expected.
+- **Cross-Browser Testing**: Playwright supports multiple browsers like Chromium, Firefox, and Safari(Webkit), enabling you to test your Shiny app's compatibility across different browser environments.
+- **Dynamic wait times** Playwright provides dynamic wait times, automatically waiting for elements to be ready before interacting with them, which eliminates the need for explicit waits and reduces flakiness caused by timing issues.
+
+For detailed information and guidance, check out the [official Playwright documentation](https://playwright.dev/python/).
+
+### Getting started with writing your first end-to-end test
+
+::: {.callout-tip collapse="true"}
+## Prerequisite: Installing Playwright Pytest
+
+`pytest-playwright` is a plugin that integrates ***Playwright*** with the ***Pytest*** framework to facilitate end-to-end testing of web applications.
+
+This can be installed by typing the following command in the terminal
+
+```bash
+pip install pytest-playwright
+```
+:::
+
+Let's say you have a shiny app the doubles the slider value with the code shown below:
+
+```python
+# app.py
+from shiny import render, ui
+from shiny.express import input
+
+ui.panel_title("Hello Shiny!")
+ui.input_slider("n", "N", 0, 100, 20)
+
+
+@render.text
+def txt():
+    return f"n*2 is {input.n() * 2}"
+```
+
+If we want to test that the shiny app works for the following scenario:
+
+1. Wait for the Shiny app to finish loading
+1. Drag the slider to value as `55`
+1. Verify the output text changes to reflect the value of `n*2 is 110`
+
+The test code to test the shiny app to emulate the above scenario would be as following:
+
+```python
+# test_basic_app.py
+from shiny.playwright import controller
+from shiny.run import ShinyAppProc
+from playwright.sync_api import Page
+from shiny.pytest import create_app_fixture
+
+app = create_app_fixture("remote/basic-app/app.py")
+
+
+def test_basic_app(page: Page, app: ShinyAppProc):
+    page.goto(app.url)
+    txt = controller.OutputText(page, "txt")
+    slider = controller.InputSlider(page, "n")
+    slider.set("55")
+    txt.expect_value("n*2 is 110")
+```
+
+#### Explanation of the test code:
+
+1. The code begins by importing the `controller` module. This module is crucial as it contains the classes that represent various user interface (UI) controllers used in the Shiny application.
+
+2. Defines ***test_basic_app*** function with ***page*** and ***app*** parameters. *page* is an instance of the Page class from the Playwright library, which represents a single tab in a browser, and *app* is an instance of the `ShinyAppProc` class, which represents the Shiny app being tested.
+
+3. Navigates to the app's URL.
+
+4. Creates instances of `OutputText` and `InputSlider` for UI elements.
+
+5. Sets the slider value to `55`.
+
+6. Checks if the output text displays `n*2 is 110` as expected.
+
+![](assets/end-to-end-test-workflow.png)
+
+### Using `shiny add test` to create test files for your shiny apps
+
+`Shiny` provides a simple way to create a test file for your shiny app. Just type `shiny add test` in your terminal/console and give the **path** to the shiny app file along with the **path** of the test file.
+
+::: {.callout-tip collapse="true"}
+## Naming of the test file
+
+The basename of the test file should start with `test_` and be unique across all test files.
+pytest automatically discovers and runs tests in your project based on a naming convention (files or functions starting with `test_*.py` or ending with `*_test.py`), eliminating the need for manual test case registration. More information about test discovery can be found [here](https://docs.pytest.org/en/8.2.x/explanation/goodpractices.html#test-discovery)
+:::
+
+
+```bash
+shiny add test
+
+? Enter the path to the app file: basic-app/app.py
+? Enter the path to the test file: ./basic-app/test_app.py
+```
+
+##### How to Use the Test File
+
+This test file includes all the necessary imports you'll need to run your tests. Follow these steps to interact with the UI elements on the Shiny app:
+
+1. Create instances of controllers
+
+    - You need to create instances of controllers to interact with different UI elements in the shiny app.
+
+2. Use methods that allow the controllers to interact with the UI elements on the shiny app
+
+
+For example: Interacting with a Slider
+
+To interact with a slider element, create an instance of the `InputSlider` controller. Here's how you can do it:
+
+```python
+slider = controller.InputSlider(page, "<id_of_the_slider>")
+slider.set("20")
+slider.expect_value("20")
+```
+Replace `<id_of_the_slider>` with the actual ID of the slider you want to test.
+
+##### Running the test
+
+Simply type `pytest` in the root directory of your shiny app file and the playwright framework will automatically run the test file.
+
+```bash
+pytest
+
+======== test session starts ========
+platform darwin -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0
+configfile: pytest.ini
+plugins: asyncio-0.21.0, timeout-2.1.0, Faker-20.1.0, cov-4.1.0, playwright-0.4.4, rerunfailures-11.1.2, xdist-3.3.1, base-url-2.1.0, hydra-core-1.3.2, anyio-3.7.0, syrupy-4.0.5, shiny-1.0.0
+asyncio: mode=strict
+12 workers [1 item]
+.
+
+
+======== 1 passed in 3.05s ========
+```
+
+Each test inside the file is shown by a single character in the output:
+
+- `.` for passing
+- `F` for failure.
+
+
+For more information about the testing methods available to the user, read the reference documentation about shiny testing API [here](https://shiny.posit.co/py/api/testing/).

docs/testing.qmd

@@ -0,0 +1,120 @@
+---
+title: Unit testing
+editor:
+  markdown:
+    wrap: sentence
+---
+
+Testing Shiny apps is important to ensure the shiny app functions as expected and to catch any bugs or errors before deployment. It helps maintain code quality, user experience, and prevents potential issues from reaching the end-users.
+
+### pytest
+
+For the example below, we will use ***pytest*** as the test framework for running our unit tests. pytest is a popular, open-source testing framework for Python. It is designed to simplify the process of writing, organizing, and running tests for Python applications and libraries.
+
+More information about ***pytest*** can be found [here](https://docs.pytest.org/en/8.2.x/contents.html).
+
+::: {.callout-tip collapse="true"}
+## Auto discovery for tests
+
+pytest automatically discovers and runs tests in your project based on a naming convention (files or functions starting with `test_*.py` or ending with `*_test.py`), eliminating the need for manual test case registration. More information about test discovery can be found [here](https://docs.pytest.org/en/8.2.x/explanation/goodpractices.html#test-discovery)
+:::
+
+
+Given a shiny app that has the following code that doubles the number for any input that a user provides.
+
+```python
+# app.py
+from shiny import render, ui
+from shiny.express import input
+
+ui.panel_title("Double your amount")
+ui.input_text("txt_box", "Enter number to double it below")
+
+
+@render.text
+def txt():
+    if input.txt_box() == "":
+        return "Please enter a number"
+    # check if input is an int or float
+    try:
+        int(input.txt_box())
+    except ValueError:
+        return "Please enter a valid number"
+    return f"n*2 is {int(value) * 2}"
+```
+
+This code presents challenges for testing due to its logic being nested within reactive code. To enhance testability, we can extract the non-reactive logic and encapsulate it in a separate function called `double_number`. This approach allows for easier isolation and testing of the core functionality, independent of the reactive framework.
+
+
+```python
+# app.py
+from shiny import render, ui
+from shiny.express import input
+
+ui.panel_title("Double your amount")
+ui.input_text("txt_box", "Enter number to double it below")
+
+
+@render.text
+def txt():
+    if input.txt_box() == "":
+        return "Please enter a number"
+    # check if input is an int or float
+    try:
+        int(input.txt_box())
+    except ValueError:
+        return "Please enter a valid number"
+    return f"n*2 is {double_number(input.txt_box())}"
+
+def double_number(value: str):
+    return int(value) * 2
+```
+
+If you want to test the logic of a function that doubles a number, you can create a test file named `test_double_number.py`. This file will contain the necessary code to verify that the function works as expected.
+
+```python
+# test_double_number.py
+
+from app import double_number
+
+def test_double_number():
+    assert double_number("2") == 4
+    assert double_number("5") == 10
+    assert double_number("10") == 20
+    assert double_number("0") == 0
+    assert double_number("-5") == -10
+```
+
+To run the test, you will simply type `pytest` in your terminal. `pytest` will automatically locate the test file and run it with the results shown below.
+
+```bash
+platform darwin -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0
+configfile: pytest.ini
+plugins: asyncio-0.21.0, timeout-2.1.0, Faker-20.1.0, cov-4.1.0, playwright-0.4.4, rerunfailures-11.1.2, xdist-3.3.1, base-url-2.1.0, hydra-core-1.3.2, anyio-3.7.0, syrupy-4.0.5, shiny-1.0.0
+asyncio: mode=strict
+12 workers [1 item]
+.          [100%]
+(3 durations < 5s hidden.  Use -vv to show these durations.)
+```
+
+If the logic in the `double_number` is erroneous, and instead it triples the number, the test will catch it by showing the difference as shown below
+
+```bash
+======================================================= test session starts =======================================================
+platform darwin -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0
+configfile: pytest.ini
+plugins: asyncio-0.21.0, timeout-2.1.0, Faker-20.1.0, cov-4.1.0, playwright-0.4.4, rerunfailures-11.1.2, xdist-3.3.1, base-url-2.1.0, hydra-core-1.3.2, anyio-3.7.0, syrupy-4.0.5, shiny-1.0.0
+asyncio: mode=strict
+12 workers [1 item]
+F       [100%]
+======= FAILURES =======
+________ test_double_number ________
+
+    def test_double_number():
+>       assert double_number("2") == 4
+E       AssertionError: assert 6 == 4
+E        +  where 6 = double_number('2')
+
+```
+
+The tests mentioned earlier are suitable for testing non-reactive functions. However, when it comes to testing the reactivity of a Shiny app, we need to leverage a different approach. In this case, we can use ***Playwright*** to automate browser interactions and perform end-to-end testing of web applications.