[User Error] Playwright traces in Artillery Cloud

[staramani-ut]

This was my error. Please see the comments below.

---

@hassy, thank you so much for enabling playwright traces! I am trying this out and the traces don't make it to the cloud instance. Could I be doing something incorrectly?

    "artillery": "^2.0.15",
    "playwright": "^1.39.0",

Here is the yaml:

    bundling:
      external: ['playwright-core']
    target: URL_REMOVED
    plugins:
      ensure:
        thresholds:
          # Common benchmarks and best practices:
          # - TTFB (Time to First Byte) under 500 ms is considered good.
          # - LCP (Largest Contentful Paint) under 2500 ms is considered good.
          # - FCP (First Contentful Paint) under 2000 ms is considered good.
          # - CLS (Cumulative Layout Shift) under 0.1 is considered good.
          - "browser.page.CLS.surveyPX.p75": 0.1
          - "browser.page.FCP.surveyPX.p75": 2000
          - "browser.page.LCP.surveyPX.p75": 2500
          - "browser.page.TTFB.surveyPX.p75": 500
    engines:
      playwright:
        aggregateByName: true
        defaultTimeout: 15
        defaultNavigationTimeout: 15
        extendedMetrics: true
        showAllPageMetrics: true
        showFailedRequests: true
        trace:
          enabled: true
        launchOptions:
          headless: true
    processor: ../../../dist/artillery/artillery.setup.js
    phases:
      - name: 150 users over 120 seconds
        duration: 120
        arrivalCount: 150

One nit: The info on https://app.artillery.io/load-tests/UUID/traces has the following, which differs from what is in this post.

[staramani-ut]

Ah, I was changing some stuff and got this to work. I will post what I changed just in case it helps someone else.

[hassy]

Please do! (There are some still-to-be-documented settings for the integration as well that can be tweaked to produce more/fewer traces) Thanks for flagging up that example in the app as well, we'll update it (it should still work, but using a configuration object under trace is the default way).

[staramani-ut]

I can't remember why, but I had code to set up the browser, which I don't need. I cleaned it up along with the events I was emitting and that made everything work.

[staramani-ut]

Traces seem to be okay when I run "locally" on my machine, but don't when I run on GitHub Actions. Anything I might be missing?

https://app.artillery.io/share/sh_3744e820cb1837eeff2694b8a6808071f6b1afbaddc36fd2b9d2d491507bbf59

[hassy]

This could be due to how traces are captured by Artillery. Full docs coming soon but the gist is that tracing is not enabled for all VUs by default (to avoid the performance impact of it). Instead, up to 3 VUs at a time will be recording a trace. If one of those VUs fails, the trace is then uploaded. It's possible that none of the 9 VUs that failed in the test you shared happened to be recording.

The number of VUs that will be recording concurrently can be changed with maxConcurrentRecordings setting. Increasing it to a larger value will make it more likely that a trace for a failure will be recorded. Could you try bumping that setting to see if it makes traces appear?

engines:
  playwright:
    trace:
      enabled: true
      maxConcurrentRecordings: 10

In theory there should be no difference between running locally vs GitHub Actions. You can also try setting recordSuccessfulVUs: true which will save & upload traces for VUs that succeded. That would help rule out a GitHub Actions-specific issue.

[staramani-ut]

Okay, this makes a lot of sense for when I saw trace for one run and not for another run locally.

I will increase the maxConcurrentRecordings, but I wonder if that would impact the actual test too much.