Docs menu

Using KwikQA

Running in CI

kwikqa ci runs in your CI job after your tests fail. It analyzes the run as analyze does, and writes the failure report, kwikqa-report.html, with each failure group's evidence. Then it publishes a summary:

  • on GitHub Actions, the job summary and one pull-request comment;
  • on Azure Pipelines, a build summary and one pull-request thread on Azure Repos;
  • on GitLab CI/CD, the summary in the job's log and one merge-request note;
  • on Jenkins, the summary in the console log and one comment on the pull request, on GitHub or GitLab;
  • on any CI system, an optional Slack or Microsoft Teams message, through your own webhooks.

It reads every report analyze reads, so a JUnit, TestNG, pytest, or Cucumber suite gets the same summary as a Playwright one: pass the report with --report, as in Other test frameworks. With a committed policy, kwikqa fix can also repair a failing test in the job and open the repair as a draft pull request, which a person reviews and merges: see Fix pull requests.

1. Create a CI token

In a company account, an admin creates the token on the Organization page, or a team's manager on the team's page; it belongs to the company or the team, so it keeps working when people leave. In a personal account, create it in Account. Name it after where it will live (for example "GitHub Actions"), and copy it right away: it is shown once. Store it as a secret named KWIKQA_TOKEN in your CI system.

Running KwikQA with the token approves its diagnosis requests, and each request's preview is printed in the job's log, so every run shows what was sent. Each run counts as one analysis. On a team plan, every CI token draws from the company's shared pool, and a team's CI token also counts toward that team's cap, if it has one. Which repositories a token can serve depends on your plan; see Plan rules in CI.

2. GitHub Actions: the KwikQA Action

Add the KwikQA Action as a step after your tests:

# .github/workflows/e2e.yml (excerpt)
permissions:
  contents: read
  pull-requests: write # only for the comment
  id-token: write # proves which repository the job runs for

steps:
  - uses: actions/checkout@v7
  - run: npm ci && npx playwright install --with-deps
  - run: npx playwright test

  - if: failure()
    uses: kwikqa/action@v1
    with:
      kwikqa-token: ${{ secrets.KWIKQA_TOKEN }}

With Cypress or WebdriverIO, run your tests as usual (npx cypress run, npx wdio run wdio.conf.ts) with KwikQA's reporter configured, and the Action finds its reports in test-results/kwikqa/. With mochawesome, JUnit, or another framework's report, set the report input.

On a failed run you get:

  • The job summary, on the run's page. It shows the counts, then one row per failure group with its size, owner, and evidence grade. It also lists the flaky tests and links to the report.
  • One pull-request comment with the same summary. A re-run of the job updates the comment instead of adding another, and a later run with no failures updates it to say so. Each job of a workflow keeps its own comment.
  • The failure report, kept as the run's kwikqa-report artifact. It holds each group's tests, diagnosis, and the evidence the diagnosis cites.

Inputs

InputDefaultEffect
kwikqa-tokennoneYour CI token, from a secret. Without it, the run is analyzed locally and isn't diagnosed.
github-tokengithub.tokenWrites the pull-request comment. The job needs pull-requests: write.
reportfoundThe test report, a file or a directory, if it isn't in a default location.
frameworknoneFor a JUnit report, the framework that wrote it: playwright, cypress, or webdriverio.
working-directory.The package that holds your tests.
outputkwikqa-report.htmlWhere to write the failure report, relative to working-directory.
localfalsetrue analyzes without asking a model, even with a token.
notifyon-failureWhen to post to Slack and Teams: on-failure or always.
slack-webhook-url, teams-webhook-urlnoneYour chat webhooks, from secrets.
upload-reporttrueKeep the failure report as an artifact of the run.
artifact-namekwikqa-reportThe report artifact's name. Give each job of a matrix its own.
versionpinnedThe kwikqa version to run. Each release of the Action names the one it was tested with.

Pull requests from forks

GitHub gives a fork's workflows no secrets and a read-only token. On such a pull request, KwikQA analyzes the run locally, writes the report and the job summary, and skips the comment. Don't switch to pull_request_target to get the token: that event runs the fork's code with your repository's secrets.

Without the Action

The Action runs npx kwikqa ci and uploads the report. You can run those two steps yourself:

- if: failure()
  run: npx kwikqa ci
  env:
    KWIKQA_TOKEN: ${{ secrets.KWIKQA_TOKEN }}
    GITHUB_TOKEN: ${{ github.token }}

- if: failure()
  uses: actions/upload-artifact@v7
  with:
    name: kwikqa-report
    path: kwikqa-report.html

3. Azure Pipelines

Run kwikqa ci as a step that runs only when an earlier step failed, and publish the report:

# azure-pipelines.yml (excerpt)
steps:
  - script: npx playwright test
  - script: npx kwikqa ci
    condition: failed()
    env:
      KWIKQA_TOKEN: $(KWIKQA_TOKEN)
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
      KWIKQA_SLACK_WEBHOOK_URL: $(KWIKQA_SLACK_WEBHOOK_URL) # optional
  - publish: kwikqa-report.html
    artifact: kwikqa-report
    condition: failed()

Store KWIKQA_TOKEN as a secret variable of the pipeline. Azure Pipelines gives a step a secret, and System.AccessToken, only through the step's env. The test step's failure keeps the build failed.

  • Build summary. The summary is a "KwikQA" section on the run's Extensions tab.
  • Pull-request thread. On an Azure Repos pull request, the summary is also one thread. A re-run of the same pipeline job updates it. It's created closed, so it never holds up the pull request under a comment-resolution policy.
  • Permission. The thread needs "Contribute to pull requests" for the pipeline's build service. Set it in Project settings → Repositories → your repository → Security, for "project Build Service (organization)". Without it, the log names the identity that was refused.
  • Other repositories. A pipeline for a GitHub or Bitbucket repository gets the build summary only.
  • Forks. A fork's build gets no secrets unless the pipeline allows it. KwikQA analyzes it locally and skips the thread.

4. GitLab CI/CD

# .gitlab-ci.yml (excerpt)
e2e:
  image: mcr.microsoft.com/playwright:v1.64.1-noble
  script:
    - npm ci
    - npx playwright test || { npx kwikqa ci; exit 1; }
  artifacts:
    when: always
    expose_as: KwikQA report
    paths:
      - kwikqa-report.html

The exit 1 keeps the job failed: kwikqa ci exits with the analysis's code, which is usually 0.

  • Variables. Add KWIKQA_TOKEN and GITLAB_TOKEN under Settings → CI/CD → Variables. Mark them masked, and leave them unprotected, or a feature branch's pipeline won't get them.
  • The token. GitLab's own CI_JOB_TOKEN can't write notes, so GITLAB_TOKEN is a token of your own, with at least the Reporter role. The note is posted as that token's user.
    • A fine-grained personal access token needs only your project, with Work Item read, create, and update. GitLab's merge-request notes come under Work Item. Make it a dedicated user's if you can.
    • Otherwise, a token with the api scope: a legacy personal access token, or a project or group access token (Premium or Ultimate on GitLab.com, any tier when self-managed).
  • Merge-request note. The summary is printed in the job's log and posted as one note on the merge request. A re-run of the same job updates it. A merge request pipeline names its merge request. A branch pipeline gets a note when its branch has exactly one open merge request in the project.
  • The report. expose_as puts a "KwikQA report" link on the merge request.
  • Forks. A fork's merge request pipeline runs in the fork, without your project's variables. KwikQA analyzes it locally and skips the note.

5. Jenkins

In a Pipeline, run kwikqa ci when the tests fail, with its tokens bound only around it. The agent needs Node.js 22, 24, 26, or later.

// Jenkinsfile (excerpt)
stage('e2e') {
  steps {
    sh 'npm ci'
    sh 'npx playwright test'
  }
  post {
    failure {
      withCredentials([
        string(credentialsId: 'kwikqa-token', variable: 'KWIKQA_TOKEN'),
        string(credentialsId: 'github-token', variable: 'GITHUB_TOKEN'),
      ]) {
        sh 'npx kwikqa ci'
      }
      archiveArtifacts artifacts: 'kwikqa-report.html', allowEmptyArchive: true
    }
  }
}
  • Summary. It's printed in the build's console log, with a link to the build, where the archived report is.
  • Pull-request comment. In a multibranch project, a pull request's build knows its pull request from the branch source. Jenkins has no pull-request API of its own, so KwikQA comments through the pull request's host, and a rebuild updates the comment:
    • GitHub, including GitHub Enterprise, with GITHUB_TOKEN: a fine-grained token with Pull requests read and write on the repository, or a classic token with the repo scope. A GitHub App credential works too, bound with usernamePassword(credentialsId: 'github-app', usernameVariable: 'GITHUB_APP', passwordVariable: 'GITHUB_TOKEN'). Its token lasts an hour, so bind it around kwikqa ci, as above.
    • GitLab, with GITLAB_TOKEN, a token as for GitLab CI/CD.
    • Bitbucket and other hosts get no comment yet.
  • Forks. Jenkins gives a fork's pull request build the credentials its Jenkinsfile binds, and that build runs the fork's code, its package.json included. If you build pull requests from forks you don't trust, don't bind tokens in their builds. The branch source's trust setting only decides whose Jenkinsfile runs.
  • The report. Jenkins serves an archived page with its script and styles blocked, and the report stays readable that way. For its styles and filters too, set a Resource Root URL on another host name, under Manage Jenkins → System → "Serve resource files from another domain". Don't loosen Jenkins' Content-Security-Policy instead.

6. Fix pull requests

In GitHub Actions, Azure Pipelines, and GitLab CI/CD, kwikqa fix can repair a failing test in the job and open the repair as a draft pull request, a merge request on GitLab. Nothing lands until a person reviews and merges it. Opt in by committing a policy in kwikqa.config.json:

"ci": { "fix": { "pullRequests": "draft", "retry": false } }

Then run npx kwikqa fix after kwikqa ci, in the same job, with the host's token. Fetch the full history, so the diagnosis can see the commit that changed your app.

GitHub Actions

permissions:
  contents: write # the fix branch
  pull-requests: write # the comment and the fix pull request

steps:
  - uses: actions/checkout@v7
    with:
      fetch-depth: 0
  - run: npm ci && npx playwright install --with-deps
  - run: npx playwright test

  - if: failure()
    uses: kwikqa/action@v1
    with:
      kwikqa-token: ${{ secrets.KWIKQA_TOKEN }}
  - if: failure()
    run: npx kwikqa fix
    env:
      KWIKQA_TOKEN: ${{ secrets.KWIKQA_TOKEN }}
      GITHUB_TOKEN: ${{ github.token }}

Also turn on "Allow GitHub Actions to create and approve pull requests" in the repository's or organization's Actions settings. GitHub doesn't start workflows for a pull request the job's GITHUB_TOKEN opens: close and reopen it, or push to it, to run them, or give kwikqa fix a token from your own GitHub App (actions/create-github-app-token) as GITHUB_TOKEN. GitHub Free has no draft pull requests in private repositories, and KwikQA opens only drafts.

Azure Pipelines

- checkout: self
  fetchDepth: 0
# … your tests and the kwikqa ci step …
- script: npx kwikqa fix
  condition: failed()
  env:
    KWIKQA_TOKEN: $(KWIKQA_TOKEN)
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

For an Azure Repos repository. The build service, "project Build Service (organization)", needs "Contribute", "Create branch", and "Contribute to pull requests" on the repository.

GitLab CI/CD

variables:
  GIT_DEPTH: "0"
e2e:
  script:
    - npm ci
    - npx playwright test || { npx kwikqa ci; npx kwikqa fix; exit 1; }

GITLAB_TOKEN needs the api scope and at least the Developer role, to push the fix branch. A project access token keeps your own name off the merge request. A fine-grained personal access token isn't verified for this yet.

What it does

  • Repairs and validates on the runner. It re-runs the failing test once to confirm the failure, asks for one repair, and applies it to the runner's own checkout. The repaired test must pass its validation runs with retries off: three times in a row, or once for a shared Maven, Gradle, pytest, or Cucumber-JVM repair. A failure is rolled back, and retried once only with "retry": true. It reuses the diagnosis kwikqa ci made, and one CI fix counts as one fix.
  • Opens one draft. A new branch, kwikqa/fix-<id>, named after the failure group, with one commit built through your code host's API, so the runner's git writes nothing. For a pull request's run it goes into that pull request's branch; for any other branch's run, such as a nightly suite on main, into that branch. It's labeled kwikqa, and its description gives the diagnosis, the repair's intent, the validation, and which models were asked.
  • Exactly the validated change. If the target branch changed the repaired file since the run, nothing is committed. KwikQA never moves a branch it didn't create, and never merges, approves, or marks a pull request ready.
  • One per failure group. While a group's fix is open, a later run of the same failure asks for no repair and costs no fix; it updates one note on the draft instead. A kwikqa/fix- branch left after a closed pull request also holds the group back until you delete it.
  • Only under the committed policy. The policy counts only if kwikqa.config.json is exactly as committed, and only for the runner's checkout. Each request, re-run, and patch is printed in the log with what approved it.
  • When it can't. A fork's pull request, a run on a tag, and a job without the token are refused before any request, so they cost nothing. On Hobby, a fix pull request is refused before any request. Pro and BYOK open them on one repository of your own, and the team plans on organization repositories. On Jenkins, fix still needs a terminal.

7. Slack and Microsoft Teams

KwikQA posts the run to incoming webhooks you store as CI secrets. It keeps no webhook anywhere, and never prints one. Alerts come with the team plans: Teams, BYOK Teams, Enterprise, and BYOK Enterprise.

  • Slack. Create an incoming webhook for the channel, and store its URL as KWIKQA_SLACK_WEBHOOK_URL.
  • Teams. In the channel, add the Workflows template "Post to a channel when a webhook request is received", and store its URL as KWIKQA_TEAMS_WEBHOOK_URL. Office 365 connector webhooks were retired in May 2026.

Pass them to the Action as slack-webhook-url and teams-webhook-url, or set them as environment variables for npx kwikqa ci. An empty variable counts as unset, so check that the secret has a value if nothing is posted.

A message names the repository and branch, links the pull request or commit and the run, and gives the counts. Then it lists each failure group by its category, where it fails, its size, and its owner, and gives the flaky count. It holds no error text, evidence, or source code: a channel is a wider audience than a build log, so the message says where to look, and the report shows what failed.

KwikQA posts only when tests failed, unless you pass --notify always (the Action's notify input). A failed post is retried once and reported in the log.

Other CI systems

On other CI systems, npx kwikqa ci writes the failure report and posts to Slack and Teams, but writes no summary or comment yet. Keep kwikqa-report.html as a build artifact.

npx playwright test || { npx kwikqa ci; exit 1; }

The exit 1 keeps the build failed: kwikqa ci exits with the analysis's code, which is usually 0.

Sharded suites

A suite split across several CI jobs can be analyzed in one of two ways.

  • One analysis after the shards, on every plan. Download every shard's report into one directory, and run npx kwikqa ci --report <directory> in a final job. KwikQA reads every report file in the directory as one run, so a cause the shards share is one failure group and one analysis. Playwright's blob reports are merged with npx playwright merge-reports first. On GitHub Actions, actions/download-artifact with merge-multiple: true puts the files together.
  • Each shard analyzes itself, on the Enterprise plans. When every shard job runs kwikqa ci, each job asks the service which of its failure groups another job of the same run already diagnoses. A group that another job has isn't sent or counted again. The summary says so, and links the run where the CI system gives its address. KwikQA takes the run and the job from the CI system. Where a job's own id isn't unique per shard, for example a matrix on self-hosted runners, set KWIKQA_SHARD to a value that is, such as ${{ matrix.shard }}. The service keeps only keyed hashes of the run, the job, and each group, for 24 hours.

Plan rules in CI

Paid plans are coming soon. These rules apply once the paid plans open. Until then, CI runs aren't limited by plan.

A plan sets which repositories a CI token can serve, how many CI jobs run at once, and, on the team plans, how many people's pull requests count in a month. Plans and usage has every plan's CI limits.

  • Hobby analyzes CI runs locally, with nothing sent. Hosted analysis in CI comes with Pro and the team plans.
  • Pro and BYOK run hosted CI on one repository of your own a month. Any other repository, such as an organization's, needs a team plan.
  • Teams and BYOK Teams run CI on organization repositories, up to 3 repositories a month for the whole team.
  • Enterprise and BYOK Enterprise run CI on organization repositories with no repository limit.
  • Concurrent CI jobs: 1 on Pro and BYOK; 2 on Teams and BYOK Teams; 10 on Enterprise and BYOK Enterprise.
  • Seats: on the team plans, a seat is an active committer, a person whose pull request ran KwikQA in CI during the month. A run without a pull request counts for the token's person, or for a company or team CI token, as one seat. Past the seats, the month gets a grace of one extra person for every full ten seats, and at least one. Past that, a new person's pull requests are refused until you add seats.
  • Draft pull requests from kwikqa fix: Pro and BYOK on one repository of your own, and the team plans on organization repositories. Hobby has none.
  • Slack and Teams alerts come with the team plans.
  • Which repository and person: the CLI says where the job runs. On GitHub Actions and GitLab.com, it also sends the job's identity token, which the CI system signs, and the service takes the repository and the person who triggered a pull request's run from it. On GitHub Actions, grant id-token: write to the job that runs KwikQA, as above. On GitLab, declare the token in that job: id_tokens: { KWIKQA_ID_TOKEN: { aud: "https://kwikqa.com" } }. A CI token is for CI jobs: once the paid plans open, it is refused outside a CI system KwikQA knows.

When a run is refused

A run that a plan rule refuses isn't diagnosed. The job prints Not diagnosed: and the message, then Analyzing locally. The analysis still runs on the runner, the local report and summary are written, and the refused request counts toward nothing. These are the messages, with the plan's name (Pro or BYOK, Teams or BYOK Teams, and so on) and the repository or author filled in. The CLI refuses the first two before any preview, and the service gives its own wording for those two cases and for the limits:

MessageRefused byWhat to do
Your plan analyzes CI runs locally, with nothing sent. Hosted analysis in CI comes with Pro and the team plans: see npx kwikqa upgrade.The CLI, before any previewThe run is analyzed locally already. For hosted analysis in CI, upgrade to Pro (one repository of your own) or to a team plan.
Your plan runs KwikQA in CI on a repository of your own. owner/name belongs to someone else, so it needs a team plan: see npx kwikqa upgrade.The CLI, before any previewRun the job on a repository of your own, or use a team plan for the organization's repository.
The Hobby plan analyzes CI runs locally, with nothing sent. Hosted analysis in CI comes with Pro and the team plans: see npx kwikqa upgrade.The serviceThe run is analyzed locally already. For hosted analysis in CI, upgrade to Pro (one repository of your own) or to a team plan.
The Pro plan runs KwikQA in CI on a repository of your own. owner/name belongs to someone else, so it needs a team plan: see npx kwikqa upgrade.The serviceRun the job on a repository of your own, or use a team plan for the organization's repository.
The Pro plan runs KwikQA in CI on one repository a month, and this month's is another. The team plans run it on more: see npx kwikqa upgrade.The serviceWait for the month to end, or move to a team plan.
The Teams plan runs KwikQA in CI on up to 3 repositories a month, and this month's are already in use. Enterprise has no limit: see npx kwikqa upgrade.The serviceWait for the month to end, or move to Enterprise or BYOK Enterprise, which have no repository limit.
This month n people's pull requests have run KwikQA in CI, and the plan has s seats. Add seats to analyze this author's pull requests: see the dashboard's Billing.The serviceAdd seats in Billing. Authors already counted this month keep working.
The Teams plan runs 2 KwikQA CI jobs at a time, and that many are running. Try again when one finishes, or see npx kwikqa upgrade.The serviceWait for a running job to finish, or move to a plan with more concurrent jobs.

It never fails your build

A summary, comment, or message that can't be published is reported in the log and skipped. Without a KwikQA token, with a revoked one, or when a plan rule refuses the run, the run is analyzed locally, and the summary says why it wasn't diagnosed. kwikqa ci exits with the analysis's own exit code.

Without an account

Without a token, kwikqa ci sends nothing to KwikQA. It groups the failures, writes the report and the summary, and says the run wasn't diagnosed. npx kwikqa analyze --local does the same analysis without publishing.

Tips

  • Use a separate token for each CI system so you can revoke one without affecting the others.
  • The jobs of one GitHub Actions matrix share a job name, so they share one comment. Analyze a sharded run once, after merging its reports, unless you're on an Enterprise plan and use the shard option in Sharded suites. Give each job its own artifact-name.
  • A diagnosis made in CI isn't reused by a later fix on your machine, because the run and checkout differ. When a failure can be repaired, the log says so: let kwikqa fix repair it in the job under a committed policy, or download the report artifact and run analyze and fix locally.
  • For the fullest Cypress or WebdriverIO diagnosis, run npx kwikqa analyze --capture on your machine: it re-runs the failing specs and records each failure's page, which a CI report doesn't contain.