Docs menu

Getting started

Install and set up

1. Install

Add KwikQA as a development dependency of the project that runs your Playwright, Cypress, or WebdriverIO tests. For a JUnit, TestNG, pytest, or other framework's report, you need only Node.js to diagnose the run: run it with npx kwikqa. Repairs of those runs need the tools in Supported frameworks.

$ npm i -D kwikqa

Check it with npx kwikqa --version. These docs use npm's commands. With another package manager, install and run KwikQA with its own:

Package managerInstallRun
npmnpm i -D kwikqanpx kwikqa analyze
pnpmpnpm add -D kwikqapnpm exec kwikqa analyze
Yarnyarn add -D kwikqayarn kwikqa analyze
Bunbun add -d kwikqabunx kwikqa analyze

Monorepos

Install KwikQA in the package that holds your tests, and run it from that package's folder, where its kwikqa.config.json goes. KwikQA finds that package's Playwright, Cypress, or WebdriverIO in its own node_modules or in the node_modules of a folder above it, up to the repository root. So the root's node_modules works, but a copy installed only in another package doesn't. It never runs one from outside the repository, so pnpm's global virtual store (enableGlobalVirtualStore) isn't supported.

2. Point KwikQA at your runs

KwikQA reads the report your test runner writes. Pick your framework.

Playwright

Add Playwright's JSON reporter next to your usual one, and keep traces for failures so the diagnosis can cite what the browser actually did:

// playwright.config.ts
export default defineConfig({
  reporter: [["list"], ["json", { outputFile: "test-results/kwikqa-report.json" }]],
  use: { trace: "retain-on-failure" },
});

Cypress

Use KwikQA's reporter, which ships in the package. It keeps every retry attempt, where each test is declared, and each error's code frame:

// cypress.config.ts
export default defineConfig({
  reporter: "node_modules/kwikqa/dist/cypress-reporter.cjs",
  e2e: { /* ... */ },
});

Already writing mochawesome or JUnit reports? KwikQA reads those too: npx kwikqa analyze --report mochawesome-report. They record only each test's final attempt, so flakiness shows only with KwikQA's reporter.

WebdriverIO

Add KwikQA's reporter next to your own. It keeps the full describe path of every test, which WebdriverIO's JSON and JUnit reporters flatten:

// wdio.conf.ts
import KwikqaReporter from "kwikqa/wdio-reporter";

export const config = {
  reporters: ["spec", [KwikqaReporter, {}]],
  // ...
};

@wdio/json-reporter and @wdio/junit-reporter output works too: point --report at their output directory. WebdriverIO with Mocha or Jasmine is supported; Cucumber is not yet.

Cypress and WebdriverIO keep no trace of a failure. Run npx kwikqa analyze --capture and KwikQA re-runs the failing specs with your own config, adding its reporter and failure capture, which records each failed attempt's DOM, command log, console, and network. Nothing in your project is edited.

Supported frameworks

KwikQA repairs and validates end-to-end browser tests, and only those. A test is end-to-end when its code reaches a browser-automation library: Selenium, Selenide, Appium, or Playwright for Java, and Selenium, Playwright, Appium, SeleniumBase, or Splinter for Python. KwikQA follows the code a test uses: its imports, the fully qualified names it writes, the fixtures and page objects it requests, the modules a pytest_plugins list names, the modules a pytest pythonpath makes importable, and, for pytest, a browser plugin's fixture when the project's own virtual environment installs that plugin. The plugins are pytest-playwright, pytest-selenium, SeleniumBase, and pytest-splinter. Gradle's testFixtures and integrationTest source sets are read with the main and test sources.

These runs are diagnosed, repaired, and validated:

  • Playwright Test, Cypress, and WebdriverIO (Mocha or Jasmine). They are browser runners, so the rule above doesn't narrow them.
  • Java, with Maven or Gradle, and TestNG or JUnit 5, when the tests are end-to-end browser tests.
  • pytest, when the tests are end-to-end browser tests.
  • Cucumber-JVM, with Maven or Gradle, when a scenario reaches a browser library through its steps and the hooks that failed in it.

Everything else is report-only: KwikQA diagnoses it, and never repairs or validates it. That includes unit tests in JUnit, TestNG, and pytest, because a repair would change what they accept. kwikqa check doesn't judge a change to a unit test. When KwikQA can't confirm that a test reaches a browser library, it treats the test as a unit test.

Maven and Gradle runs need a JDK already in JAVA_HOME. With the project's Maven or Gradle wrapper, its distribution must already be unpacked under MAVEN_USER_HOME (or ~/.m2) or GRADLE_USER_HOME (or ~/.gradle). If it isn't, run ./mvnw --version or ./gradlew --version once yourself, then validate again. pytest runs need the project's .venv or venv. KwikQA never installs or downloads any of these, and Maven and Gradle validation is unavailable on Windows.

Other test frameworks

KwikQA also reads the reports these write, and diagnoses the run:

  • Vitest (--reporter=junit), Jest (jest-junit), and Mocha (mocha-junit-reporter);
  • pytest (--junitxml);
  • Playwright's own junit reporter;
  • JUnit and TestNG through Maven Surefire or Failsafe (target/surefire-reports), including Selenium tests;
  • JUnit and TestNG through Gradle (build/test-results/test), including the test-retry plugin's retries;
  • TestNG's own testng-results.xml;
  • Allure results (allure-results) from TestNG, JUnit, Playwright, and pytest;
  • CTRF reports, from Playwright, Jest, pytest, and other tools that write them;
  • Cucumber's JSON report, from cucumber-js (--format json:<file>) and Cucumber-JVM (the json plugin).

Point --report at the file or its directory:

$ npx kwikqa analyze --report target/surefire-reports --html

You get failure groups, a diagnosis, and the failure report. Repairs apply only to the runs named under Supported frameworks, read from the reports a repair can validate, and for Java, Python, and Cucumber-JVM runs only to their end-to-end browser tests; for any other run, fix says so before it sends anything. --capture works only for Playwright, Cypress, and WebdriverIO. For Playwright repairs, use its JSON reporter as shown above.

Some details depend on the reporter:

  • TestNG. testng-results.xml keeps retried attempts and failed configuration methods, which Surefire's XML of the same run doesn't correlate. Under Maven, add TestNG's XML reporter to Surefire's configuration (the listener property set to org.testng.reporters.XMLReporter), then point --report at test-output.
  • Allure. Allure adds each run's results to its directory, and KwikQA reads the directory as one run, so clean it between runs: mvn clean, or pytest's --clean-alluredir. Otherwise an earlier run's results read as retries.
  • Cucumber. Its JSON keeps only each scenario's final attempt, so a scenario that passed on a retry reads as passed. Cucumber-JVM 8 needs a JSON library on the test class path to write it, such as tools.jackson.core:jackson-databind.
  • Gradle. Gradle names a JUnit parameterized test by its display name, which includes its arguments. To keep test data out of KwikQA's titles, set junit.jupiter.params.displayname.default = {displayName}[{index}] in junit-platform.properties.
  • Jest and pytest. jest-junit records no test file unless you set addFileAttribute: true, and pytest records each test's line only with -o junit_family=xunit1.
  • Other producers. JUnit XML from a tool KwikQA doesn't recognize is still read and diagnosed, as an unidentified framework.

Where KwikQA looks

--report takes a report file or a directory of one run's reports. Without it, KwikQA tries the PLAYWRIGHT_JSON_OUTPUT_NAME and KWIKQA_REPORT_DIR environment variables, then test-results/kwikqa-report.json, test-results/kwikqa/ (KwikQA's Cypress and WebdriverIO reporters), playwright-report/results.json, test-results/results.json, mochawesome-report/, cypress/reports/, cypress/results/, results.json, and report.json. A JUnit report that doesn't say which framework wrote it needs --framework cypress or --framework webdriverio.

3. Create the repair policy

$ npx kwikqa init

init looks at the files git tracks and proposes a kwikqa.config.json listing which files KwikQA may edit. It writes the file only if you agree, and offers to add .kwikqa/ (local run records) to .gitignore. Review and commit the config like any other code. Configuration explains every field.

You can run analyze without kwikqa.config.json. Only fix requires it.

4. Sign in

$ npx kwikqa login
Open https://kwikqa.com/device/ and enter the code: WDJB-MJHT
Opened that page in your browser, with the code filled in. Check it matches, then approve it.
Signed in as octocat.

The page opens in your browser with the code filled in; otherwise, open it yourself and enter the code. Sign in to the website if you aren't already, with GitHub for a personal account or your work email for a company account, check the code, and choose Approve sign-in. The CLI on another machine, such as a remote server, signs in the same way: open the page on any device. Approve only a code you started yourself. The Hobby plan needs no payment details. The CLI stores its token in your user configuration directory, never in the project. Check your plan and usage with:

$ npx kwikqa status

5. Run it

After a run with failures, continue with your first repair. To try KwikQA without an account or sending anything, run npx kwikqa analyze --local: it builds the evidence summary on your machine only.