Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
834 changes: 834 additions & 0 deletions Bunoshfile.js

Large diffs are not rendered by default.

54 changes: 3 additions & 51 deletions docs/ai.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
permalink: /ai
title: Testing with AI 🪄
title: Testing with AI
---

# 🪄 Testing with AI
# Testing with AI

**CodeceptJS is the first open-source test automation framework with AI** features to improve the testing experience. CodeceptJS uses AI provider like OpenAI or Anthropic to auto-heal failing tests, assist in writing tests, and more...

Think of it as your testing co-pilot built into the testing framework

> 🪄 **AI features for testing are experimental**. AI works only for web based testing with Playwright, WebDriver, etc. Those features will be improved based on user's experience.
> This is guide on using AI features inside CodeceptJS. To control CodeceptJS via AI Agents, see [Agentic Testing Guide](/agents/).

## How AI Improves Automated Testing

Expand Down Expand Up @@ -164,50 +164,6 @@ The AI SDK supports 20+ providers including:

See [AI SDK Providers](https://ai-sdk.dev/docs/foundations/providers-and-models) for complete list and configuration details.

## Writing Tests with AI Copilot

If AI features are enabled when using [interactive pause](/basics/#debug) with `pause()` command inside tests:

For instance, let's create a test to try ai features via `gt` command:

```
npx codeceptjs gt
```

Name a test and write the code. We will use `Scenario.only` instead of Scenario to execute only this exact test.

```js
Feature('ai')

Scenario.only('test ai features', ({ I }) => {
I.amOnPage('https://getbootstrap.com/docs/5.1/examples/checkout/')
pause()
})
```

Now run the test in debug mode with AI enabled:

```
npx codeceptjs run --debug --ai
```

When pause mode started you can ask GPT to fill in the fields on this page. Use natural language to describe your request, and provide enough details that AI could operate with it. It is important to include at least a space char in your input, otherwise, CodeceptJS will consider the input to be JavaScript code.

```
I.fill checkout form with valid values without submitting it
```

![](/img/fill_form_1.png)

GPT will generate code and data and CodeceptJS will try to execute its code. If it succeeds, the code will be saved to history and you will be able to copy it to your test.

![](/img/fill_form2.png)

This AI copilot works best with long static forms. In the case of complex and dynamic single-page applications, it may not perform as well, as the form may not be present on HTML page yet. For instance, interacting with calendars or inputs with real-time validations (like credit cards) can not yet be performed by AI.

Please keep in mind that GPT can't react to page changes and operates with static text only. This is why it is not ready yet to write the test completely. However, if you are new to CodeceptJS and automated testing AI copilot may help you write tests more efficiently.

> 👶 Enable AI copilot for junior test automation engineers. It may help them to get started with CodeceptJS and to write good semantic locators.

## Self-Healing Tests

Expand Down Expand Up @@ -409,9 +365,7 @@ ai: {

CodeceptJS uses three main prompts for AI features:

- `writeStep` - generates test code in interactive pause mode
- `healStep` - suggests fixes for failing tests
- `generatePageObject` - creates page objects from HTML

To customize a prompt, generate it using:

Expand Down Expand Up @@ -452,9 +406,7 @@ You can also override prompts programmatically in config:
```js
ai: {
prompts: {
writeStep: (html, input) => [{ role: 'user', content: 'As a test engineer...' }]
healStep: (html, { step, error, prevSteps }) => [{ role: 'user', content: 'As a test engineer...' }]
generatePageObject: (html, extraPrompt = '', rootLocator = null) => [{ role: 'user', content: 'As a test engineer...' }]
}
}
```
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/ApiDataFactory.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: ApiDataFactory
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## ApiDataFactory
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/Appium.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: Appium
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## Appium
Expand Down
9 changes: 9 additions & 0 deletions docs/helpers/Detox.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ title: Detox
# Detox


---
permalink: /helpers/Detox
sidebar: auto
title: Detox
---

# Detox


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## Detox
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/FileSystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: FileSystem
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## FileSystem
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/GraphQL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: GraphQL
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## GraphQL
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/GraphQLDataFactory.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: GraphQLDataFactory
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## GraphQLDataFactory
Expand Down
1 change: 1 addition & 0 deletions docs/helpers/JSONResponse.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: JSONResponse
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## JSONResponse
Expand Down
29 changes: 13 additions & 16 deletions docs/helpers/Playwright.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ sidebar: auto
title: Playwright
---


<!-- Generated by documentation.js. Update this documentation by updating the source code. -->

## Playwright
Expand Down Expand Up @@ -46,14 +47,14 @@ Type: [object][6]
* `url` **[string][9]?** base url of website to be tested
* `browser` **(`"chromium"` | `"firefox"` | `"webkit"` | `"electron"`)?** a browser to test on, either: `chromium`, `firefox`, `webkit`, `electron`. Default: chromium.
* `show` **[boolean][27]?** show browser window.
* `restart` **([string][9] | [boolean][27])?** restart strategy between tests. Possible values:* 'context' or **false** - restarts [browser context][45] but keeps running browser. Recommended by Playwright team to keep tests isolated.
* `restart` **([string][9] | [boolean][27])?** restart strategy between tests. Possible values:* 'context' or **false** - restarts [browser context][44] but keeps running browser. Recommended by Playwright team to keep tests isolated.
* 'session' or 'keep' - keeps browser context and session, but cleans up cookies and localStorage between tests. The fastest option when running tests in windowed mode. Works with `keepCookies` and `keepBrowserState` options. This behavior was default before CodeceptJS 3.1
* `timeout` **[number][18]?** * [timeout][46] in ms of all Playwright actions .
* `timeout` **[number][18]?** * [timeout][45] in ms of all Playwright actions .
* `disableScreenshots` **[boolean][27]?** don't save screenshot on failure.
* `emulate` **any?** browser in device emulation mode.
* `video` **[boolean][27]?** enables video recording for failed tests; videos are saved into `output/videos` folder
* `keepVideoForPassedTests` **[boolean][27]?** save videos for passed tests; videos are saved into `output/videos` folder
* `trace` **[boolean][27]?** record [tracing information][47] with screenshots and snapshots.
* `trace` **[boolean][27]?** record [tracing information][46] with screenshots and snapshots.
* `keepTraceForPassedTests` **[boolean][27]?** save trace for passed tests.
* `fullPageScreenshots` **[boolean][27]?** make full page screenshots on failure.
* `uniqueScreenshotNames` **[boolean][27]?** option to prevent screenshot override if you have scenarios with the same name in different suites.
Expand All @@ -73,13 +74,13 @@ Type: [object][6]
* `chromium` **[object][6]?** pass additional chromium options
* `firefox` **[object][6]?** pass additional firefox options
* `electron` **[object][6]?** (pass additional electron options
* `channel` **any?** (While Playwright can operate against the stock Google Chrome and Microsoft Edge browsers available on the machine. In particular, current Playwright version will support Stable and Beta channels of these browsers. See [Google Chrome & Microsoft Edge][48].
* `ignoreLog` **[Array][10]<[string][9]>?** An array with console message types that are not logged to debug log. Default value is `['warning', 'log']`. E.g. you can set `[]` to log all messages. See all possible [values][49].
* `channel` **any?** (While Playwright can operate against the stock Google Chrome and Microsoft Edge browsers available on the machine. In particular, current Playwright version will support Stable and Beta channels of these browsers. See [Google Chrome & Microsoft Edge][47].
* `ignoreLog` **[Array][10]<[string][9]>?** An array with console message types that are not logged to debug log. Default value is `['warning', 'log']`. E.g. you can set `[]` to log all messages. See all possible [values][48].
* `ignoreHTTPSErrors` **[boolean][27]?** Allows access to untrustworthy pages, e.g. to a page with an expired certificate. Default value is `false`
* `bypassCSP` **[boolean][27]?** bypass Content Security Policy or CSP
* `highlightElement` **[boolean][27]?** highlight the interacting elements. Default: false. Note: only activate under verbose mode (--verbose).
* `recordHar` **[object][6]?** record HAR and will be saved to `output/har`. See more of [HAR options][3].
* `testIdAttribute` **[string][9]?** locate elements based on the testIdAttribute. See more of [locate by test id][50].
* `testIdAttribute` **[string][9]?** locate elements based on the testIdAttribute. See more of [locate by test id][49].
* `storageState` **([string][9] | [object][6])?** Playwright storage state (path to JSON file or object)
passed directly to `browser.newContext`.
If a Scenario is declared with a `cookies` option (e.g. `Scenario('name', { cookies: [...] }, fn)`),
Expand Down Expand Up @@ -2794,8 +2795,6 @@ Returns **void** automatically synchronized promise through #recorder

### waitForVisible

This method accepts [React selectors][44].

Waits for an element to become visible on a page (by default waits for 1sec).
Element can be located by CSS or XPath.

Expand Down Expand Up @@ -2959,16 +2958,14 @@ Returns **void** automatically synchronized promise through #recorder

[43]: https://playwright.dev/docs/api/class-page#page-wait-for-url

[44]: https://codecept.io/react

[45]: https://playwright.dev/docs/api/class-browsercontext
[44]: https://playwright.dev/docs/api/class-browsercontext

[46]: https://playwright.dev/docs/api/class-page#page-set-default-timeout
[45]: https://playwright.dev/docs/api/class-page#page-set-default-timeout

[47]: https://playwright.dev/docs/trace-viewer
[46]: https://playwright.dev/docs/trace-viewer

[48]: https://playwright.dev/docs/browsers/#google-chrome--microsoft-edge
[47]: https://playwright.dev/docs/browsers/#google-chrome--microsoft-edge

[49]: https://playwright.dev/docs/api/class-consolemessage#console-message-type
[48]: https://playwright.dev/docs/api/class-consolemessage#console-message-type

[50]: https://playwright.dev/docs/locators#locate-by-test-id
[49]: https://playwright.dev/docs/locators#locate-by-test-id
Loading