Playwright is a popular test automation tool that offers a lot of reporting options for its users such as built-in reporters, custom reporters, and support for integrating third-party reporters. The Playwright’s default in-built reporter is the list reporter. However, when running tests via the CI tool, Playwright will switch to the Dot reporter by default. There is also a good reason why the Dot Reporter is chosen as the default Playwright Reporting option during execution in Continuous Integration tools. We have even made a YouTube video explaining it and recommend you check it out.
Like any tool or feature, there will always be a few drawbacks. Based on our experience of working with Playwright while delivering automation testing services to our clients, we were able to overcome these drawbacks with a few workarounds. So in this blog, we will be sharing how you can customize the Dot reporter to address these drawbacks and enhance your Playwright reporting. But before that, let’s take a look at what the disadvantages are.
Disadvantages of Dot Reporter:
- During the execution process, the Dot Reporter will not display the number of tests completed. So you’ll have to manually count if you want to get the total number of tests executed.
- In the event of a failure, an ‘F’ will appear in red. But the issue is that it will not indicate which specific test has failed during execution.
Customization of Dot Reporter:
As stated earlier, Playwright reporting has built-in options and customizing capabilities as well. So, let’s delve into the customization aspect to address the disadvantages of Dot Reporter. If you prefer to watch the entire step-by-step tutorial as a video, you can check out our video covering the same. Or you can prefer to continue reading as well.
Step 1: Creating Reporter Listener Class
- Create a folder by the name ‘utils’ inside your project directory.
- Create a TypeScript file using the name ‘CustomReporter’ with the below code
import type {Reporter, FullConfig, Suite, TestCase, TestResult, FullResult} from '@playwright/test/reporter';
class CustomReporter implements Reporter {
}
export default CustomReporter;
Step 2: Configure Reporter Listener in Playwright Config file
- Open the playwright.config.ts file
- Add the reporter listener file that you created in Step 1 in the Playwright config file
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests',
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: './utils/CustomReporter.ts',
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
/* Base URL to use in actions like `await page.goto('/')`. */
// baseURL: 'http://127.0.0.1:3000',
/* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
trace: 'on-first-retry',
},
/* Configure projects for major browsers */
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
{
name: 'webkit',
use: { ...devices['Desktop Safari'] },
},
],
});
Step 3: Declare & Initialize Properties
- In the CustomReporter class, add three class properties
- totalTests-To hold total tests in the test suite.
- totalTestsExecuted-To count the number of tests that have been executed in the current execution.
- noOfTestsPerLine-To count the number of test statuses or results to be shown in a line.
- Initialize the properties in the constructor
class CustomReporter implements Reporter {
totalTests: number;
noOfTestsPerLine: number
totalTestsExecuted: number
constructor() {
this.totalTests=0
this.noOfTestsPerLine=0
this.totalTestsExecuted=0
}
}
Step 4: Add the onBegin method
- Add the onBegin method.
- Save the total tests to be executed in the totalTests variable.
class CustomReporter implements Reporter {
totalTests: number;
noOfTestsPerLine: number
totalTestsExecuted: number
constructor() {
this.totalTests=0
this.noOfTestsPerLine=0
this.totalTestsExecuted=0
}
onBegin(config: FullConfig, suite: Suite) {
this.totalTests = suite.allTests().length;
console.log(`Executing ${this.totalTests} test(s)`);
}
}
Step 5: Add the printTotalExecuted method
This method will be called to print how many tests have been executed against the total tests.
class CustomReporter implements Reporter {
totalTests: number;
noOfTestsPerLine: number
totalTestsExecuted: number
constructor() {
this.totalTests=0
this.noOfTestsPerLine=0
this.totalTestsExecuted=0
}
onBegin(config: FullConfig, suite: Suite) {
this.totalTests = suite.allTests().length;
console.log(`Executing ${this.totalTests} test(s)`);
}
printTotalExecuted(){
process.stdout.write(`[${this.totalTestsExecuted}/${this.totalTests}]\n`);
this.noOfTestsPerLine=0
}
}
Step 6: Add the onTestEnd method
After the execution of each test, the onTestEnd method will be invoked.
class CustomReporter implements Reporter {
totalTests: number;
noOfTestsPerLine: number
totalTestsExecuted: number
constructor() {
this.totalTests=0
this.noOfTestsPerLine=0
this.totalTestsExecuted=0
}
onBegin(config: FullConfig, suite: Suite) {
this.totalTests = suite.allTests().length;
console.log(`Executing ${this.totalTests} test(s)`);
}
printTotalExecuted(){
process.stdout.write(`[${this.totalTestsExecuted}/${this.totalTests}]\n`);
this.noOfTestsPerLine=0
}
onTestEnd(test: TestCase, result: TestResult) {
}
}
Step 7: Print the Total Tests Executed
- Inside the onTestEnd method, check many tests are executed in a line.
- If the count is equal to 50, then invoke the printTotalExecuted method to print the total tests executed so far and the total tests in the test suite.
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
}
Step 8: Increament totalTestsExecuted & noOfTestsPerLine variables
It is possible to print the skipped status in ANSI Yello color. You can also use different color codes based on your preference. You can check the available color codes here .
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
++this.totalTestsExecuted
++this.noOfTestsPerLine
//Printing Skipped Status in ANSI yellow
if (result.status === 'skipped') {
process.stdout.write('\x1b[33m°\x1b[39m');
return;
}
}
Step 10: Printing Retry Status
- If a test has Timed out or Failed and Playwright does know what status needs to be marked, then the test will be marked for Retry.
- Since the test will be rerun, we need to decrease the totalTestsExecuted variable to ensure accuracy.
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
++this.totalTestsExecuted
++this.noOfTestsPerLine
//Printing Skipped Status in ANSI yellow
if (result.status === 'skipped') {
process.stdout.write('\x1b[33m°\x1b[39m');
return;
}
//Printing the test that marked for retry
if (test.outcome() === 'unexpected' && result.retry < test.retries) {
process.stdout.write(`\x1b[33mx\x1b[39m`);
--this.totalTestsExecuted;
return;
}
}
Step 11: Printing Failure Status & Test Title
- Concatenating test title with failure status.
- After printing the status & title, call the printTotalExecuted method to print Total Tests Executed and Total Tests in the Test Suite.
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
++this.totalTestsExecuted
++this.noOfTestsPerLine
//Printing Skipped Status in ANSI yellow
if (result.status === 'skipped') {
process.stdout.write('\x1b[33m°\x1b[39m');
return;
}
//Printing the test that marked for retry
if (test.outcome() === 'unexpected' && result.retry < test.retries) {
process.stdout.write(`\x1b[33mx\x1b[39m`);
--this.totalTestsExecuted;
return;
}
//Printing failure status and test name
if (test.outcome() === 'unexpected' && result.status === 'failed') {
process.stdout.write('\x1b[31m'+"F("+test.title+")"+'\x1b[39m');
this.printTotalExecuted()
return;
}
}
Step 12: Other Statuses (Flaky, TimedOut, & Passed)
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
++this.totalTestsExecuted
++this.noOfTestsPerLine
//Printing Skipped Status in ANSI yellow
if (result.status === 'skipped') {
process.stdout.write('\x1b[33m°\x1b[39m');
return;
}
//Printing the test that marked for retry
if (test.outcome() === 'unexpected' && result.retry < test.retries) {
process.stdout.write(`\x1b[33mx\x1b[39m`);
--this.totalTestsExecuted;
return;
}
//Printing failure status and test name
if (test.outcome() === 'unexpected' && result.status === 'failed') {
process.stdout.write('\x1b[31m'+"F("+test.title+")"+'\x1b[39m');
this.printTotalExecuted()
return;
}
if (test.outcome() === 'unexpected' && result.status === 'timedOut') {
process.stdout.write('\x1b[31mT\x1b[39m');
return;
}
if (test.outcome() === 'expected' && result.status === 'passed') {
process.stdout.write('\x1b[32m.\x1b[39m');
return;
}
if (test.outcome() === 'flaky') {
process.stdout.write('\x1b[33m±\x1b[39m');
return;
}
}
Step 13: Finally, Add onEnd Method
- Print total tests executed just in case it is missed before the end of the execution.
- Print the status of the entire execution.
onEnd(result: FullResult) {
if (this.noOfTestsPerLine !== 0) this.printTotalExecuted();
console.log(`\nFinished the run: ${result.status}`);
}
Full Code:
import { FullConfig } from '@playwright/test';
import { FullResult, Reporter, Suite, TestCase, TestResult } from '@playwright/test/reporter';
class CustomReporter implements Reporter {
totalTests: number;
noOfTestsPerLine: number
totalTestsExecuted: number
constructor() {
this.totalTests=0
this.noOfTestsPerLine=0
this.totalTestsExecuted=0
}
onBegin(config: FullConfig, suite: Suite) {
this.totalTests = suite.allTests().length;
console.log(`Executing ${this.totalTests} test(s)`);
}
printTotalExecuted(){
process.stdout.write(`[${this.totalTestsExecuted}/${this.totalTests}]\n`);
this.noOfTestsPerLine=0
}
onTestEnd(test: TestCase, result: TestResult) {
if (this.noOfTestsPerLine==50){
this.printTotalExecuted()
}
++this.totalTestsExecuted
++this.noOfTestsPerLine
//Printing Skipped Status in ANSI yellow
if (result.status === 'skipped') {
process.stdout.write('\x1b[33m°\x1b[39m');
return;
}
//Printing the test that marked for retry
if (test.outcome() === 'unexpected' && result.retry < test.retries) {
process.stdout.write(`\x1b[33mx\x1b[39m`);
--this.totalTestsExecuted;
return;
}
//Printing failure status and test name
if (test.outcome() === 'unexpected' && result.status === 'failed') {
process.stdout.write('\x1b[31m'+"F("+test.title+")"+'\x1b[39m');
this.printTotalExecuted()
return;
}
if (test.outcome() === 'unexpected' && result.status === 'timedOut') {
process.stdout.write('\x1b[31mT\x1b[39m');
return;
}
if (test.outcome() === 'expected' && result.status === 'passed') {
process.stdout.write('\x1b[32m.\x1b[39m');
return;
}
if (test.outcome() === 'flaky') {
process.stdout.write('\x1b[33m±\x1b[39m');
return;
}
}
onEnd(result: FullResult) {
if (this.noOfTestsPerLine !== 0) this.printTotalExecuted();
console.log(`\nFinished the run: ${result.status}`);
}
}
export default CustomReporter;
Conclusion:
In this blog, we have shown how to overcome the Playwright reporting issues usually seen with the Dot Reporter. In addition, you can use the onEnd method to print a summary of the entire execution, including Total Passed, Total Failed, Total Skipped, and Total Flaky.
The customization of Playwright Dot Reporter is a valuable tool for developers and testers looking to enhance their automated testing processes. Through the use of custom reporters, users have the ability to tailor their test reports to fit their specific needs and preferences.
One of the main benefits of using custom reporters is the flexibility it offers. With Playwright Dot Reporter, users can choose which information they want to include in their reports and how they want it displayed. This allows for more targeted and organized reporting, making it easier to interpret test results and identify any issues that may arise.
Comments(0)