In our Allure Report Tutorial, we will be explaining the steps to generate an Allure report and show you the different features that are available as reporting is an integral part of every automation testing framework. Without proper reports, you will not be able to analyze the results of your execution and take further action. Though automation tests are created and executed by people with technical knowledge, the people who read the reports may not possess too much technical knowledge. So it is important for reports to be visually strong and easy to understand.
We have chosen Allure as it is an open-source test reporting tool that can be used to represent the results of your test execution concisely in the form of interactive and comprehensive web reports. We also have first-hand experience in using Allure reports to create top-notch reports while delivering high-quality automation testing services to our clients. So using that experience, we have created and used a sample report in our Allure Report tutorial to make it easier for you to understand.
Allure Report Tutorial
You’d have to follow the below-mentioned installation steps if you haven’t yet installed Allure on your computer. If you already have Allure installed and are aware of the dependencies, you can directly proceed further in our Allure Report Tutorial. We have listed the important annotations you’ll have to know, defined the project structure, and provided the code snippets you can use to create your Allure report.
Installation & Set up
1. Download the latest version as a zip archive from Github Releases.
2. Unpack the archive to the allure-commandline directory.
3. Navigate to the bin directory.
4. Use allure.bat for Windows or allure for Unix platforms.
5. Add allure to the system PATH.
Note: Allure CLI requires Java Runtime Environment to be installed.
Add the Dependencies & Run
- Update the Properties section in the Maven pom.xml file
- Add Selenium, JUnit4 and Allure-JUnit4 dependencies in POM.xml
- Update Build Section of pom.xml in Allure Report Project.
- Create Pages and Test Code for the pages
Once you have completed all the above steps, you can run the test and generate your Allure Report by following the steps mentioned later on in our Allure report tutorial. Now let’s take a look at the code for your POM.xml file.
Pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>Selenium-Allure-Demo</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<selenium.version>3.141.59</selenium.version>
<testng.version>7.4.0</testng.version>
<allure.testng.version>2.14.0</allure.testng.version>
<maven.compiler.plugin.version>3.5.1</maven.compiler.plugin.version>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<aspectj.version>1.9.6</aspectj.version>
<maven.surefire.plugin.version>3.0.0-M5</maven.surefire.plugin.version>
</properties>
<build>
<plugins>
<!-- Compiler plug-in -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven.compiler.plugin.version}</version>
<configuration>
<source>${maven.compiler.source}</source> <!--For JAVA 8 use 1.8-->
<target>${maven.compiler.target}</target> <!--For JAVA 8 use 1.8-->
</configuration>
</plugin>
<!-- Added Surefire Plugin configuration to execute tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.plugin.version}</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>TestNG.xml</suiteXmlFile>
</suiteXmlFiles>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
</argLine>
</configuration>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.aspectj/aspectjweaver -->
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>${selenium.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.testng/testng -->
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>${testng.version}</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/io.qameta.allure/allure-testng -->
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-testng</artifactId>
<version>${allure.testng.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Project structure with Allure Report:
In our Allure report tutorial, we have basically kept two packages called ‘pages’ and ‘tests’. The pages package will include all the pages you want to test and the tests package will include the different tests that you want to perform.
Pages Package:
There are two pages that we have added to our package, and they are the Dashboard page and the Login page.
Dashboard page:
package AllureDemo.pages;
import io.qameta.allure.Step;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.testng.Assert;
public class DashboardPage {
WebDriver driver;
By dashboardPageTitle = By.xpath("(//span[.='Dashboard'])[1]");
By options = By.xpath(
"//div[@title='Assign Leave']");
public DashboardPage(WebDriver driver) {
this.driver = driver;
}
@Step("Verify title of Dashboard page")
public void verifyDashboardPageTitle() {
String DashboardPageTitle = driver.findElement(dashboardPageTitle).getText();
Assert.assertTrue(DashboardPageTitle.contains("Dashboard"));
}
@Step("Verify Quick Launch Options on Dashboard page")
public void verifyQuickLaunchOptions() {
String QuickLaunchOptions = driver.findElement(options).getAttribute("title");
Assert.assertTrue(QuickLaunchOptions.contains("Assign Leave"));
}
}
Login page:
package AllureDemo.pages;
import io.qameta.allure.Step;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.testng.Assert;
public class LoginPage {
WebDriver driver;
By userName = By.name("username");
By password = By.name("password");
By titleText = By.name("username");
By login = By.xpath("//button[@type='submit']");
By errorMessage = By.xpath("//p[.='Invalid credentials']");
public LoginPage(WebDriver driver) {
this.driver = driver;
}
// Set user name in the textbox
public void setUserName(String strUserName) {
driver.findElement(userName).sendKeys(strUserName);
}
// Set password in the password textbox
public void setPassword(String strPassword) {
driver.findElement(password).sendKeys(strPassword);
}
// Click the login button
public void clickLogin() {
driver.findElement(login).click();
}
@Step("Verify title of Login Page")
public void verifyPageTitle() {
String loginPageTitle = driver.findElement(titleText).getAttribute("name");
Assert.assertTrue(loginPageTitle.contains("username"));
}
/* Failed Test */
@Step("Verify error message when invalid credentail is provided")
public void verifyErrorMessage() {
String invalidCredentialErrorMessage = driver.findElement(errorMessage).getText();
Assert.assertTrue(invalidCredentialErrorMessage.contains("Invalid credentials"));
}
@Step("Enter username and password")
public void login(String strUserName, String strPasword) {
// Fill in the user name
this.setUserName(strUserName);
// Fill in the password
this.setPassword(strPasword);
// Click the Login button
this.clickLogin();
}
}
Tests Package:
For our Allure Report tutorial, we have added 3 tests by the names base test, dashboard test, and login test. Let’s take a look at the code snippets for each of these tests below.
Base test:
package AllureDemo.tests;
import AllureDemo.pages.DashboardPage;
import AllureDemo.pages.LoginPage;
import io.qameta.allure.Step;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import java.util.concurrent.TimeUnit;
public class BaseTest {
public static WebDriver driver;
LoginPage objLogin;
DashboardPage objDashboardPage;
@Step("Start the application")
@BeforeMethod
public void setup() {
System.setProperty("webdriver.chrome.driver",
"C:\\Users\\Codoid\\Documents\\QA-SAM-March2023-Stories\\resources\\drivers\\chromedriver_win.exe");
driver = new ChromeDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://opensource-demo.orangehrmlive.com/");
}
@Step("Stop the application")
@AfterMethod
public void close() {
driver.close();
}
}
DashboardTests
package AllureDemo.tests;
import AllureDemo.pages.DashboardPage;
import AllureDemo.pages.LoginPage;
import io.qameta.allure.*;
import org.testng.ITestNGListener;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Epic("Web Application Regression Testing")
@Feature("Dashboard Page Tests")
@Listeners(TestExecutionListener.class)
public class DashboardTests extends BaseTest {
LoginPage objLogin;
DashboardPage objDashboardPage;
@Severity(SeverityLevel.BLOCKER)
@Test(priority = 0, description = "Verify Dashboard Page")
@Description("Test Description : After successful login to application opens Dashboard page")
@Story("Successful login of application opens Dashboard Page")
public void DashboardTest() {
objLogin = new LoginPage(driver);
// Login to the application
objLogin.login("Admin", "admin123");
// Go to the dashboard page
objDashboardPage = new DashboardPage(driver);
// Verify the dashboard page
objDashboardPage.verifyQuickLaunchOptions();
}
private class TestExecutionListener implements ITestNGListener {
}
}
LoginTests
package AllureDemo.tests;
import AllureDemo.pages.DashboardPage;
import AllureDemo.pages.LoginPage;
import io.qameta.allure.*;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Epic("Regression Tests")
@Feature("Login Tests")
@Epic("Web Application Regression Testing")
@Feature("Login Page Tests")
@Listeners(TestExecutionListener.class)
public class LoginTests extends BaseTest {
LoginPage objLogin;
DashboardPage objDashboardPage;
@Severity(SeverityLevel.NORMAL)
@Test(priority = 0, description = "Verify Login Page")
@Description("Test Description : Verify the title of Login Page")
@Story("Title of Login Page")
public void verifyLoginPage() {
// Create Login Page object
objLogin = new LoginPage(driver);
// Verify login page text
objLogin.verifyPageTitle();
}
/* Failed Test */
@Severity(SeverityLevel.BLOCKER)
@Test(priority = 1, description = "Login with invalid username and password")
@Description("Test Description : Login Test with invalid credentials")
@Story("Unsuccessful Login to Application")
public void invalidCredentialTest() {
// Create Login Page object
objLogin = new LoginPage(driver);
objLogin.login("test", "test123");
// Verify login page text
objLogin.verifyErrorMessage();
}
}
TestExecutionListener
package AllureDemo.tests;
import io.qameta.allure.Attachment;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class TestExecutionListener implements ITestListener {
@Attachment(value = "Screenshot of {0}", type = "image/png")
public byte[] saveScreenshot(String name, WebDriver driver) {
return (byte[]) ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
}
@Override
public void onTestFailure(ITestResult result) {
saveScreenshot(result.getName(), BaseTest.driver);
}
}
TestNG XML
Test Execution
In order to execute your tests, you have to open command prompt, go to the project directory, and write the below command.
allure serve allure-results
Allure in Cucumber BDD
You can also add Allure to your Cucumber BDD framework by adding the below dependency in the POM.xml file.
<!-- https://mvnrepository.com/artifact/io.qameta.allure/allure-cucumber-jvm -->
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-cucumber6-jvm</artifactId>
<version>2.14.0</version>
</dependency>
</dependencies>
Add the following plugin to your Runner cukes file by referring to the below screenshot
"io.qameta.allure.cucumber6jvm.AllureCucumber6Jvm"
Sample Allure Report
Before we proceed to see the sample report in our Allure report tutorial, let’s first explore the 5 most important or commonly used annotations used in Allure. You can expand the capabilities of an Allure report by using more annotations, but these 5 will help you generate your basic first report.
- Allure Report Overview
- Categories in Allure Report
- Suites in Allure Report
- Graphs in Allure Report
- Behaviors in Allure Report
- Timeline in Allure Report
- Packages in Allure Report
Important Annotations used in the Allure report
- @Step – This annotation is used as a step definition to define any modifier with the parameter.
- @Epic – This annotation is used to define the large component or a whole product under which the Allure depends on.
- @Feature – We can use it to represent all the possible test coverage options based on the Test scenarios.
- @Stories – Every Epic has many stories and each of those stories represent a sub-product or sub-component.
- @Attachment – At the end of our test execution, we will have to add screenshots to our reports. And this annotation can be used to define the position of the screenshot.
Allure Report Overview
As you can see in the above screenshot, the Allure report overview has different sections such as suites, environment, categories, and so on. Let’s find out what they are in our Allure report tutorial.
Suites
A suite is a group of tests that share a common context such as a specific test environment or a particular test category. Suites are represented by colored blocks in the report’s timeline, and you can drill down into the details of each suite by clicking on them.
Environment:
The environment section contains information about the operating system, browser, framework, and version of the tools used.
Features By Stories:
In order to group test cases by features or user stories in an Allure report, you can use the @Feature or @Story annotations in your test code.
Categories:
Once this annotation is added to the test, the test cases will be grouped under the “smoke” and “regression” categories when your Allure report is generated. It is also possible to use the allure.category property to add the category information to the test case.
Executors:
Allure also supports custom executors that can be used when your current executors do not match the requirements of your project. Kindly note that they have to be implemented as a separate component.
Graphs:
Graphical representation of the status, severity, duration, and duration trend will also be available in an Allure report’s overview.
Timeline:
This section in an Allure report’s overview provides a graphical representation of the test execution. It will show the duration of each test case, and also specifies the order in which they were executed.
Categories in Allure Report
Next up in our Allure report tutorial, we’re going to explore the Categories tab that gives you the way to create custom defects classification for your test results.
By default, there will be two categories of defects in an Allure report:
- Product defects (Tests that failed due to an issue in the product)
- Test defects (Broken tests)
But you also have the option to create custom defect classifications such as skipped tests, outdated tests, and so on. You can do so by adding a categories.json file to the allure-results directory before report generation.
categories.json
[
{
"name": "Ignored tests",
"matchedStatuses": ["skipped"]
},
{
"name": "Infrastructure problems",
"matchedStatuses": ["broken", "failed"],
"messageRegex": ".*bye-bye.*"
},
{
"name": "Outdated tests",
"matchedStatuses": ["broken"],
"traceRegex": ".*FileNotFoundException.*"
},
{
"name": "Product defects",
"matchedStatuses": ["failed"]
},
{
"name": "Test defects",
"matchedStatuses": ["broken"]
}
]
(mandatory) category name
(optional) list of suitable test statuses. Default [“failed”, “broken”, “passed”, “skipped”, “unknown”]
(optional) regex pattern to check test error message. Default “.*”
(optional) regex pattern to check stack trace. Default “.*”
Test result falls into the category if their status is in the list and both the error message and stack trace match the pattern.
Suites in Allure Report
As you can see in the above screenshot, the Suites tab shows a standard structural representation of the executed tests, grouped by the suites and classes that can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data such as status breakdown, severity, and duration diagrams. They will be very effective in helping us understand the status of our test execution easily.
Behaviors in Allure Report
Behaviors group the test results according to Epic, Feature, and Story tags. It also consists of the status of each and every execution and has a detailed description as well.
Timeline in Allure Report
Allure adaptors will collect the accurate timings of the tests and arrange them in a sequential or parallel structure based on the timing. This visualization of the test execution will be available in the Timeline section of our Allure report.
Packages in Allure Report
The packages tab represents a tree-like layout of the test results, grouped by different packages as shown in the above image.
Advantages of Allure Report
In general, effective reporting can help shorten the development/QA lifecycle by providing both the developers and testers with all the information they will need. As seen above in our Allure report tutorial, it enables us to categorize the test failures into bugs, broken tests, and logs. Other vital information such as steps, fixtures, attachments, timings, history, and integrations with TMSs are also available to configure bug-tracking systems.
Managers and stakeholders will have a clear understanding of the project’s big picture, progress, timeline, defects clustering info, and so on. Now let’s take a look at the technical advantages that Allure reports have to offer.
- Allure is an open-source report engine built using AngularJS.
- It supports various frameworks such as JUnit, TestNG, Cucumber-Java, Selenide, Pytest, behav, Jasmine, Mocha, RSpec, Spock, PHPUnit, SpecFlow, NUnit, MSTest, etc.
- You can create visually rich reports that can be easily understood even by non-technical team members such as Stakeholders and Business Analysts.
- It has a command line interface to generate reports based on maven surefire plugins.
- It supports popular CI/CD platforms like Jenkins, TeamCity, Banboo, Gradle, Maven, and CodeFresh.
- Modularity and extensibility of Allure guarantee that you will always be able to fine-tune it to make Allure suit your needs better.
Note: Allure allows you to generate a single HTML file for easy sharing and viewing of test reports.
Why Use a Single HTML File?
- Simplified Sharing: No additional setup is needed to view the report.
- Compact and Portable: All required data is packaged into a single, standalone file.
- Browser Compatibility: Open and view in any modern browser without additional dependencies.
Steps to generate single Html file :
Open cmd for framework location enter “allure generate –single–file D:\QA\QA-test-automation-codoid-Feb21\allure-results”
It will generate single html file in “allure-report” folder as below.
Conclusion
As a lightweight, flexible, multilingual test report tool, Allure Framework provides a neat web report that not only illustrates what has been tested in a concise manner. It also gives everyone involved in developing the maximum amount of useful information from testing every day. We hope that our Allure Report tutorial has given you a clear understanding of how to generate Allure reports for your project. As a leading automation testing company, we have also written a blog highlighting the key differences between Allure report and Extent report. So make sure to subscribe to our newsletter to not miss out on any of our latest blogs and updates.
Comments(10)
Posted on Nov 10, 2023
1 year ago
Pretty! This has been a really wonderful post. Many thanks for providing these details.
Posted on Oct 16, 2023
1 year ago
The depth of research in your articles is unparalleled.
Posted on Oct 14, 2023
1 year ago
I just like the helpful information you provide in your articles
Posted on Oct 14, 2023
1 year ago
This is really interesting, You’re a very skilled blogger. I’ve joined your feed and look forward to seeking more of your magnificent post. Also, I’ve shared your site in my social networks!
Posted on Oct 14, 2023
1 year ago
This is my first time pay a quick visit at here and i am really happy to read everthing at one place
Posted on Sep 17, 2023
1 year ago
Great information shared.. really enjoyed reading this post thank you author for sharing this post .. appreciated
Posted on Aug 24, 2023
1 year ago
Awesome! Its genuinely remarkable post, I have got much clear idea regarding from this post
Posted on Aug 19, 2023
1 year ago
This was beautiful Admin. Thank you for your reflections.
Posted on Jun 15, 2023
1 year ago
I am truly thankful to the owner of this web site who has shared this fantastic piece of writing at at this place.
Posted on Jun 12, 2023
1 year ago
I appreciate you sharing this blog post. Thanks Again. Cool.