Previously I've described basic report samples based on Cucumber-JVM. Since I'm using this reporting solution on practice quite frequently the number of requirements and enhancements grows so since last time I had to add some more additional features which can be in use.
Some of new features are:
- Cucumber extension to support failed tests re-run
- Detailed results report which supports the following:
- Detailed results report generation in HTML format
- Detailed results report generation in PDF format
- Screen shot are included into both the above reports
<dependency> <groupId>com.github.mkolisnyk</groupId> <artifactId>cucumber-reports</artifactId> <version>0.0.5</version> </dependency>or the same thing for Gradle:
Since Cucumber failed tests re-run functionality was described before in this post I'll concentrate more on detailed results report features.
Why do we need the detailed results report?
First question which may appear here is: why do we need such report? Most of the details can be retrieved from standard Cucumber HTML report. Well, main reason is that existing HTML results report isn't enough and on practice we need some additional features. Some of them are:
- Report itself should have minimal external dependencies (e.g. styles, scripts) so that it is easier to publish it as build artifact
- It is very useful to have report showing screen shots on every error encountered
- The entire report should also have some portable and self-contained format so that it can be sent via e-mail. Thus, PDF format can be quite useful for that
Detailed Report Structure
The image below shows how detailed report looks like:
- Table of Contents
- Detailed Results Report
Overview section contains aggregated data on entire run results. In particular it shows the number of passed/failed/skipped features/scenarios and steps with the % of passed items in comparison to entire number of items. Here is how typical overview section looks like:
Table of Contents section
Table of contents is the hyper-linked list of features and scenarios highlighting the execution status. It is convenient addition to the report which provides possibility to navigate to each specific scenario result. Thus we have fast way to get to the test we're interested in.
Detailed Results Report section
Detailed results report looks similar to standard HTML report and simply contains the description of steps with the status and detailed stack trace is case of error. Also, every scenario result contains hyperlink to the table of contents. Thus we can easily navigate to scenario we need and then switch to any other scenarios we're interested in.
How to Generate Detailed Report
The report generation is based on standard Cucumber JSON results report post-processing. So, before generating detailed results report we should specify which results report file we should use for report generation. So, typical Java code for report producing looks like:
CucumberDetailedResults results = new CucumberDetailedResults(); results.setOutputDirectory("target/"); results.setOutputName("cucumber-results"); results.setSourceFile("./src/test/resources/cucumber.json"); results.executeDetailedResultsReport(false);This code produces detailed report stored at the following path: target/cucumber-results-test-results.html.
Adding Screen Shots
Often it's very useful to have the screen shot on error to see what was the real problem which caused error. It's not the silver bullet but it's definitely helpful for analysis. Since currently described report is produced as the result of post-processing but not as the part of hooks the way screen shots are generated are out of report generator scope. We simply should assume that we already have some folder containing screen shot files which fit the following rules:
- Image files are of PNG format
- Each failed test contains only one screen shot as we normally have only one error on failed test. Maybe in the future I'll add some additional options to handle multiple files but currently it is the way we do things now.
- Screen shot file names are taken from the names of corresponding scenario IDs where special characters like ; or spaces are replaced with underscore.
CucumberDetailedResults results = new CucumberDetailedResults(); results.setOutputDirectory("target/"); results.setOutputName("cucumber-results"); results.setSourceFile("./src/test/resources/cucumber.json"); results.setScreenShotLocation("../src/test/resources/"); results.executeDetailedResultsReport(false);The highlighted part defines where to look for screen shots.
And the last feature is the ability to generate file in PDF format. Actually, when we generate HTML report with all the above features it is still hard to transport as it may contain references to image files. In order to make the report more portable (e.g. if we need to send it then via e-mail). In order to reach this we simply need to convert generated detailed report HTML to PDF file. This flow is handled by the parameter of the executeDetailedResultsReport method. If it is set to true the PDF report is generated in addition to HTML one. If it is false only HTML report is generated. So, in order to produce PDF output we should modify previous code sample this way:
CucumberDetailedResults results = new CucumberDetailedResults(); results.setOutputDirectory("target/"); results.setOutputName("cucumber-results"); results.setSourceFile("./src/test/resources/cucumber.json"); results.setScreenShotLocation("../src/test/resources/"); results.executeDetailedResultsReport(true);The above example will produce PDF file located by the path: target/cucumber-results-test-results.pdf.
That were additional Cucumber reporting features introduced recently. So, currently we have reports which can be set as e-mail message body + some additional reports which can be provided as attachments. This is good foundation for quite convenient notification system where we don't need to keep all our reports as build artifacts as well as now we have one major report where we can get both overview and detailed information. The list of enhancements will grow as long as more additional requirements appears. But the stuff we have right now is already good.