Search

Wednesday, 6 May 2015

Ploblem Solved: Cucumber-JVM running actions before and after tests execution with JUnit

Ploblem Solved: Cucumber-JVM running actions before and after tests execution with JUnit

Background

It is frequent case when we need to do some actions before and/or after entire test suite execution. Mainly, such actions are needed for global initialization/cleanup or some additional reporting or any other kind of pre/post-processing. There may be many different reasons for that and some test engines provide such ability, e.g. TestNG has BeforeSuite and AfterSuite annotations, the JUnit has test fixtures which may run before/after test class (it's not really the same but when we use Cucumber-JVM it's very close to what we need).

Problem

The problem appears when you want to add some actions at the very latest or very earlies stages of tests execution and you use Cucumber-JVM with JUnit. In my case I wanted to add some reports post-processing to make an advanced Cucumber report generation. In this case JUnit fixtures didn't help as AfterClass-annotated method runs before Cucumber generates final reports.

At the same time adding @BeforeAll and @AfterAll hooks question raised on Cucumber side as well. And there was even some solution proposed. Unfortunately, authors decided to revert those changes as there were some cases when it does not work.

So, the problem is that I need something to run after entire Cucumber-JVM suite is done but neither Cucumber nor JUnit gives me built-in capability for doing this.

Solution

JUnit itself gives an ability to customize test runner classes. Actually, Cucumber runs JUnit tests via Cucumber JUnit runner which is already customized. But we can extend Cucumber capabilities by wrapping Cucumber runner inside our custom runner. That doesn't seem to be the smoothest solution as ideally this functionality should be provided by Cucumber itself. But if the solution looks ugly while without it things are getting even worth then this solution is no longer ugly. So, the problem is solved in three major steps:

  1. Create BeforeSuite and AfterSuite annotations
  2. Create wrapper on Cucumber JUnit runner
  3. Use it
So, let's bring some more details on that.

Create BeforeSuite and AfterSuite annotations

The @BeforeSuite and @AfterSuite annotations are simply annotations without any parameter which are assigned to specific method. So, their implementation may look like:

package org.sample.cucumber.annotations;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD })
public @interface AfterSuite {

}
and the same for @BeforeSuite:
package org.sample.cucumber.annotations;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD })
public @interface BeforeSuite {

}
So, now we can annotate some methods. Let's go to the next step.

Create wrapper on Cucumber JUnit runner

Custom runner implementation has several ideas on the back of it. They are:

  • Extended Cucumber wrapper eventually runs Cucumber runner itself
  • In addition to Cucumber runner invoke we look for methods annotated with @BeforeSuite and @AfterSuite annotations. If there are such methods we run them before/after invoking actual Cucumber runner. The sample implementation looks like:
package org.sample.cucumber;

import java.lang.annotation.Annotation;
import java.lang.reflect.Method;

import org.junit.runner.Description;
import org.junit.runner.Runner;
import org.junit.runner.notification.RunNotifier;

import org.sample.cucumber.annotations.AfterSuite;
import org.sample.cucumber.annotations.BeforeSuite;

import cucumber.api.junit.Cucumber;

public class ExtendedCucumberRunner extends Runner {

    private Class clazz;
    private Cucumber cucumber;

    public ExtendedCucumberRunner(Class clazzValue) throws Exception {
        clazz = clazzValue;
        cucumber = new Cucumber(clazzValue);
    }

    @Override
    public Description getDescription() {
        return cucumber.getDescription();
    }

    private void runPredefinedMethods(Class annotation) throws Exception {
        if (!annotation.isAnnotation()) {
            return;
        }
        Method[] methodList = this.clazz.getMethods();
        for (Method method : methodList) {
            Annotation[] annotations = method.getAnnotations();
            for (Annotation item : annotations) {
                if (item.annotationType().equals(annotation)) {
                    method.invoke(null);
                    break;
                }
            }
        }
    }

    @Override
    public void run(RunNotifier notifier) {
        try {
            runPredefinedMethods(BeforeSuite.class);
        } catch (Exception e) {
            e.printStackTrace();
        }
        cucumber.run(notifier);
        try {
            runPredefinedMethods(AfterSuite.class);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}
Additional attention should be paid to the highlighted code. It doesn't use any object instance which means that @BeforeSuite and @AfterSuite annotations should be applied to static methods in order to make things working. OK, we're almost there. Here it goes the last step.

Use it

Now we can define our Cucumber test suite with custom runner. In JUnit/Cucumber-JVM each test suite corresponds to single JUnit class. So, we can declare something like:

package org.sample.cucumber;

import org.junit.runner.RunWith;

import org.sample.cucumber.annotations.AfterSuite;
import org.sample.cucumber.annotations.BeforeSuite;
import org.sample.cucumber.ExtendedCucumberRunner;

import cucumber.api.CucumberOptions;

@CucumberOptions(
        plugin = {"html:target/cucumber-html-report",
                  "json:target/cucumber.json",
                  "pretty:target/cucumber-pretty.txt",
                  "usage:target/cucumber-usage.json"
                 },
        features = {"output/" },
        glue = {"org/sample/cucumber/glue" },
        tags = { }
)
@RunWith(ExtendedCucumberRunner.class)
public class SampleTestClass {
    @BeforeSuite
    public static void setUp() {
        // TODO: Add your pre-processing
    }
    @AfterSuite
    public static void tearDown() {
        // TODO: Add your post-processing
    }
}
Highlighted parts are the places where we use our custom objects.

Generally, that's it! Now you have your own way to define pre/post conditions which are performed at the very beginning/at the end of entire test suite which works for JUnit in combination with Cucumber-JVM.

36 comments:

  1. this.clazz is always setting to null. It is never being initialized

    Method[] methodList = this.clazz.getMethods();

    ReplyDelete
    Replies
    1. Good catch. Thank you. Apparently the clazz field should be initialized in the constructor and it should take value from the constructor parameter. I've updated the code in the post. Again, thank you for spotting this problem.

      Delete
    2. Perfect, works like a charm!!!

      Delete
  2. Thanks, just what I was looking for!

    ReplyDelete
  3. I've created BeforeSuite, AfterSuite annotations and wraper. Here is how I use wraper http://take.ms/0oTuA
    HTML page with results wasn't generated after the last scenario because of error http://take.ms/VB5O7
    Please help to solve this problem

    ReplyDelete
  4. Hey the idea is elegant, but I am getting this error when @AfterSuite block executes, please help me out..

    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.sample.cucumber.ExtendedCucumberRunner.runPredefinedMethods(ExtendedCucumberRunner.java:39)
    at org.sample.cucumber.ExtendedCucumberRunner.run(ExtendedCucumberRunner.java:49)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
    Caused by: java.lang.NullPointerException
    at limeroad.LimeroadOrderCancellation.setUp(LimeroadOrderCancellation.java:22)
    ... 12 more

    ReplyDelete
    Replies
    1. Heyy...Nickolay

      Sorry my bad....I had not declared log4j so was getting this error...any way great post....

      Delete
    2. Good that the problem was resolved. All those tiny errors are the biggest source of pain.
      Also, if you use @BeforeSuite and @AfterSuite annotations keep in mind that they are assigned to static methods only. It is done this way because at the moment when they are run the class instance doesn't exist either yet or anymore.

      Delete
  5. In my @BeforeSuite method, I'm checking an environment property. if it is running in a certain environment, I want my method to stop all tests from running. How best do I do this? I tried just throwing an exception, but the tests still run. I've tried System.exit(), and that works like a charm, but is that really the best way? thanks.

    ReplyDelete
  6. In my @BeforeSuite method, I'm checking an environment property. if it is running in a certain environment, I want my method to stop all tests from running. How best do I do this? I tried just throwing an exception, but the tests still run. I've tried System.exit(), and that works like a charm, but is that really the best way? thanks.

    ReplyDelete
    Replies
    1. Hi Kate,

      The ExtendedCucumber runner invokes BeforeSuite method this way:

      try {
      runPredefinedMethods(BeforeSuite.class);
      } catch (Exception e) {
      e.printStackTrace();
      }

      It means that if you throw any exception it will be intercepted. But you can try to throw anything else. E.g. you can throw an Error (or AssertionError) or at least an instance of the Throwable class. This way it should interrupt the entire suite.

      System.exit() seems to be very radical way to stop things.

      Delete
  7. This is really helpful implementation, Thanks for sharing the post.

    I was trying to update the @tags in BeforeSuite based on command line parameter which I also use for soem preprocessing.

    Can you please let me know how we can update the parameters of @CucumberOptions provided in annotation?

    ReplyDelete
  8. And/or read it in BeforeSuite.

    ReplyDelete
    Replies
    1. Firstly, I don't think it's good idea to modify parameters like tags which are supposed to be read-only. As for annotation modification, it's normally prohibited by Java as annotation is initialised at the early stage. Well, I've seen some hacks in code which could make such modifications but it's definitely some kind of "birth hack" and I don't recommend you doing this.

      Secondly, the solution described in this post would hardly do what you ask for. The problem is that the @BeforeSuite method is something to run before even Cucumber starts processing. It means at the point when BeforeSuite method is executed there's no structures which store tags and all relevant stuff.

      However, let's see this situation differently. You are asking for something which is hardly supported by existing toolset. Maybe if you describe why do you need that there would be some better solution for doing what you want.

      Delete
  9. Thanks Nickolay, for quick response.

    Yes I understand that cucumber is not into picture from this approach at this point.

    What I want is to override tags before cucumber kicks-in.

    Or if we can create cucumberoptions in BeforeSuite not as tag but as an object and pass on to cucumber.

    Actually, I have a scenario where I supposed to run my servers in BeforeSuite and the run each scenario. In order to run the server I am relying on command line variable, and I am using same CLvar for tags to pick respective scenario for test run.

    Hope this clarifies my approach and requirement .

    Thanks again,

    ReplyDelete
  10. Thanks Nickolay, for quick response.

    Yes I understand that cucumber is not into picture from this approach at this point.

    What I want is to override tags before cucumber kicks-in.

    Or if we can create cucumberoptions in BeforeSuite not as tag but as an object and pass on to cucumber.

    Actually, I have a scenario where I supposed to run my servers in BeforeSuite and the run each scenario. In order to run the server I am relying on command line variable, and I am using same CLvar for tags to pick respective scenario for test run.

    Hope this clarifies my approach and requirement .

    Thanks again,

    ReplyDelete
    Replies
    1. I'm afraid tags modification is not the best case here. Tags are designed to split tests to pre-defined groups. Nothing more. If you want to use tags defined in command line you don't need to run the JUnit test runner.

      But if you still need to run some specific test suite depending on server and use ExtendedCucumber at the same time, you can create multiple test classes with Cucumber annotations where each of them is targeted to specific tags.

      If you need to pass values to start servers in the BeforeSuite method you can pass required value either via environment variable or system property and retrieve it in BeforeSuite method. Or even more, you can move server start logic at some higher level, e.g. you can run the batch where we start server first and then run test suite.

      Delete
  11. Thanks Nickolay, These suggestion will help me. Let me try this approach.

    ReplyDelete
  12. When I tried the above approach I am getting the following error. Could you please help me to figure it out

    cucumber.runtime.CucumberException:

    Classes annotated with @RunWith(Cucumber.class) must not define any
    Step Definition or Hook methods. Their sole purpose is to serve as
    an entry point for JUnit. Step Definitions and Hooks should be defined
    in their own classes. This allows them to be reused across features.
    Offending class: class cucumberTest.SampleTestClass

    at cucumber.runtime.junit.Assertions.assertNoCucumberAnnotatedMethods(Assertions.java:13)
    at cucumber.api.junit.Cucumber.(Cucumber.java:52)
    at cucumberTest.ExtendedCucumberRunner.(ExtendedCucumberRunner.java:20)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
    at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
    at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
    at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
    at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
    at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.(JUnit4TestReference.java:33)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestClassReference.(JUnit4TestClassReference.java:25)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:48)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:444)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

    ReplyDelete
    Replies
    1. If you see such error your class probably contains some methods annotated with one of the following:
      cucumber.api.java.After
      cucumber.api.java.Before

      or it contains Given/When/Then statement. None of those entities should be present in the test class

      Delete
  13. I don't have any of the above methods or Given/When/Then statements in the SampleTestClass. Still I am getting this error

    ReplyDelete
    Replies
    1. Then check parent class. This error exactly states that you have some methods annotated with one of the standard Cucumber annotation. You can take a look at the actual code which caused this issue at Cucumber-JVM GitHub repository. So, if you have any method annotation which full class name starts with cucumber you'll get this error. And this is standard Cucumber-JVM behaviour

      Delete
  14. I have done same.
    Step 1: Created SampleTestClass
    package org.sample.cucumber;

    import org.junit.runner.RunWith;
    import org.sample.cucumber.annotations.AfterSuite;
    import org.sample.cucumber.annotations.BeforeSuite;
    import org.sample.cucumber.ExtendedCucumberRunner;

    import cucumber.api.CucumberOptions;
    @RunWith(ExtendedCucumberRunner.class)


    @CucumberOptions(
    plugin = {"html:target/cucumber-html-report",
    "json:target/cucumber.json",
    "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json"
    },
    features = {"Features/" },
    glue = {"org/sample/cucumber" },
    tags = { }
    )

    public class SampleTestClass {
    @BeforeSuite
    public static void setUp() {
    // TODO: Add your pre-processing
    }
    @AfterSuite
    public static void tearDown() throws Exception {
    // TODO: Add your post-processing

    }
    }


    Step 2: Created AfterSuite interface
    package org.sample.cucumber.annotations;

    import java.lang.annotation.ElementType;
    import java.lang.annotation.Retention;
    import java.lang.annotation.RetentionPolicy;
    import java.lang.annotation.Target;

    @Retention(RetentionPolicy.RUNTIME)
    @Target({ElementType.METHOD })
    public @interface AfterSuite {

    }



    Step 3: Created BeforeSuite interface
    package org.sample.cucumber.annotations;

    import java.lang.annotation.ElementType;
    import java.lang.annotation.Retention;
    import java.lang.annotation.RetentionPolicy;
    import java.lang.annotation.Target;

    @Retention(RetentionPolicy.RUNTIME)
    @Target({ElementType.METHOD })
    public @interface BeforeSuite {

    }



    Step 4: created ExtendedCucumberRunner class
    package org.sample.cucumber;

    import java.lang.annotation.Annotation;
    import java.lang.reflect.Method;

    import org.junit.runner.Description;
    import org.junit.runner.Runner;
    import org.junit.runner.notification.RunNotifier;

    import org.sample.cucumber.annotations.AfterSuite;
    import org.sample.cucumber.annotations.BeforeSuite;

    import cucumber.api.junit.Cucumber;

    public class ExtendedCucumberRunner extends Runner {

    private Class clazz;
    private Cucumber cucumber;

    public ExtendedCucumberRunner(Class clazzValue) throws Exception {
    clazz = clazzValue;
    cucumber = new Cucumber(clazzValue);
    }

    @Override
    public Description getDescription() {
    return cucumber.getDescription();
    }

    private void runPredefinedMethods(Class annotation) throws Exception {
    if (!annotation.isAnnotation()) {
    return;
    }
    Method[] methodList = this.clazz.getMethods();
    for (Method method : methodList) {
    Annotation[] annotations = method.getAnnotations();
    for (Annotation item : annotations) {
    if (item.annotationType().equals(annotation)) {
    method.invoke(null);
    break;
    }
    }
    }
    }

    @Override
    public void run(RunNotifier notifier) {
    try {
    runPredefinedMethods(BeforeSuite.class);
    } catch (Exception e) {
    e.printStackTrace();
    }
    cucumber.run(notifier);
    try {
    runPredefinedMethods(AfterSuite.class);
    } catch (Exception e) {
    e.printStackTrace();
    }
    }

    }

    Step 5 : Run SampleTestClass as Junit

    Result: I see simple html report(With given,when,then conditions)

    Please help me on how to get advanced reports.

    ReplyDelete
  15. Replies
    1. Custom reports are not generated because you have no instructions doing this.

      I suggest to use cucumber-reports library for this. It is the library which contains all the reporting and runner functionality I described in this and some other posts. You can find more information about the library (including Maven dependency definition) at the official library documentation page

      In particular, you can find some Unit sample here: http://mkolisnyk.github.io/cucumber-reports/extended-cucumber-runner#junit.

      Delete
  16. Thanks so much for your help :)
    I'm trying to implement the same.

    ReplyDelete
  17. Hi Nickolay,
    When trying to send an email with the index.html report file generated, we call the send mail method in the @Aftersuite but the mail is sent with an attached file which is empty. I did debug the script and see that the index.html is populated with some data before the script reaches the @Aftersuite tag. Though there is data the file appended to the email is blank. Would request you to inform why is a blank html file being sent, is it to do with the static reference to the file.

    ReplyDelete
    Replies
    1. The report is empty because at the moment the @AfterSuite is executed the initial JSON report isn't generated yet. Here is the related post describing the same problem.

      Delete
  18. This is not working at all for me.
    Always getting Null Pointer Exception and Before and AFterSuites annotations are not calling before suite and after suite

    ReplyDelete
    Replies
    1. From this article:


      Additional attention should be paid to the highlighted code. It doesn't use any object instance which means that @BeforeSuite and @AfterSuite annotations should be applied to static methods in order to make things working.


      So, try to declare BeforeSuite and AfterSuite methods as static.

      Delete
  19. Hi,

    Thanks for the great post. I am trying to read data from the excel sheet and load in a map in before suite method. And if I want to call the feature file repetitively for each row of data how i do it ?

    Thanks in advance,
    Soma

    ReplyDelete
    Replies
    1. If you simply want data-drive your tests the Cucumber has scenario outlines for that. All your test data can be defined in the feature file and processed in standard way, so that your scenario will run for each data row.

      Delete
  20. Exactly. The requirement is something like we shouldn't use the example section of the feature file. Have tried creating a wrapper above the cucumber options. With before suite method i am able to read the data from excel . But couldn't repetitively call the feature file for multiple data set. Is there any way to proceed from this point.

    ReplyDelete
  21. Exactly. The requirement is something like we shouldn't use the example section of the feature file. Have tried creating a wrapper above the cucumber options. With before suite method i am able to read the data from excel . But couldn't repetitively call the feature file for multiple data set. Is there any way to proceed from this point.

    ReplyDelete
    Replies
    1. If you are required not to use standard data-driven ability for Cucumber maybe you don't need to use Cucumber at all. You can use Parameterised JUnit runner.

      Alternatively, if you need to keep Excel spreadsheet as a storage of secure data you can also use Examples but instead of exposing data you just can define the row number to get data from Excel spreadsheet. Thus, you use standard data-driven functionality and actual data is not exposed.

      Anyway, hacking Cucumber runner itself is definitely not the way to go.

      Delete