Search

Loading...

Wednesday, 20 March 2013

JBehave vs Cucumber JVM comparison

The interest in BDD approach is growing and the more and more different engines appear in that space. However, there're some well known players in that area. We can compare their features however the final choice is still based on the technology used. E.g. if we compare Behat and SpecFlow we definitely can find some gaps and advantages but if we're going to use e.g. C# the Behat is not a subject of interest here. And vice versa.

The more interesting case is when 2 or more engine appear in the same technological area. And it's twice more interesting when these are one of the most popular engines on one of the most popular technology. In this post we'll compare JBehave vs Cucumber against their applicability for Java test automation. We compare them using the same scale as for BDD Engines Comparison post with some additional detalization on features. Additionally, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

All right, we have JBehave, we have Cucumber, so ...

LET THE MORTAL COMBAT COMPARISON BEGIN!!!

What would be compared

Firstly, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

During comparison we'll cover the following areas/features:

  • Documentation
  • Flexibility in passing parameters
  • Auto-complete
  • Scoping
  • Composite steps
  • Backgrounds and Hooks
  • Binding to code
  • Formatting flexibility
  • Built-in reports
  • Conformity to Gherkin standard
  • Input Data Sources
And again we'll use 4 grade scale for evaluating each of the features. The grades are classified in the following way:
GradeCriteria
0Feature is not represented
1Feature is partially available with some restrictions
2Mainly feature exists but without any extra options
3Full-featured support

Documentation

The documentation is the first entry point while trying to use any library/tool/other software whatever. So, we'll start with it here as well.While evaluating this feature we'll take a look at the following items:

  • Official site - every more or less formed solution should have an official site and an entry point for new people. Otherwise it would be hard to find it within miriads pages in the Internet space
  • Getting started guide - when you're new to any engine there should be some dedicated page where to start from. Basically it contains some step-by-step instruction for creating first sample using the engine/tool whatever. Having such page is the big plus to the documentation
  • General Usage guide - indeed for many different reasons we should refer to manual for some specific feature or some complex action. So, it would be great if we have consolidated set of pages for reference
  • API documentation - since both engines are used via API it would be nice to have them documented on the API level as well
  • Code examples - that would be another reference point demonstrating some specific feature capabilities. In most cases the only live sample is much better than tons of documentation pages
  • Specialized forums - every well-grown solution consolidates people around. They make communities, discussion forums. Such resources are very useful for discussing some specific cases or non-standard tricks which are quite frequent but problem-specific almost all the time
All those items are evaluated using boolean logic like "exists" or "doesn't exist". And again, our comparison will be more targeted to Java platform, so if something exists for Cucumber but without similar stuff for Java it's treated the same way as it doesn't exists. So, here is the comparison:
FeatureAvailable in
JBehave
EvidenceAvailable in
Cucumber
Evidence
Official siteJBehave Official site Cucumber Official site
NOTE
There's no dedicated site for Cucumber-JVM however it's quite stupid to say that there's no official site at all. So, for now this option counts for Cucumber JVM but for any next split decision point I'll tend to say rather no than yes for feature availability
Getting started guideGetting started pageThe information about that is distributed between various samples and blog posts but there's no dedicated page showing all steps in Java
General Usage guide The reason is the same as before. There's no dedicated reference to Java usage as there're too few Java-specific features
API documentationJBehave Javadoc modulesJavadoc main page
Code examplesJBehave examples: A dime a dozen modulesCucumber examples
Specialized forumsJBehave - BDD LinkedIn Group
  • Cucumber group on LinkedIn - quite populated place with big number of active discussions
  • Cukes Google group

Based on above features I can evaluate JBehave and Cucumber with the following grades:

EngineGrade
JBehave3
Cucumber-JVM2
Well, I wouldn't say that Cucumber has weaker documentation. The key thing is that Cucumber is spread across multiple platforms and it's the core part which is mostly documented. At the same time JBehave is more specialized. So, the documentation is more targeted and technology specific.

Flexibility in passing parameters

In this section we'll compare JBehave and Cucumber capabilities against their flexibility in building instructions and varying them in different fashion to cover more specific cases which correspond to the same action. Here we'll check the following options:

  • Parameter variants - in some cases we have fixed scope of possible values to pass as the parameter. Instead of using generic regular expression we should be able to enumerate acceptable options. That's what's done using parameter variants
  • Parameters injection - in some cases the sequence of the input parameters to method is not the same as we see convenient for our phrases. In such cases we should be able to define explicitly which part of the text goes to some specific parameter
  • Tabular parameters - when we pass some complex structures or arrays of complex structures there should be some mechanism to group them in granular way so they are still readable and at the same time they're convenient to parse. Tabular parameters is one of the ways for that
  • Examples - that looks similar to tabular parameters except this feature is used for different purpose when we want to run the same scenario against multiple different input parameters
  • Multi-line input - relatively small feature however it's quite frequently used and it's nice when such feature is supported
  • Extra features - any other features. They're not so essential as previous features however they're nice to have
So, the following table compares JBehave and Cucumber against the above features:
FeatureAvailable in
JBehave
EvidenceAvailable in
Cucumber
Evidence
Parameter variantsPattern VariancesSample code
Parameters injectionReference pageSomething similar
Tabular parametersTabular parameters in JBehaveDataTable samples
ExamplesParametrized scenariosScenario Outlines
Multi-line input Docstrings sample
Formatted inputParameter converters - nice feature for passing complex types within one instruction + the ability to customize conversionSample code containing an example. In particular:
@Given("^today is (.+)$")
public void today_is(@Format("yyyy-MM-dd") Date date) {
 calculator = new DateCalculator(date);
}
There may be some other small features but in most cases they're rather some small syntax sugar or missing just because there's no need of them in some specific engine.

After the above features comparison we have the following grades for our engines:

EngineGrade
JBehave2
Cucumber-JVM3
Looks like it's getting hot with such battle. No wonder, as we compare quite well-grown engines which have a bunch of features.

Auto-complete

This is very useful feature as when the test automation solution grows the number of keywords is growing as well. So, after some time it's hard to keep all phrases in memory. Indeed, some standards and agreements may help with that but only partially. The most effective helper for writing BDD instructions is an auto-complete ability displaying the most applicable options for the typed text.

Since the feature is useful it's represented with various plugins for IDE:

NOTE
Since both plugins are applied to the same file types they may be in conflict with each other. So, in order to omit such problem it's recommended using only one of those plugins
So, both engines have such support. That's reflected in the evaluation scores:
EngineGrade
JBehave3
Cucumber-JVM3

Scoping

Scoping is needed to narrow down the set of classes which would contain necessary bindings of simply get the sub-set of tests. So, there're 2 ways of stope restriction:

  • Tests scope - identifies which sub-set of tests to run. Additionally it can be sub-divided into:
    • Feature/story scope - only some specific stories are in scope
    • Scenario scope - only some specific scenarios are in scope
  • Step scope - identifies which sub-set of steps to use while looking for step binding
That makes 3 criteria which fits the grade scale. So, we can evaluate both engines against those features:
FeatureAvailable in
JBehave
EvidenceAvailable in
Cucumber
Evidence
Feature/Story scopeMeta-filteringTags
Scenario scopeMeta-filteringTags
Step scopeThe scope of classes is defined during at the configuration so steps are initially scoped by designFor Cucumber it's treated as rather antipattern than some built-in solution
So, based on the above features we're getting the following grades:
EngineGrade
JBehave3
Cucumber-JVM2
Well, the difference here is not so essential and mostly based on ideological aspects however things which are objectively useful are available in both engines.

Composite steps

This feature is useful when we'd like to re-use steps so that we can call some test instructions by code bound. However, since all the steps are actually annotated methods this feature is not essential. The Composite steps in JBehave is built-in feature. It is also available in Cucumber but:

  • These's no trace of it in JVM part
  • There's no actual need of it in Java implementation
  • If there's a real need of it it can be done using reflection
So, JBehave supports composite steps as full-scale feature. Cucumber-JVM can do the same but it's rather workaround than something fully supported. So, it makes the following grades:
EngineGrade
JBehave3
Cucumber-JVM1

Backgrounds and Hooks

This part of features covers the ability to perform some additional actions prior/after main events like feature/scenario/step execution. I've distributed this group into Backgrounds and Hooks as these features are similar by idea but fundamentally different by the purpose. When backgrounds are the part of the scenarios the hooks are rather part of the engine and their usage is mostly intended to expand the default engine behavior. That's why we'll evaluate engines accross those features separately.

Backgrounds

Backgrounds are supported by Cucumber however that's the weakest place of JBehave. There's no backgrounds there yet. That's one of the major things which don't conform the common Gherkin standards which are used accross many other BDD engines. Hope, that would be added in the nearest future but for now we have the following grades:

EngineGrade
JBehave0
Cucumber-JVM3

Hooks

For hooks the situation is a bit more tricky. Hooks are available in JBehave as well as they're defined for Cucumber though the Java implementation has 1 major restriction: there's no hook before/after step. That's valid for both engine. All other hooks are defined and if necessary we can include hooks for test engines in use (JUnit or TestNG). But it's a matter of fact the hooks cover only some part of possible cases. Here is the comparison table:

HookAvailable in
JBehave
Available in
Cucumber
Before/After step
Before/After scenario
Before/After feature
Before/After run
So, both engines are equal here and both of them have the same grade:
EngineGrade
JBehave2
Cucumber-JVM2

Binding to code

Well, this check-point is redundant here as JBehave and Cucumber use the same approach for binding text instructions to the code using annotations. This is the most compact way for that. So, we'll evaluate both engines with maximal grade:

EngineGrade
JBehave3
Cucumber-JVM3

Formatting flexibility

Before I compared multiple engines against this characteristics and JBehave got minimal grade just because by default it doesn't ignores heading spaces before Given\When\Then statements. That's frustrating gap. However, if we dig deeper we can find that JBehave has StoryLoader interface and we can implement loadStoryAsText method to handle heading spaces and use it. Thus, we won't have such problems with formatting. Despite such flexibility isn't enabled by default it can be included using standard libraries. So, in this area JBehave isn't far behind and both engines can be evaluated with the following grades:

EngineGrade
JBehave2
Cucumber-JVM3

Built-in reports

Indeed there's nothing we can do with the testing unless we have some representative results. So, reporting and results representation in general is the vital part of any engine. Also, there can be some other reporting types showing rather usage statistics than test execution status. It's useful information as well (e.g. it can be used to analyze how frequently we use the steps). Also, different formats are applicable for different areas. E.g. the HTML report format is nice to have as an output artifact while it's not convenient to see it in the console output. So, there're different report types and the more of them are available the more information we can get from run results. So, let's compare JBehave and Cucumber against supported report types:

Report typeAvailable in
JBehave
Available in
Cucumber
Console output
Pretty console output
Structured file (e.g. XML)
Well-formatted readable file (e.g. HTML)
Usage report
Extra report types
One thing to mention here is the availability of extra report types. The JBehave has PrintStreamOutput which allows customizing the output and write results in your own format. At the same time the Cucumber has various extra pre-defined formats. Again, these're likely bells and whistles but yet that's nice additional feature. So, that makes the following grades:
EngineGrade
JBehave2
Cucumber-JVM3

Conformity to Gherkin standards

Both JBehave and Cucumber provide extra layer for test instruction definitions. It's the level of scenarios in natural language which are represented as external source for the core engine. It gives an ability to switch the core code or the entire programming language when necessary. Or it's also good when you write the API for multi-lingual use. In this case we should test it against the same scenarios but using different implementation basically depending to the programming language. For this purpose it is important to have all libraries in use to conform the common standards. For BDD engines it's the Gherkin. Many engines support it so if you create tests using one of those engines and then you decide to switch to some other platform you don't need to change at least test scenarios. That's why the compliance to some common standards is important. In this section we'll take one of the most frequently used keywords/attributes and check how do they conform Gherkin.

Keyword/attributeAvailable in
JBehave
Available in
Cucumber
Comment
FeatureIn JBehave the analog is Story. However, there's support of Gherkin style keywords which includes support of the Feature
Scenario 
Given\When\Then 
And\But 
Scenario Outline 
Examples 
BackgroundNo backgrounds no support of it
TagTags have the same format but JBehave uses Meta keyword for that and that brings differences
Tables 
Given the above results I can conclude that JBehave conforms Gherkin by near 2/3 while Cucumber is 100% compliant. That's reflected in grades:
EngineGrade
JBehave2
Cucumber-JVM3

Input Data Sources

In some cases it's useful to store scenarios somewhere at the external location e.g. some tracking system. I'd say even more. If we want to involve many different people in tests automation we should use some shared resource accessible to everyone where those scenarios are defined. Also, some scenario may be an input for other scenarios or we simply need to reduce the copy/paste for repetitive set of instructions the same way we do while writing the program code. So, that's what all those features are about. So, we'll check them separately.

External Data

It's needed when we want to customize the input for our test instructions. For this purpose JBehave has StoryLoader interface and we can implement loadStoryAsText method which can be defined the way we need and read data from whatever we can reach. I won't even dig too deep to get some example. Recently, I made an integration between JBehave and JIRA. What about Cucumber? Nothing at the moment.

Nuff said. That's the score:

EngineGrade
JBehave3
Cucumber-JVM0

Inclusions

The inclusions are needed to minimize code duplications. That's why they're are used in code, that's the purpose for scenarios as well. JBehave provides something similar to that named Given Stories it runs the stories specified in the referenced file prior to main story. It's not pure inclusion and mostly it can be omitted by groupping stories into one file. However, there're some cases where something like that is needed. Unfortunately Cucumber doesn't support this feature. Well, in some sense it's even good as such distributed scenarios are harder to trace and harder to maintain as it leads to external dependencies resolution. The funniest thing here is that such feature could be really needed for backgrounds as a lot of cases have some common pre-condition accross the features (usually it's login/logout operations). But Cucumber doesn't support inclusions and JBehave doesn't support backgrounds.

For this feature we'll make the following evaluation scores:

EngineGrade
JBehave2
Cucumber-JVM0

Well, generally speaking the external data support is the weakest place in Cucumber. It was as good as JBehave in many other features but here it fails by all points.

Summary

It's time to summarize all the results and join them under total score table. Here it is:

Engine\FeatureDocumentationFlexibility in
passing parameters
Auto-completeScopingComposite
steps
Backgrounds
and Hooks
Binding
to code
Formatting
flexibility
Built-in
reports
Conformity to
standards
Input Data
Sources
Total
BackgroundsHooksExternal
Data
Inclusions
JBehave323330232223230
Cucumber-JVM233213233330028

Well, that was a good battle but JBehave appeared to be stronger at the end. However, Cucumber appeared to be quite strong opponent despite Java is not the main area of it's applicability. It simply lost some score points just due to some features were not migrated to Java yet. Nevertheless, I can conclude that both JBehave and Cucumber are equally applicable for Java-based automation and the choice between them is mostly based on what you need. If we think about choice between those engine I'd propose the following: if you look for some sort of "golder standard" the Cucumber is for you. But if you want something more you should look at JBehave

10 comments:

  1. Brilliant synthesis, this how comparisons should be done!
    p.s. found that JBehave may support Gherkin more now
    http://jbehave.org/reference/stable/story-syntax.html
    but i have not even tried using JBehave, so this is purely based on that link (im deciding which to use myself :)

    ReplyDelete
  2. I see that the last update of this page is Apr 14, 2013. That's 2 weeks after I posted this article. As I can see there's support for "Feature" and "Scenario Outline" keywords which were marked as non-supported for JBehave. I'll update comparison table with that information however that wouldn't change any grades at the moment (tags and backgrounds are still out of standard).

    Nevertheless, thanks for the link. I always should be reminded that actively developed software acquires more and more features.

    ReplyDelete
  3. http://www.thucydides.info/ support to JBehave adds another benefit of nice reporting.

    ReplyDelete
  4. Also, Web runner and Story navigator in Jbehave gives it some edge too.

    ReplyDelete
  5. I've worked with both JBehave and Cucumber JVM.
    Cucumber JVM is my favourite as it is slightly easier to use.
    I find the JBehave documentation to be extensive but hard to understand.

    ReplyDelete
    Replies
    1. I agree with the statement that Cucumber is easier to use. But the problem is how to explain it in some measurable fashion. All the characteristics described in the chapter were based on some observable facts. And yet, some things were considered quite subjectively. So, I could include something saying about usage easiness. But I don't know how to express it in some comparable form? Once I have it that would be another good comparison criteria

      Delete
  6. Used both of them. Probably you're right - Cucumber is easier to use, but let's clarify about comparison criteria. Do you compare them just from user point of view?

    But let's be fair. Just a few words in support of JBehave:
    1. I don't understand why tags are marked red for JBehave. It has tags anyway. You need to use meta keyword, but it's just JBehave way, let's say it's another syntax.

    2. Meta keyword is very powerful in JBehave, it also allows you to set values to tags and filter by tag values.
    Take a look at these articles:
    http://jbehave.org/reference/stable/meta-info.html
    http://jbehave.org/reference/stable/meta-filtering.html
    http://jbehave.org/reference/stable/meta-parametrisation.html
    I'm not sure this stuff is supported by Cucumber.

    3. GivenStories. Does Cucumber have this?

    ReplyDelete
    Replies
    1. 1) Tags are marked as red only at the "Conformity to Gherkin standards" section. It's just because if you take any other engine like Cucumber, SpecFlow, Behat, Freshen and many others, you'll see the keyword "tag" rather than something else. In other words, JBehave has it's own syntax while other engine including Cucumber support some common standard. That's the explanation why Tags are marked red there. Imagine, you are using some other language apart from Java. With JBehave you're tightly bound to JBehave specifics while the language supported by Cucumber is more universal. And that was the aim of that section
      2) Take a look at the "Scoping" section. It contains references there. And it's marked that JBehave is better there
      3) GivenStories - see the "Input Data Sources/Inclusions" section. It's mentioned here and it's mentioned as an advantage of JBehave.

      Delete
  7. Thanks. Agree. JBehave has its own specific dialect, sometimes it looks very confusing, because Cucumber version of Gherkin is rather language standard than just another dialect.

    Anyway we can spend years discussing differences:) Thanks for this article. It's very nice indeed.

    ReplyDelete
  8. Very helpful comparison especially for me who is researching the myriad of BDD tools for implementation. Since both of them are running neck to neck in comparison and since I am new to this whole stuff left indecisive which one to choose.

    ReplyDelete