Running features in behavior driven development (BDD) automation test suite in parallel is fairly straightforward. With NoCodeBDD, running features in parallel is a default option and can be switched between serial and parallel run through click of a button, as shown below:
Enabling BDD to run in parallel is the easiest part. However, there are several things you need to handle in order to make your BDD Automation Suite to run in parallel. This article explains the list of issues that I encountered in a large project that I was working on and the steps we had to take to move from a long running serial BDD automation test suite to parallel. We were using Cucumber in the project. You can also check out the following article, which details the lessons learned in rolling out BDD in a large project: https://blog.nocodebdd.com/lessons-learnt-in-automating-behaviour-driven-development-bdd-for-a-large-project/).
Maybe everything would have been different. Hindsight is a mocking bitch for sure. – Author: Tiffany King
Yes, quite a lot would have been different for us if we had started running behavior-driven development (BDD) workflows in parallel pretty much from day one. Any test suite should give fast feedback to developers. Here is a great article from Stackify on why faster feedback is important https://stackify.com/continuous-testing-feedback-loops/.
During the initial stages of the project, when we had few feature files, running BDDs serially took only minutes. As a result, it didn’t impact our delivery. However, as the project grew, we added several feature files, meaning our Automated BDD test suite took longer and longer to run.
Finally, we ended up with a run time of more than an hour to run the BDD test suite. Though this may not sound like a major issue, all the developers running the entire BDD suite may amount to multiple times each per story. It soon became evident that of all development tasks, running BDDs took most of the time.
The prioritization of running BDDs in parallel came after implementing a large BDD test suite toward the middle stage of our project. Then, it became really hard to go back and refactor the code, as the stories were in flight; we thus didn’t have enough time to address this. To get the BDDs to run in parallel, we had to:
- Pass unique IDs in each step
- Create stateless scenarios
- Test data set-up
- Find a mechanism to handle asynchronous responses
- Use tags to separate scenarios that can’t be run in parallel
- Avoid coding mistakes
Let’s look at each of these points in detail.
While writing feature files, we reused IDs between feature files, scenarios, and steps. These IDs were used for database transactions and message queue (MQ) transactions. That’s any delete/update operation on the database, and pulling off the Q should be based on the ID. This was perfectly fine as long as the test suite ran in parallel and didn’t work when we tried to run test suite serially.
Consider the following two feature files:
Feature 1 Feature: Check Employee Status Scenario: Check employee status is "Permanent" Given database doesn't contain employee id "EMP001" And database is populated with employee id as "EMP001" and status as "Permanent" When employee status check api is called to check status of employee id "EMP001" Then response message should contain status as "Permanent" for employee id "EMP001"
Feature 2 Feature: Get Employee designation Scenario: Get employee designation as "System Consultant" Given database doesn't contain employee id "EMP001" And database is populated with employee id as "EMP001" and designation as "System Consultant" When employee designation check api is called to check status of employee id "EMP001" Then response message should contain designation as "System Consultant" for employee id "EMP001"
When we tried to run these two features at the same time, we experienced data collision: feature 1 could override feature 2’s data, and vice-versa. To get these two features to work, they must use Unique IDs (i.e., EMPxxx), and the DB and MQ code must change to access unique values.
Keeping scenarios stateless is a general BDD practice, and it’s even more important when it comes to parallel running. Keeping scenario stateless means that a scenario should not be dependent on running another scenario. This dependency would make it difficult to maintain your BDD test suite. When it comes to parallel run, if there are dependencies between scenarios, you can’t really run them in parallel.
Setting and clearing up test data
When it comes to parallel run, setting up and clearing up the right test data is critical. In your Given step, make sure any test data that you set up is unique to that scenario. For example, if you are testing a user search scenario, make sure the data you set up in the given step containing a list of users is unique to the scenario. If your test data set up is not unique, then when you run your test suite in parallel, you might end up with data collision. This is hard to debug.
Find a mechanism to handle asynchronous responses
Asynchronous responses can be anything, such as a response received from a website asynchronously. For example, after a user action, the response would be received sometime later. The user can continue to progress without waiting for the response, e.g., a particular element in the UI will be shown after few seconds. Asynchronous responses are quite a common pattern when using MQ.
One of the efficient ways to handle asynchronous responses is to first define an SLA of the response message. In your BDD, implement a solution in such a way that you assert for the expected value every few seconds until the SLA expires. If you can’t find the expected value within the SLA time, then fail the step.
For an example, after clicking a button in a web page, if you expect a particular message or element to show after few seconds, then write your step definition so that you keep looping to check for the element or the message every few seconds or milliseconds until the expected time. This time should be given to you as part of requirement, and it’s one of the points to discuss during scenario discovery if there is a similar use case.
Use tags to separate scenarios that can’t be run parallel
As your BDD scenarios grow, you might have a few scenarios that you can’t run in parallel. You could tag these scenarios separately and run them as serial either before or after you run the rest of the scenarios in parallel. This would give you great flexibility in moving only a small subset of scenarios, which may be either difficult or not possible to run in parallel.
One of the cardinal sins we committed at a very early stage of the project was the usage of static variables in Java step definitions to share state. To make things worse, a Java class with static variables was used as a base class, and all step definition classes extended this parent class. As the project grew, when BDDs took longer to run due to number of feature files, we couldn’t run them in parallel because we had state collision due to static variables. It also became very difficult to refactor without doing a complete overhaul of the entire BDD framework.
When implementing step definition, take care of all coding issues that you need to follow to handle parallel scenarios. You also need to have a strong coding practice and review process in place, as implementing and maintaining code to run scenarios in parallel is really a difficult task to achieve
Running a BDD automation Suite in parallel from day one has huge benefits. Most projects start with a serial run, and when they realize the automation suite takes time, for many, it could get too late to move to parallel and worst case they may even move to manual testing. I learned this the hard way; thus, in NoCodeBDD, we made parallel run as default. You can download a free version of NoCodeBDD by going to www.nocodebdd.com/download.
Jerome Josephraj is the Founder and CTO of NoCodeBDD. He has more than 20 years of experience in Software Development and have been rolling out Behavior Driven Development for large projects in United Kingdom. He has a strong Java Development Background and have been working as a Technical Consultant. In one of his recent large enterprise projects, he has successfully rolled out Behavior Driven Development consisting of more than 1000 scenarios. He was also the Founder of http://www.campusinteraction.com, a student analytics portal, which was later bought by Preludesys. Based on his previous start-up experience and a strong Behavior Driven Development experience, he founded NoCodeBDD to accelerate BDD automation by 10x.