A lot of debate surrounds whether behavior-driven development (BDD) is good for UI Automation. In our experience, using BDD only for UI Automation has a few issues:
- Automating BDD at the UI layer tends to make your BDD automation suite brittle, flaky, and not scalable
- Teams tend to focus on UI elements more than the behavior of the application
- When it comes to UI, teams tend to define BDD scenarios as manual scripts (check BDD imperative vs. declarative: https://community.saucelabs.com/forum-45/best-practice-imperative-v-declarative-testing-scenarios-160)
In this article, let’s look at these issues in detail and discuss an alternative that works fantastically well for many of our clients—something that you could easily adopt.
Definition of BDD?
According to Wikipedia, BDD is
an agile software development process that encourages collaboration among developers, quality assurance testers, and customer representatives in a software project. It encourages teams to use conversation and concrete examples to formalize a shared understanding of how the application should behave.
The key, operative words in the definition of BDD are collaboration and concrete examples to develop a shared understanding of how the application should behave.
This is not only critical for any team rolling out BDD in their project, but also for any team delivering high-quality software. This article covers more about BDD collaboration and an awesome technique called OOPSI.
Let’s pin the above definition, as it forms the basis of any discussion around automating at the UI layer. Before we look at how automating at the UI layer is not the right approach, let’s look at what it means to automate at the UI layer.
What is Automating BDD at the UI layer?
Simply put, automating BDD at the UI layer involves BDD scenarios using automated scripts in Selenium, Cypress, Appium (for mobile) or using scriptless tools like NoCodeBDD. The following video shows how this is done using NoCodeBDD in minutes, without having to write any code:
Issues in Automating BDD at the UI layer
When BDD automation—or any automation, using BDD or not—is done at the UI layer, it could suffer from several, if not all, the following issues:
- Not scalable
- Doesn’t test the behavior of your application
- Paves the way to imperative BDD steps, which are similar to manual test scripts and don’t bode well for automation
Let’s look at these issues in detail in the below sections.
When BDD scenarios are automated for the UI layer, particularly if they are automated using XPath, they are almost certain to fail intermittently. These XPaths can change for various reasons:
- Page content has changed (e.g., static text gets included at the top of the page, which would change XPath for an element automation script is referring to)
- Each browser might have different XPath values (particularly when it comes to IE)
- Each browser resolution might have a different XPath
Even if you use IDs (i.e., element IDs), your UI layer is likely to change more often than the application system. As and when the UI changes, your BDD will break and will embrittle the entire BDD automation.
Flaky means that the tests fail intermittently. For example, when you run your BDD automation suite, it will show all scenarios have passed. But when you run the same set of tests without any changes to your application or BDD, it might fail. In other words, your BDD automation suite is not reliable: it’ll pass sometimes and fail other times, even if there are no changes.
This is most certainly the top reason why you should avoid automating BDD—or, for that matter, any automation—at the UI layer level. There could be several reasons why your BDD Automation would be flaky:
- Network speed – You might have written a script to check for an element, and it might pass on your machine. But when it is run on a different machine due to slowness, the UI element might take time to load; as a result, your test will fail. There are few techniques to handle this, but this has proven to be one of the contributors to flaky tests.
- Asynchronous calls – Your UI might make an Ajax call, and depending on the result from the Ajax call, it might take some action. When you automate for this scenario, your test might end up working intermittently because there is no guarantee of when the Ajax call might return.
- Dynamic Elements – These elements’ attributes change when the page is reloaded, even though the page content remains the same. These are difficult to handle and will make your test flakier.
- Device configuration – This particularly applies to mobile devices, as there could be many devices with an umpteenth number of configurations. Your UI might render completely differently in each version of the device; as a result, your UI automation might work in one version and not in another.
When your BDD is automated for the UI Layer, it is more likely to be slow. Since there could be several layers the UI has to traverse through, it would take that much longer to get the result when your automation scripts interact with UI. Thus, it affects the overall automation running time tremendously.
Over time, your entire automation suite becomes so slow as to become unusable. Agile is all about getting fast feedback. When your automation suite doesn’t provide this, the team becomes less agile. As a result, your time to market increases.
As more and more scenarios focused on UI automation get added to your BDD Automation suite, the combination of brittleness, flakiness, and sluggishness makes it less scalable.
When the team has only a few scenarios, UI automation might seem to work. Yet inevitably, the team will add more scenarios, and slowly all the issues discussed above will start seeping in. This all causes the automation suite to become less reliable and fail more often than not for issues that are not related to functionality.
Not Testing Behavior
Software products are always delivered to a user. You might write a backend software or a software that sits behind several other software—but at end of the day, your software is always used by a human. Therefore, it’s critical to understand what the user wants and what behavior software should provide to satisfy the user’s need.
Let’s put it simply: if a team needs to deliver a high-quality product, the first and foremost is they need to understand the requirements well. That’s the starting point. Get that wrong, and it doesn’t matter how swanky your technical architecture is, whether you have 100% automation, or tech superstars—everything would go wrong. They’ll end up delivering a product that works fantastically well—while delivering upon the wrong requirements.
How does this impact automating at the UI level?
When the team focuses on BDD automation at the UI level, they tend to focus on the website rather than the behavior of the application. In other words, they tend to lose track of the big picture and focus on the web elements.
Thus, the team may end up writing imperative scenarios, which are explained in the next section, rather than declarative. Declarative scenarios tend to focus more on the behavior and exactly what the customer wants.
Imperative scenarios are those that explain every action but not the behavior. An example of an imperative scenario is given below:
For an end user, these steps don’t mean much. In other words, when a customer gives a requirement for a registration page, they don’t give something like:
- User enters first name
- User enters last name
- User selects age
- User presses submit button
Customers give requirements from a behavior perspective something like:
- As a user, when I enter all mandatory fields on the registration page, I should be registered in the system and a dashboard should be shown. Mandatory fields are First name, last name, and age.
When the scenarios are defined as imperative, the team loses sight of actual requirements and ends up focusing on the details. As a result, they might end up delivering/testing invalid requirements, as the actual requirement is lost in the detail.
Conversely, declarative scenarios explain the behavior—not the steps. The following screen shows how the previous written imperative scenario is shown as a declarative scenario:
Imperative scenarios are also likely to change more often than declarative scenarios, which focus on the actual behavior. For instance, if capturing first name and last name is changed to capture just name, but in the background you still split the name and send it as first name and last name to the server-side, you end up changing your BDD scenarios—even though the behavior of the application hasn’t changed.
Despite all the aforementioned issues, it is important to have BDD scenarios at the UI layer level. Let’s look at why we might require scenarios at the UI layer level.
Why BDD Automation is required at the UI Layer level
When your application is quite heavily based on UI, it is important that you have adequate scenarios to cover that. Ignoring the UI layer and moving everything to the API layer might not work, as some of the necessary behavior might not get tested.
If your automation script focuses on API, then the automation script would pass all the fields that are required for the API to work and check if it gets stored successfully in the DB.
However, let’s assume the UI is developed in such a way that, when the user clicks the student or alumni radio button, it passes only the “student” value to API, even if the user selects Alumni; then the DB will always store the alumni type as student.
Obviously, storing alumni as students should be checked, and it should fail. Yet if the BDD scenario actually tests the UI layer, your script would actually select alumni and your scenario would fail: you’d be checking for alumni, but DB the record would have stored that as student. Because your UI to API mapping is wrong, all records are stored as student. So, checking the UI layer is extremely important in ensuring that all scenarios are covered properly and that your application is behaving as intended.
So how do you go about testing a UI layer without introducing all the issues mentioned above? The following section shows how you can use a technique that allows you to focus on UI scenarios while at the same time reducing the issues related to UI automation. This technique has worked fantastically for many of my own projects and those of many of our clients.
Solution; Use a Combination of API and UI
One of the best solutions that I have implemented in many large projects with thousands of scenarios is to effectively use a combination of UI scenarios and API scenarios. Using this technique, you basically split the scenarios for a particular functionality to check it using UI and via API.
The splitting of scenarios should be done in such a way that the UI scenario focuses up to the point where all the elements are covered; API scenarios should focus much more in-depth on the functionality.
Let’s look at an example of how this could be applied in a real-world application. The application for which the BDD automation is written has a UI, which looks something like this:
The API takes all the following fields and stores it in DB. You could split the scenario as follows:
The first two scenarios focus on UI. The first scenario checks if all entered information on the UI works as expected. In this case, it checks whether it gets saved in DB successfully. The second scenario checks whether all fields are validated successfully, and whether the right error messages are shown. These two scenarios should take care of all UI-related checks and make sure that the handshake between UI and API is working as expected (i.e., all data from the UI are transferred to the API and are stored in DB successfully; and all error messages are shown correctly at the UI level).
The next four scenarios handle all API-related scenarios, which is where the core business logic gets checked. Different behaviors and rules of your application can be checked in these scenarios and can be applied at the API level.
This approach efficiently shifts all heavy work—checking different application behavior and rules—to the API layer and keeps the scenarios to a minimum at the UI layer.
By combining the UI and API layers, you can achieve the best of both worlds: testing the UI layer and at the same time reducing the issues of automating the UI layer for all the BDD scenarios.
Using NoCodeBDD, both UI and API automation can be achieved in minutes. Download a free version of NoCodeBDD from https://www.nocodebdd.com/download
Jerome Josephraj is the Founder and CTO of NoCodeBDD. He has more than 20 years of experience in Software Development and have been rolling out Behavior Driven Development for large projects in United Kingdom. He has a strong Java Development Background and have been working as a Technical Consultant. In one of his recent large enterprise projects, he has successfully rolled out Behavior Driven Development consisting of more than 1000 scenarios. He was also the Founder of http://www.campusinteraction.com, a student analytics portal, which was later bought by Preludesys. Based on his previous start-up experience and a strong Behavior Driven Development experience, he founded NoCodeBDD to accelerate BDD automation by 10x.