Why Functional UI Test Automation (FT) Is Flaky?

It is widely accepted that functional UI test automation (FT) helps save QAs a lot of time so that they can focus more on User Acceptant Testing and Exploratory Testing. However, there are some drawbacks that prevent many organizations and teams from successfully adopting it. The cost of writing and maintaining scripts as well as the level of flakiness are typically higher than other types of testing. That is also the reason why some people often criticize UI automation testing.

Nowadays, web application evolves really quickly with Agile-style short development cycles. Moreover, In favor of agile development and deployment, the architecture of web applications on both backend and frontend side is getting increasingly complex with multiple components separated logically and physically that are only linked together via web services. Even with a substantial amount of automated unit tests and integration tests, doing FT is still necessary to make sure the software will behave as desired.

Therefore, instead of abandoning FT, we had been actively looking for a way to mitigate its defects in our team, and as a result, maximize its benefits and ROI. When we take a closer analysis, we realize that there are three main causes that reduce the effectiveness of UI automation testing:

1. The time spent on writing good and stable XPath/CSS locators, and update them whenever the underlying HTML markup changes.
2. The network itself is sometimes flaky which leads to test failures.
3. The code does not handle some rare cases such as race condition.

The third cause is actually by bugs inside the software and should really be fixed by the engineering team (don’t call it flaky tests!). The second one can be mitigated by putting the application under test (AUT) and the tests in the same data center with a robust network (e.g. Amazon Web Services). Therefore, we focused on resolving the first cause.

Testable UI:

To solve this problem, we need to understand why unit testing is highly robust and reliable. If you have googled around the internet on unit testing, you can see thousands of articles about writing testable code. Indeed, if the code is not structured carefully, the effectiveness of unit testing will be greatly affected, and it will take longer to write unit tests with good coverage.

Similarly, having a good XPath locator for an element is always easy even with QAs if the element were composed of unique and stable attributes. They may be made up by the developers themselves, or by the JavaScript libraries used inside the software. The list of these attributes, along with their changes, should be communicated timely between developers and QAs. We call this kind of UI “testable UI”, and the responsibility to write testable UI lies in the hand of the developers.

However, we do not let our developers writing automated UI tests. One reason is the budget – it depends on teams and companies with here we need to focus our development resources on build new features. Another reason is that QAs often do a better job in writing tests, because they focus on the users’ perspective (that’s what the word “functional” is for), while developers’ first priority is often to prove their code works (no offense!). Therefore, we came up with a very simple workflow:

1. Developers write the code.
2. They then write very simple automated tests to prove that they have covered all expected scenarios.
3. These tests are attached to corresponding Jira issues.
4. QAs write complete automated tests based on “hints” in developers submitted tests.

We call simple UI tests that developers write “functional UI unit tests”. They are very short and dead simple scripts that can be thrown away later. They should be composed without much time thinking about structure, design, coding convention etc.

This workflow brings about many benefits for both developers and QAs:

1. QAs can verify very quickly if the developers have understood and covered expected scenarios.
2. It encourages developers to make the UI testable, with all elements easy to locate. QAs can also verify the testability by taking a look at the functional unit tests.
3. The direct consequence of #1 and #2 is that the communication overhead will be reduced. QAs rarely need help from developers to resolve issues in the test scripts, or to clarify their understanding of scenarios.
4. QAs spend less time on writing XPath/CSS locators and instead can pay more attention to the stories and the test project’s structure, which should be treated as a complete and serious software project on it own.

For the next step, we need a good tool to ease this workflow.

Enter Katalon Recorder:

To help make the aforementioned workflow even more effective we adopt Katalon Recorder, a small extension for Chrome and Firefox. This extension is a companion tool with Katalon Studio – one of the best automation testing tools 2018 . Albeit its simple appearance, Katalon Recorder has a lot of hidden gems inside for us. The most important feature that we are making use of at this moment is its ability to generate XPath and CSS locators, with built-in locator builders as well as custom locator builders. There are other useful features and I will go through them in later sections.

If you are interested in Katalon Recorder, please install it in your browsers (Chrome version, Firefox version).

XPath/CSS Locator Builders:

Katalon Recorder let users generate XPath/CSS locators by point-and-click at the elements on the web page. Katalon Recorder includes a handful of locator builders – they recognize tags, attributes, IDs, classes and calculate as many different locators as they can.

What makes Katalon Recorder truly stand out from other similar tool is the ability to import custom locator builders. These builders are written in the plain-old JavaScript, which is our developers’ native language!. We have developed several locator builders – they are shared between different teams. The beauty of this feature is that all rules and guidelines for the locators are now encoded nicely inside those scripts – “document as code” as it is called. Here is a sample of a custom locator builder taken from their documentation:

LocatorBuilders.add('custom locator id goes here', function(e) {

if (e.id) {

return "css=" + e.tagName + '#' + e.id;

}

return null;

});

With the help of these builder QAs can generate recommended locators simply by point-and-click:

You can read more about this feature here.

Other Helpful Features From Katalon Recorder:

Custom locator builders are the most praised feature for us, but there are also other features that our developers and QAs have been using regularly, and they have saved us a lot of time.

1. Record and playback all actions, along with locators generated in #1.
2. Export recorded tests to other programming languages, with the ability to write custom exporters.

Record and playback

Tests in Katalon Recorder can be created by just recording your actions on the web page. They can be organized into test suites. Each step in a test consists of three parts: command, target, and value.

• Command: This column describes the type of action to be taken. For example, if we need to open an app then “Open” would be used and to enter a value “Type” is used.
• Target: In this column, element locators are defined. The locators are generated as described in the previous section.
• Value: This defines the value, for example, if we have a text box in which we need to enter a text. The value of the text would be defined under “value” column.

This design may sound unadorned to professional engineers, but that’s exactly what we expect – something dead simple to remove distractions from developers’ work. The complete tests are designed and written by our QAs after all.

Export tests to other programming languages:

Tests created by above methods can be exported into different programming languages. While the built-in set only includes popular programming languages, Katalon Recorder provides a platform for us to build custom exporters. They translate all tests into our in-house Selenium-based frameworks that are different from team to team. You can learn more about the platform here.

List of Katalon Recorder’s supported languages:

Conclusion

The XPath/CSS locators are just only one problem that needs solving in automated functional UI testing. There are other challenges that we have to overcome in order to integrate this type of test into the continuous delivery pipeline as an accountable component inside. These challenges and our approaches will be discussed in following articles.

If you are not a regular reader of this website then highly recommends you Sign up for our free email newsletter!! Sign up just providing your email address below:

Happy Testing!!!