Ten Tips for Starting Test Automation – Part 2/2

If you didn’t yet catch the first part of the post, you can find it here.

As weak as your test data

To test deleting a bank account, you need to have a bank account created first. To test user login, you need a registered user information created. Practically all testing relies on the use of test data. How this data is created for testing purposes depends on the test scope and the context. In unit and integration tests, the test data may be generated using suitable mock interfaces and in-memory databases initialized at test setup, whereas a system-level test may rely on real database populated with SQL script, and so on.

Especially for end-to-end testing of complex systems it may not be always trivial to automate the creation of realistic test data, and you may be tempted to use manually created test data in your automated tests. Although using manually created test data may be an applicable solution in some cases, it has in general many limitations that may bite you back later. Manually created data may expire unless maintained periodically, you may not be able to edit or delete it if other tests rely on it, and running multiple tests utilizing same data can cause flakiness and unexpected behaviour. The worst case would be using shared test data, such as data used both by automation and exploratory testing, that may change its state uncontrollably.

Find ways to automate the creation of test data. Each test should rely on unique data that it may use, modify or delete on test execution. Ensure test data is always in a known state before test execution. Also remember to test the product with realistic data set sizes that are expected in production use.

Success of automation depends on the quality of test data. Avoid use of manually configured, shared or uncontrollable test data. Find ways to mock, populate or virtualize data sources in your test environment to be suitable for your testing needs.

Works on my computer

I bet everyone has heard someone retort “Works on my computer”? Too much time is wasted on investigation and fixing issues related to differences in various test environment configurations causing the software behave unexpectedly.

Most often these problems are a consequence of manual setup and configuration of test environments leading to configuration nuances between different environments. Manual configuration is time consuming, prone to human errors, non-reproducible, and lacks rollback support to previous configurations. As a result, what works on your environment may not work on your colleagues’ - seemingly similar - environment and vice versa.

Another challenge are unstable test environments that may fail randomly during test setup and execution. Unstable environments create misinformation in terms of false test results, eat up credibility of the testing, and analysing these errors take valuable time away from actual testing work. In worst case these environmental issues can kill the benefits otherwise gained from automation.

Find ways to automate the setup and management of test environments. Use configuration management and virtualization tools (e.g. Docker, Puppet, etc.) to create infrastructure-as-code configuration that can be used throughout the development pipeline to setup reproducible development and testing environments for different phases and needs.

Focus on creating a pipeline from developer workstation to production where environments work in similar manner. Avoid manual configurations and create automated scripts for creation, configuration and deployment of test environments with suitable tools.

Timing is everything

The world is asynchronous. Whether your tests are expecting response from native GUI, web application (browser), API or proprietary interface, the expected events will most likely be happening in an asynchronous manner. If your tests are not prepared for this, they may become flaky and fail unexpectedly due unknown response times of the application under test.

Typical example of this is Selenium WebDriver based browser tests that can be extremely flaky unless appropriate waiting of correct web element state is used. In case the test does not wait an element to become present and visible before trying to interact with it, the test may fail unexpectedly. Too often these types of issues are resolved by adding fixed delay (sleep) before interaction. Although a common practise, it is a bad approach making the test slow to execute with no guarantee of stability unless excessively long delays are used. Preferred solution in this context would be to set up an explicit wait that waits until the element is ready for interaction.

Regardless of used test tools and context, it is important to understand the implications that timing handling has to your tests, and how to mitigate them correctly to avoid test flakiness. Identify asynchronous events in test interactions and apply solutions to handle them without fixed delays.

In your tests, know what is expected to happen and wait for it. Don’t rely on fixed delays that will slow down test execution with no guarantee for stability.

Simplify, isolate the unknown

When developing automated tests, keep them as simple as possible. Keep tests short and avoid adding too much content in a single test (suite). Create test suites that focus on a specific feature of your product.

Create tests that are atomic and well focused. Atomic tests are order independent and not dependent on the execution or results of other tests. Use suitable setup and teardown functions to keep the product in known state to achieve atomicity. Each test should have only one focus that it tests. Failure of test should explicitly identify the problematic part in the product without excessive result analysis.

Tests should also be resilient to changes outside the test scope. Avoid using product features outside your defined test scope that may be subject to unexpected changes. As an example, if you are testing an embedded application that is part of a larger web portal, create tests that focus only on that specific app. When interacting with the application under test, create selectors that are resilient to changes that may happen in the portal page outside current test scope.

Keep your tests simple. Tests should tell directly what part of the product failed. Avoid vague shotgun tests without known scope. Create atomic tests that are resilient to changes happening outside the defined test scope to avoid test maintenance work and false negatives.

Don’t repeat yourself

When working with multiple projects, or even single project with multiple test cases, avoid writing duplicate test code. If the same test step or functionality is used in multiple test suites and projects, modularize the functionality as its own test library that can be imported and reused in other projects.

Creating reusable test modules and libraries brings multiple benefits. First, creating new projects will become easier as you will have a growing set of reusable functions available vs. always starting from the scratch. Second, it will ease the maintenance of the test assets when product functionality changes. When the functionality changes, the changes need to be done only in a single point in test code instead of refactoring all the related test projects and test cases.

As you start creating reusable test libraries for your product, don’t forget to document and communicate availability of those libraries within your teams and organization. Otherwise there is a risk of individual teams creating duplicate test libraries causing risk of confusion and extra maintenance effort.

Reuse, reuse, reuse. Develop test modules and libraries to unify the way your tests behave. Avoid duplicate code and the need to fix same issue in multiple places.

Final thoughts

Getting started with test automation can feel overwhelming. If you are a tester starting out automation without earlier programming experience, the learning curve can be especially steep while learning both coding and automation basics in parallel. But don’t stop. Keep going, learn and remember that no one is a master in the beginning.

The great news is that there are great resources and communities to support your journey in automation. When stuck or unsure about the best way to continue, seek help. Check out available documentations, tutorials and resources on the Internet, read a book, or ask a question in testing communities such Ministry of Testing or Testers.io.

Also check out available training and commercial offerings near you. There are many great services and companies with automation expertise to support, coach and ensure successful adoption of test automation in your organization.

Start, keep going and get connected. There are great communities, events, trainings & partners to support your automation challenge.