Test Environment Management: The Secret Ingredient for DevOps

Nikhil Kaul, Senior Product Marketing Manager at SmartBear Software

The rapid adoption of DevOps necessitates continuous delivery and a shift toward higher levels of test automation. As a result, QA teams often invest a lot of time and effort in ensuring their tests are automated as a part of the continuous testing cycle. The right test automation tools allow teams to increase test speed and coverage. In fact, a common framework agile teams use while navigating this process is the test automation pyramid – an approach that focuses on automating tests at three levels: unit, service and user interface (UI).

At the base of this pyramid is unit testing. By running more unit tests at a lower level, teams can get feedback faster and develop a solid testing base to build upon. The intermediate layer consists of API tests, which work under the UI of an application. Finally, the top of pyramid represents end-to-end UI tests.

To implement a test pyramid approach, teams spend a lot of time and energy hunting for the right test automation tools. As an example, for unit level testing, teams use JUnit or NUnit. Similarly, for API level testing, teams love SoapUI, while for UI testing, TestComplete or open source tools like Selenium could be used.

A DevOps chain can only be as strong as its weakest link. And often one area that is neglected during such a test pyramid approach is the way teams go about managing their test environments.

Test automation efforts often falter as the efficiency gained by investing in test automation is lost when teams still maintain the test infrastructure manually. In the case of UI testing, teams can spend hours provisioning, maintaining, monitoring and tearing down the underlying test environment infrastructure.

As an example, without a proper strategy, teams spend a significant portion of their testing cycles ensuring the right testing environments are available. And often, even when these environments are available, there are large discrepancies between test environments and production environments.

Traditional Ways of Managing Test Environments

There are three common tactics teams use to manage test environments. One of the easiest ways is to use local machines (i.e., browsers hosted locally). The second way is using virtual machines such as Amazon EC2. Finally, the last way teams go about test environments is by leveraging device labs, which primarily consists of rented or bought mobile devices.

Regardless of which of these three routes a team takes, there is manual work required to maintain and upgrade environments when new browsers, operating systems, devices or resolutions are introduced.

In addition to the manual efforts required to maintain these UI test environments, there are other challenges such as reduced speed to delivery, redundancy and discrepancies.

Speed and Agility Concerns

Setting up environments on local machines as a part of your testing cycle automatically slows down the entire process. QA teams frequently find running UI tests on virtual machines can help overcome up front hardware costs, but there is still a manual component involved in ensuring the right configurations are available when tests are being kicked off. Provisioning clean test environments often means spinning up new VMs with new configurations which can be time consuming and adds to the labor cost.

Parallel Execution Needs a Lot of Work

Having an ability to run tests in parallel across various browsers and OS combinations can be useful to speed up the testing process and increase coverage. However, often setting up parallel execution capabilities for UI tests with traditional test environment management efforts like local machines, VMs and device labs can be a painstaking process.

To run UI test on different devices, teams need to set up a hub with multiple node devices. Registering the node to the hub involves specifying arguments like browser type, version, hostname, port etc. If these arguments change or get updated, QA might have to go back to the command line or configuration file to update. Additionally, adding an iPad or an Android device as a node requires even more additional steps.

Redundant Activities

Setting up environments locally results in duplicated efforts as each individual has to perform the same exact activity. In case, teams are maintaining an in-house device lab, the redundant efforts associated with upgrading various browsers and OS combinations can grow exponentially. Keeping systems and devices up-to-date becomes more challenging as teams grow and new environments (web browsers, operating system version and resolutions) are added to the mix to improve coverage.

Increased Cost

Setting up local environments, using VMs or device labs can often result in cost overruns. In fact, costs encountered while using these options can be primarily broken down into three different buckets: device costs, labor costs and licensing costs.

Device cost represents the cost associated with having devices with the right configurations of operating systems, resolutions and browser versions.

Labor cost consists of the time and effort your operations, test or DBA team spends on maintaining, upgrading or even tearing down test environments, database servers and labs, among others.

Licensing costs include the inevitable part of using licenses of additional software to ensure environments are working as expected, both with device labs and VMs. An example, you’ll likely want to know that if and when your test environments go down. Setting up this type of notification and monitoring system for your environments requires additional spend on a third-party vendor.

UI Test Environment Management: What to Look for?

While looking for the best way to optimize and manage your test environments, there are four key aspects to take into consideration.

1. Test Environments Should be Easy to Install

Easy installation helps ensure that application development and delivery (AD&D) teams don’t spend considerable amount of time with set up and configuration.

2. Test Environments Should be Easy to Enhance

If a new browser version is released, the needs of your end-user will change accordingly – forcing you to modify your test environments. Therefore, easy to enhance test environments can thereby be really helpful, especially in a DevOps environment.

3. Test Environments Should be Easy to Share

The easier it is to share test environments among team members, the more scalable the process becomes. The time it takes for your set of tests to complete a run can be drastically reduced by running UI tests in parallel. To do that, environments should be easy to share among team members.

4. Test Environments Should be Easy to Tear Down

As an end user yourself, you should be able to spin down, reconfigure or reinstall your environments as needed.

The Right Way to Set Up UI Test Environments

Having cloud-based test environments available on demand can help overcome scalability, cost and maintenance challenges posed by use of local machines, virtual machines or an in-house testing lab.

Cloud-based environments, like TestComplete Environment Manager, give on-demand access to over 1,500 combinations of browser versions, operating systems and resolution settings. Application development and delivery teams can easily and quickly execute and report on automated UI tests across test environments without setup or configuration.

Other benefits of using cloud-based test environments include:

Easy to Install, Enhance, Share and Teardown

Cloud-based test environments are incredibly easy to manage and are accessible anytime, eliminating the need for an installation process altogether. You simply need to link those environments to your tests. Cloud-based environments are also simple to enhance. When new browser versions are released, a good cloud-based environment solution will update these, thereby eliminating your need to spend time on upgrades.

As you’re accessing a cloud-based environment through a URL, it is easy to share with anyone, enabling you to scale as needed. Tearing down environments in this case is as simple as deciding to not use the environments.

Parallel Test Execution Without Setup

The challenges associated with setting up environments for parallel execution can be easily overcome with cloud-based environments as they are built with frameworks to handle concurrency.

Report Across Environments with Zero Configurations

This is a gold mine. The reporting is built-in, so QA managers can pull metrics and see how tests are trending across operating systems, browsers or resolution settings – all in one place.

Combine Manual and Visual Tests with Automated Tests

Cloud-based test environments offer complementary benefits to automated testing which you won’t get through virtual machines, on premise systems or local devices.

For example, you will often realize that manual or visual testing is needed to uncover issues that are hard to find through automated test scripts. These include:

1. Establishing aspects of the UI are of the right color, shape or size
2. Uncovering overlapping elements
3. Ensuring the appearance or usability of a website looks as expected

The right cloud-based test environment management tool will have out-of-the-box capabilities to tie these into your test automation framework.

Nikhil Kaul is the Senior Product Marketing Manager of Testing Products at SmartBear Software. Prior to SmartBear, Nikhil served Digité, a leading provider of Application Lifecycle Management solutions, in various roles including Senior Software Executive. There, he gained insights into software development and testing market. Nikhil received a master’s degree in business administration from Georgetown University. He is very involved in discussions taking place in development and testing communities. Follow him on Twitter @kaulnikhil or read his blog at: http://blog.smartbear.com/author/nikhil-kaul