Life of automated tests

Automated Test Need Maintenance

What stakeholders hate to hear is the news "Automated test will not work in this environment need maintenance". In the current industry trend many organizations will manage multiple versions of code going through different kind of changes or unique configurations settings. I am sure many of you in test automation area may be looking for one tool which supports sustenance of automated tests % while product is going through changes, 

When to start focusing on test automation?

Some industry veterans recommend to look at automation effort only when product is stable. But some people say automated tests always increase speed of going to market  so start anytime it is worth. Who is right? we are not sure. But challenge of sustaining automated test % in a given environment is always there for all of us. 

Multi stage of product 

A build developed goes through development environment, QA environment, Pre-production environment and production environment. On top of it if you have a key customer configuration environment where support traffic is tough to handle increases challenges for final deployment decision.

Development Environment

The dynamically changing code is on development environment. Here we always wish to see that smoke test suite is automated for all feature and it runs every day. This is also a stage where person responsible for writing code ensure it is robust. Maximum issues are found later are traced back to escape from this stage. I know it requires a revolutionary change for developer to arrest few issues when they write code (Look at STAG Software one of the offerings in this zone. Very nicely articulated on trend and challenges here). 

QA Environment

Next is QA environment where we push the build from development environment to QA as per project plan. Here we wish even basic interfaces are tested with automated tests and it should work. Many new features that released to QA may get tested with automated tests. What we group as regression tests also executed here. % automated regression tests is a vital key metrics organization to watch.

Pre-production Environment

Next to this is pre-production or staged environment where product is all most tested in a real production environment kind of situation. More of  smoke test suite is what we prefer though regression is welcome here. From my experience I witnessed many organization doing regression test suite running on QA environment itself. The test cases may vary between each stage so tool must support version number for test cases so that during execution depending on environment we can pick the right test case version and related test data to executed automated tests.

Production Environment

Final is a production environment. This is a short window of production downtime to deploy the new version and hand over to support team. Automated smoke test full or partial to be executed based on history of deployment related issues from the field. Many times the package of release will have some scripts to be executed to ensure protection of existing configuration status, db changes impact etc. This event is closely watched by many and rollback option must be available if risks are seen after deployment. Failure here is a painful issue to manage for all levels in the organization.

The configuration of key customers whose support issue traffic was painful to handle in previous release must be tried at pre-production environment automated test results must be available for final decision on these configurations in particular. With testing on cloud possible now we should focus on firing automated tests on different key configurations settings. Therefore automation tool should support cloud execution also.