Agile Testing practices mapped to Continuous Delivery

In this post, I will  attempt to map the agile test practices to the continuous delivery practices. Some of these practices are related and if done effectively right from test strategy, it can greatly help in delivering valuable software seamlessly. I am sure readers of this post may agree that test code lives as longs as the code lives. So we must invest on  test code and the quality of it.

What is Continuous Delivery

Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time. It is used in software development to automate and improve the process of software delivery.

Picture Courtesy: Wikipedia

In Continuous Delivery, there is deployment pipeline that is formed to ensure quick and seamless feedback to the developers and testers about the quality of the check- in.  The tests are divided into various types like Unit Test, Functional Test, “ities” like Reliability/Security /Maintainability testing which is run in different pipelines depending on the time taken to execute the test cases. These test cases is run in various stages as the time taken for these test cases execution is less to high. It is important that the test cases and scripts are designed in such a way that these aspects can be done in a seamless way when the code and test code is checked in. These points requires a much deeper and strong test engineering practice understanding and adherence to the core agile principles of collaboration, self organization, technical excellence,  attention to working software .

The key test practices important for the effective continuous delivery practices are divided at Task, Story, Sprint and Release Level. There is definitely some overlap between each of the practice across the various levels. I will try to explain the details of each of the practices more from a test requirement for continuous delivery perspective.

Level	Test Practice	Details
Release	Test Strategy	Test Strategy is like an umbrella practice in which the team can think about how the test will be designed, organized, executed, reported, analyzed etc. Test strategy should cover the details about: -          Different type of test cases that will be written like Unit Test, Integration Test, Functional Test, “ities” Test . This is an important aspect as some thinking in the beginning will help in later test organization in an effective way. -          Automation Architecture -          Test Suite organization -          Development and Test Collaboration -          Etc.
	Automation Architecture	Automation architecture is the architecture of the overall Test automation – How the various components will be tested. All the principles of a System architecture is equally applicable for a test automation system. Sometimes Automation architecture is seen as some list of tools or framework only.
	Test Suite Organization	Test Suite organization is the practice in which it is decided how the test cases will be grouped and organized. Example All UT test cases can be grouped under UT Test Suite at each feature level, similarly the FT and the ‘ities’ test cases.
	Operation /Infrastructure Test	Operation /Infrastructure test are all the tests related to the environment, infrastructure etc. Many times we see that everything works fine till we have deployed the system in the production and once it is deployed to the production some functionality don’t work . This happens due to test not being thought from an operation or infrastructure angle. Practices like Test Driven Development for Infrastructure code can be considered.
	Testing for ‘ilities’	Testing for ‘ilities’ are test from performance, reliability, stress, volume, security , maintainability and related aspects.
	Hardening and ZFR	Hardening and Zero Feature Release is the release or time period for any system development where there is 1-2 release being planned and done with no new feature added. The test is intended to test the stability of the system with no new features added. This is also termed as hardening.
	Test Pyramid	Test Pyramid is a practice in which there is a pyramid formed with the unit test, functional test, ‘ities’ test in decreasing order. What it means is that overall test should comprise of 50-60% unit test , 20-30% Functional test and 10-20% ‘ities’ test.
	Deployment Pipeline Design	Deployment pipeline design is about thinking ahead about what kind of testing will be done in which pipeline and ensuring that all the infrastructure readiness like build scripting, servers etc is made available from this perspective.
Sprint	Development and Test Collaboration	Development and test team should brainstorm about the scenarios, implementation approach, test design, test organization and related aspects.  Many times team work in silos and lots of anti patterns gets generated due to this. One such anti pattern is ‘Dual Test Pyramid’. I will have a separate post for all the anti patterns.
	Test Pyramid implementation	Test Pyramid Implementation is about ensuring that at constant interval we check that the tests are being written based on our initial strategy and also fine tune the test strategy based on the implementation.
	Deployment Pipeline implementation	Deployment pipeline implementation is about orchestrating the right test suite at the right pipeline as per the test strategy and ensure that feedback is provided to the right stakeholder at right time
	Exploratory Testing	Irrespective of how much automation we do or how much test design we do, exploratory testing should always be done. It can help in bringing out many such scenarios which we may not have been thought about. Further these test cases can be automated and integrated to the system
Story	Story Level Test	Story level testing is testing done at each story level where each type test cases are written at story level
	No false alarms	Many times it is seen that if test fails we are not sure if the issue is with test code or the code. Such false alarms should be avoided and clean test code practices should be followed. It is a good idea to do test code refactoring if such symptoms are seen.
Task	Clean Test Code	Clean test code practices are similar to following coding guidelines and ensure that there are no bad smells in the test code. Test Code should be given equal importance as it lives as long as the code lives.
	Dev Testing	Development test should focus on the unit test and key functional test as per the test strategy. Here the Development and Test team should closely collaborate and ensure that whatever is written is organized appropriately and fits into the right deployment pipeline.
	Local /Private Build	Private Build is a practice in which developer runs the unit test and static check is done before the code is checked in to the configuration library. This is an important discipline to ensure that there is always test available as a safety net. It also ensures that each check-in does not break any build.
General	TIER	Testing is Everybody’s Responsibility. Both Developer and Testers should take equal responsibility of testing. In many organizations, the ratio of tester to developer ratio is 1:3. But the ratio of test code to code is almost 1:1. By basic common sense we can make out it is not possible to write all test case by test team only. All should take equal responsibility of testing and ensure the quality of the testing.

Let me add some some examples which are taken from the best practices of open source

Example 1: Developer Test Suite organization for hadoop-hdfs module

First level structure:

Second level structure inside the hdfs folder:

Next level structure inside the server / datanode folder:

Note:

1. This structure is by design and not by accident
2. It is a commonly used good practice to mimic your product code organization in your test code organization

Example 2: Why do we need developer testing if we already have system testing?

Have you heard about the “Zune bug”?

Zune was a portable digital music player from Microsoft. On December 31, 2008, when Zune users across world were trying to use it for their New Year eve’s party, it froze. The music players just hung and did not respond.

The issue was a simple code level defect which anyone with basic programming knowledge could have detected. The defect was in the following piece of code:

year = ORIGINYEAR; /* = 1980 */

while (days > 365)
{
   if (IsLeapYear(year))
   {
       if (days > 366)
       {
           days -= 366;
           year += 1;
       }
   }
   else
   {
       days -= 365;
       year += 1;
   }
}

As you could notice, the defect is in the missing else block of the “if(days > 366)” condition which will result in an infinite loop when “days = 366” (on Dec 31st, 2008).

This a defect which system tests may miss. However, a unit test with the input “days = 366” can catch it easily. Developer testing can ensure that the code we wrote is indeed the code we wanted to write.

Example 3: Clean Test code

We will use the example of a test case from hadoop-hdfs to observe some clean test code practices.

The first candidate for refactoring in test code is often the duplicated setup/configuration code. The snippet above shows two good practices to make this cleaner.

A test case which is longer than a few lines is difficult to understand and maintain. The code snippet above extracts out the lengthier logical code segment to an inner function.

Mocking can accelerate your test automation by handling dependencies more predictably and can help you create more maintainable tests as shown in the snippet above.

Good test practice implementation is an important factor for effective continuous delivery system.  In the next posts I will cover some of the anti patterns which I have observed in this.

Good test practice implementation is an important factor for effective continuous delivery system.  Let us given consider test code equally important as source code. As long as we don't do this, effective continuous delivery or deployment system may only remain a WISH.