Anti Patterns of Continuous Integration

In this post I would try to bring my perspective of what are the top reasons where Continuous Integration fails or in many cases the benefits are not seen immediately. These are some of the anti patterns of continuous integration. Many cases I have seen that team looses hope and feel that it is an overhead which Agile practices has brought in. One of the reason may be that the Team / Team member or the management is looking for a quick solution without deep thinking about how Continuous Integration practice can be done really well. Please note that patience and strong will to do is the only way to become successful:

Reason 1- Every team situation, competency , technology, domain and dynamics is different. Just copying the CI Design,Architecture or roadmap from some other team is a wrong idea. Every team should have their own vision about how do they want the CI to look like. What all they want to run in the Continuous Integration System. Some teams try to run all the tools of inspection, all test cases together and starting with real continuous integration immediately without the infrastructure in place. That may not be a good idea in my opinion. It is always better to have a step by step approach. We can start with periodic build, incremental build and slowly move into the continuous integration approach. But if the team really have the competency then they can start immediately. It is really required to do this analysis.

Reason 2- Assuming that all the team members are very good in test coding skills is a wrong idea. Many times it is found that the test suite organization skills are poor in team members and there is no systematic training. The test code writing skills are not good in some cases due to which the test cases does not work properly. This is one of the reason where CI does not work effectively and there are frequent build failures.

Reason 3- User stories are not granular enough. During the user stories documentation I have seen that the stories has not been written with an overall perspective of granularity so that the developers can really check-in the code frequently and also if the story card can be realized effectively in short period. This is not quite easy and I am sure many people may have different opinion about this. One of the reason which I have seen for this is that the Business Analyst does not write the story as vertical slices from the user perspective. It does not depict the end to end functionality in the story. The requirements are sometimes written in a modular way. Due to this it is very difficult for a coder to checkin at frequent intervals. One pattern is observed that this issue is actually not known and we expect Continuous Integration to work.

Reason 4- CI should be actually run in the clone of production environment. But in big products which are used by many customers there are many environments and the multi-combinational testing, it is required that the strategy of testing is analyzed deeply.
Following is a good example -
Try to answer this

Each user on a computer system has a password which is generally 6-8 characters long, where each character is an uppercase letter or digit. Each password must contain at least 1 digit.
How many password combination is possible.?
The answer is:
2,68,4,483,063,360 possible variations of password exists
1 Test Case Design Time = 1 minute ,60 Test Cases = 60 minutes
1 day= 480 Test Procedure , 155 years to create and execute a complete Test.

In many cases I have seen the team don't analyze the stategy and the CI is run in some combination which is actually not the right representation of the production environment. Teams can use some of the techniques like parwise testing / OAT(Orthogonal Array Testing) to analyze the right combinations. But it is important to use these techniques with good judgement about the test sufficiency and coverage. The Test Strategy is really important to understand the real customer environment, where the test suites should be run and this work should happen during the Infrastructure readiness time for CI,not during the Test Execution time.

Reason 5- The Test Frameworks does not ensure that the SUT system architecture is taken into consideration while designing the overall automation framework. This creates issues, where the frameworks are not scaleable and there are issues related to overall test execution later.
Test Teams must ensure that the Automation framework considers all the aspects of testing like the system under test architecture, test inputs, test outputs, SUT precondition programe state, SUT postcondition programme state, environmental inputs and outputs, component dependency, APIs , capturing the results, analysis of the results etc, the various monitoing tools. Since the focus of the article is continuous integration, I may not elaborate more about the Test Frameworks. Important point is that for running continuous integration systems effectively it is important to have robust test automation system and the automation framework should be scaleable and reliable.
I have found this article very useful and I would highly recommend this for further reading.
http://www.softwarequalitymethods.com/Papers/autoarch.pdf

Reason 6- I have seen sometimes that the team does not have good build scripting skills due to which there are issues. Sometimes the CI Tool chosen does not support administration console for build dependency management etc where it becomes need that there need to be good knowledge on build scripting.

Reason 7- Overall CI Architecture of the system is not very clear. Here CI Architecture does not mean Tool. It means how the different components in the overall system is organized and how it depend on each other and whether the Continuous Integration system is also organized in similiar manner. For big system it is very important that there should be multiple Component level CI System which further builds the overall System level CI. It is also seen that the build staging (Stages to ensure faster feedback in the respective CI System) is not very clear.
These causes issues where the CI takes longer time to execute and finally the team does not use the CI for actual feedback.

Reason 8- All the test cases and inspections tools are run together in one stage without any staging (/Build Pipeline) in place. It is advisable to have continous integration in stages so that feedback can be provided faster. Sometimes all the test cases including Unit Test Case, Component Level Test Cases, Integration Test Cases, Performance , Reliability Test Cases are run together in one system. You can refer to the previous post to know more detail about Staged Build:
http://thinkinginagile.blogspot.com/2011/06/continuous-integration-what-is-staged.html

Reason 8- Test Suite Organization Structure not followed appropriately- Sometimes it is seen that the Test Suites are not categorized properly as unit, functional, integration, non functional test cases due to which it cannot be staged and also it cannot give analysis or test results at feature level.

Reason 9- Local Build not being used effectively- There may be many reasons for local build not being used effectively. Lack of awareness, process or infrastructure issues may be the reason. You may refer to the following post to know the anti patterns of Local Build:
http://thinkinginagile.blogspot.com/2011/07/anti-patterns-of-local-build-private.html

Reason 10- Lack of readiness for the tools and infrastructure- The team must ensure that before the actual execution of the project, the team should have all the infrastructure ready so that the time can be really productive during the actual work and focus on the functional work rather than infrastructure readiness. Sometimes I have seen team is not ready with the tools that need to be integrated with CI system, CI tool may not have been finalized , CI Server PC may be not of good configuration or slow and other related issues. These kind of issues diverts the attention of the team from the main work to tool related aspects which needs to be corrected.

Reason 11- Infrequent Check-ins- Team Members checks in the code weekly once or sometimes two week once. This defeats the purpose of continuous build and integration. There may be many reasons for this. It may be story card granularity issue, discipline issue or any other.

Reason 12- Broken builds are not given enough attention and the other team members keep checking- in the code without fixing the build.

Anti Patterns of Local Build / Private Build

Local Build is a very important practice in Agile which helps in getting sucessful build more frequently. To get more details about what is Local Build you may refer to the previous post:
http://thinkinginagile.blogspot.com/2011/07/private-build-local-build.html
However due to lack of awareness, process or discipline and due to other infrastucture issues there are some anti patterns to Local Build which is furnished below:

• Running all the test cases in the Local Build- Developer runs the functional / integration test cases (typically that takes long time) as part of the local build.


• Running all kinds of Inspection tools in Local Build- Developer runs all the inspection tools like Static Code Analyzer, Test Static Code Analyzer, Simian for Duplicate Code analysis, Useless code detector, Code Complexity Measurement tool etc. ( Just running the static code for the source code is enough. For the detailed feedback anyway the mainline build is there. However if there is a trend seen where the check-in often breaks due to issues from other tools, then the team should focus on competency improvement and ensure that all members are trained well to write good code and test code. It is not advisable to add so many inspection tools to the Local Build as it consumes more time.)


• Unit Test Cases ,in reality is not Unit level cases- Test cases are categorized as unit test cases though it is not unit test cases. The key aspect of unit tests is having no dependency on outside entities like databases, which increases the time it takes to setup and run tests.


• Test Suite Organization Structure not followed well- Sometimes it is seen that the way the test cases are organized , it is difficult to identify the unit test cases separately. When deeply analyzed, it is found that the team does not have any protocol to follow naming convention to identify the test suite and test cases for the test type.


• Not using the same directory structure as configuration Library- Not using the same directory structure as the configuration library due to which the environment and other aspects are not same. The typical directory structure for source code and test suite can be :

• IDE does not support a one click Local Build - Some teams don't use good IDE that can support automated check-in from the IDE and taking the latest code from the configuration library.(Especially in C projects)



• Local build takes more than 3-5 minutes for providing result- I have seen sometimes Local / Private Build takes around 30 minutes to run and the developer has to wait to get the feedback and he /she sits idle. It creates productivity dip and the motivation level of developer is less to run local build every time.



• Developer PC is not of good configuration- Developer PC configuration is not good due to which Local Build takes more time.



• Members taking short cuts- Sometimes the team members in hurry takes short cuts. If this is not addressed, it becomes a culture and later correcting this becomes an issue. Running Local Build is a healthy practice and it is more of a culture.

Lean Principles and Agile-Aren't they same

The Five key principles of Lean are identified as Value, Value Stream, Flow, Pull, Perfection. Agile principles depicts the key aspects of giving highest priority to customer, better collaboration at all levels, working software as primary measure and other aspects. Details mentioned in previous post:
http://thinkinginagile.blogspot.com/2011/03/agile-manifesto.html.
Aren't Lean and Agile two names for the same thing?
Well, Lean and Agile share the same goals. The practices that make up the various agile methodologies also support the lean principles. In fact Lean takes a wider view of the entire business context in which the software development is done. Lean considers Agile software development as supporting practices for Lean Software development.

Value
Value is defined by the customer. We must understand what is really considered as value from the customer's perspective.
Value Stream
Once the value is identified, we need to create a series of steps or processes to produce the product. This is also called the Value Stream. Each Step is either categorized as value-added , non value added but necessary, or non value added waste (which can be actually removed)
Flow
The development process must be designed in such a way that the step of activities should flow continuously. If the value chain stops at any point then it is evident that some waste is getting produced.
Pull
Let customer pull the value. It means that we don't start developing until it is really required. Decide as late as possible to develop anything.
Perfection
Continuously identify areas were waste can be removed and the step of activities can be made perfect.

References
The Art of Lean Software Development - Curt Hibbs, Steve Jewett & Mike Sullivan

Continuous Integration Build Metrics

Continuous Integration Build Metrics provide insight into the Build Optimization aspects, Health of the Source Code, Health of the Test Code, System Size. These indicators can greatly help in taking certain corrective and preventive action based on the continuous integration build results. Below are some of the key metrics indicators that can be obtained from CI -

1) Number of Source Lines of Code (SLOC) - This indicates the total software or system size in LOC. (Unit of Measurement - LOC)

2) Number of Test Code Lines of Code -This indicates the total test code size in LOC. (Unit of Measurement - LOC)

3) Compilation Time -Total Compilation time taken by the build. (Unit of Measurement - Hours / Minutes/ Seconds)

4) Total Test Execution -Time Total time taken to execute the Unit Test, Functional Test, Non Functional Test. (Unit of Measurement - Hours / Minutes/ Seconds )

5) Total Inspection -Time Total Time taken to execute the inspection tools. (Unit of Measurement - Hours / Minutes/ Seconds )

6) Total Deployment -Time Total Time taken to deploy the product / software into the target environment from the continuous integration machine. (Unit of Measurement - Hours / Minutes/ Seconds )

7) Total Build Time --Indicates the total time taken to run all the builds (including inspection , functional, non functional build) . (Unit of Measurement - Hours / Minutes/ Seconds )

8)Successful Build Rate -It is the division of the total number of successful build by the total builds at a give time interval. (Unit of Measurement - Percentage (%) )

9) Build Repair Rate -It indicates the time taken to repair a failed build. (Unit of Measurement - Hours / Minutes/ Seconds)

10) Version Control System Load Time -Indicates the time taken to check out/ update the project from the continuous integration build. It gives pointers about the network bandwidth, processor, memory, disk drive sufficiency and the peak time load of the version control system. (Unit of Measurement - Hours / Minutes/ Seconds )

11) Number of Errors open in the Source Code -Total Numbers of Static Tool Errors open in the test code of the Latest Build . The number of rules can be configured in the rules file of the tool. (Example- PC Lint, Checkstyle, PMD, Findbugs etc). (Unit of Measurement - Number )

12) Number of Errors open in the Test Code-Total Numbers of Static Tool Errors open in the source code of the Latest Build . The number of rules can be configured in the rules file of the tool. (Example- PC Lint, Checkstyle, PMD, Findbugs etc). (Unit of Measurement - Number )

13) Unit Testing Line Coverage -Indicates the total lines of source code covered through the unit testing. (Unit of Measurement - Percentage (%))

14) Unit Testing Function Coverage -Indicates whether the function (or subroutine) in the program been called through the unit testing. (Unit of Measurement - Percentage (%) )

15) Unit Testing Branch Coverage -Indicates the branch coverage through unit testing. (Unit of Measurement - Percentage (%) )

16) Functional Testing Line Coverage -Indicates the total lines of source code covered through the functional testing.(Unit of Measurement - Percentage (%))

17) Functional Testing Function Coverage -Indicates whether the function (or subroutine) in the program been called through the functional testing. (Unit of Measurement - Percentage (%))

18) Functional Testing Branch Coverage -Indicates the branch coverage through functional testing. (Unit of Measurement - Percentage (%) )

19) % of Duplicate Code Duplicate Code -Threshold can be set for block of code and Tools like Simian checks the duplication in percentage for the block of code set as threshold. It Ignores whitespace, curly braces, comments, imports, includes, package declarations, etc. (Unit of Measurement - Percentage (%)" )

20) Average Code Complexity -Indicates the average code complexity value of all the functions. (Unit of Measurement - Number)

21) Max Code Complexity -Indicates the maximum code complexity value among all the functions. (Unit of Measurement - Number )

Continuous Integration Build Duration or Time

The important point while running the CI build is to provide feedback as soon as possible. A good rule of thumb is to keep the integration build no more than 10 minutes. The next question would be - what all test cases and inspections tools should we run in the integration build. Well, the 10 minute build need not run all type of test and inspection tools. The key aspect is whatever test cases that could be run as soon as possible and that gives maximum confidence about the system can be run. For Example the Overall Build be separated into stages to get faster feedback. In the Stage 1 only the Unit Test Cases and the Smoke test cases can be run , In the Stage 2 the Functional test cases can be run and in the Stage 3 the non functional test cases can be run. The build is considered PASS from end to end perspective only when Stage 1, Stage 2 and Stage 3 is Pass.Some Rule of Thumb for reference:

Local Build / Private Build < 3 minutes
Stage 1 (Integration / Primary build )< 10 minutes
Stage 2 (Functional / Secondary Buid) <2-3 hours
Stage 3 (Non Functional Build ) - Depending about the type of test cases.
Inspection Build (Can be run in any stages depending on the time taken for each tool)

Typically the Stage 3 build are long running test cases for reliability, performance which has many combinations and it can also take 1 day - weeks depending on the type of test cases. For daily operations for development purpose, the Build till Stage 2 can be considered.

Private Build/ Local Build

Local Build comprises of process of compilation of the added source code and test code, running of the unit cases, running of the inspection tool especially the Static Code analyzer (Like PMD, Checkstyle, PCLint etc). We can also run functional test cases and more inspection tools like duplicate code detection tool like simian etc., but this may increase the time of feedback for the developer. So it is advisable to keep only the key aspects (compilation, unit testing, static code check) as part of private / local build.
Local Build process can be described in following steps :
Step 1- Developer finishes the task (coding, unit testing, integration and functional test)
Step 2: - Developer takes the latest code from the configuration management library which comprises of the latest code changes.
Step 3- Runs the Local / Private Build with the latest code.
Step 4- If the build passes, then commit is done to the configuration library. If the build fails then rework is done and step 1 onwards is repeated.

Below antscript shows how to run a Local or Private Build using Ant. This is just for reference. Actual Build Script need to be changed based on the project situation.

The Thumb Rule is that Local Build should be as fast as possible. It can be less than 3 minutes. The practice of Local Build can ensure that whenever the developer checks-in the code, he / she does not break the main-line build and always completes the work with good quality. Some of the commonly occuring anti patterns, which should be avoided while following the local build practice are as follows:

• Developer runs the functional / integration test cases (typically that takes long time) as part of the local build.


• Wrong understanding of Unit Testing. Test cases are categorized as unit test cases though it is not unit test cases. The key aspect of unit tests is having no dependency on outside aspects like databases, which increases the time it takes to setup and run tests.


• Some teams don't use good IDE that can support automated check-in from the IDE and taking the latest code from the configuration library.(Especially in C projects)


• Sometimes there is tendency to run all kind of test cases and also the inspection tools in the private build. This increases the execution time for local build.


• Not using the same directory structure as the configuration library due to which the environment and other aspects are not same. The typical directory structure for source code and test suite can be :
  
  
  This is also is one of the reason where the private build passes but the mainline build still fails.


• Developer Machine is not of good configuration and it takes long time to execute the local build.
  


• Taking Short-cuts. Lack of Discipline.

There is obviously no debate that Local Build Practice can bring lot of benefits to the team, if used effectively. It also compliments to the continuous integration practices. This practice can really ensure that the team is efficient and can deliver good quality with certainty.

Exploratory Testing in Agile Teams

Sometimes the Test Team has some doubts that when all the test cases are automated and when at each story level there is testing happening, "do we really need to exploratory testing?". "Is exploratory testing applicable in incremental and iterative testing".

The answer is "Yes".

Irrespective of how much automation we can bring in and how much planned testing we do, Exploratory testing has its own place and it must be done to ensure that there is no residual defects in the system. Exploratory Testing can be well planned. During the Iteration the Developer and the Tester and sit together and conduct the exploratory testing. Automation can be used to do test setup, data generation, repetitive tasks. In the exploratory testing , each tester / developer can use his / her own skills and knowledge of the system to find the inside defects of the system. Some of the tips for doing the exploratory testing systematically is mentioned below:

• Understand the critical features that the customer is going to use. Give focus to those modules first.
• As a Tester /Developer, the time for the overall exploratory testing is limited. Should have a test strategy in place comprising of the focused testing approach for the critical features, troubled features (explained in the end), how to generate the test data, Techniques usage (if any)- like equivalance partitioning, orthogonal array testing techniques, Boundary value analysis, error guessing mthod etc. This strategy can help you to use the limited time very effectively.
• Think about what the normal user would do and do those steps. Have an attitude of breaking the system and be sure that there are deep defects which are not unearthed and this is your chance to do it.
• Use some random approach to do the testing.
• Think about what the expert and a normal user would do.
• Look into the troubled features during the normal testing (means during the normal testing, there may be features on which more defects may have come). Target those features and conduct some exploratory testing.

Exploratory testing may provide good insight into any of the defects which were not found during the normal testing. Always plan for exploratory testing in all the agile projects irrespective of the time availability. Plan for exploratory testing as an activity during the release and iteration planning time. It always gives good results and gives more confidence about the system.

References-

Continuous Integration - What is Staged Build / Staging of Build

The main purpose of Staging of the Build is to get faster feedback from the Continuous Integration System. A programmer cannot wait for more than 2 hours to get feedback about the check-in. So the Build is distributed in various stages to ensure faster and step by step feedback. This is applicable for typical teams maintaining (or going to maintain) larger code and test code base.  Following are the various Stages.

• Once the programmer checks-in the code, the Inspection and the Commit Build can be parallely triggered. In the Inspection Build, the tools related to Code Complexity of Code and Test Code, Static Tools and other Inspection Tools like Design Validation , Architecture Validation etc. can be run. Commit Build should give feedback in less than 10 minutes. If teams can get feedback in less than 5 minutes then it is great result. The important point is- teams should try all possible means to get faster feedback from the commit build.

• If the Commit Build is Pass, then the Stage 2- Secondary Build is triggered. In this Build , the Longer running Integration, Component and System Level Test Cases are run. Typically the thumb rule is to get the result from the secondary build in less than 2-3 hours. To get this result we can run the Build in a Cluster Setup as shown in the Figure above.The inspection build may take longer time based on  the different tools being run. Teams can decide to fail the build based on analysis of the report.s For example if the Static Code errors are introduced in the new build then immediately the secondary build can be stopped.

• Once the Secondary Build is Pass, the Non Functional Test Build can be triggered. This comprises of Reliability, Availability, Volume, Stress, Performance Related Test Cases. These test cases takes much longer time and it depends upon the kind of System under Test and the related Test Cases that are being executed.

• Further Stages can be added depending upon the need and also based on the kind of system.

References:

http://martinfowler.com/articles/continuousIntegration.html

     

References for Continuous Integration Tools

In this post, I have tried to bring the various Continuous Integration tools which are available in the industry. Also, below is the link in which all the detailed features of continuous integration tool has been compared. This is a good reference to teams which are planning to choose a CI tool for the development purpose. http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix

An important point while selecting the CI tool is to check if the CI tool supports all the tools related to Continuous Inspection (Static Code related tools, Code Coverage, Design Validation , Architecture validation etc), Continuous Testing ( Functional, Memory leak , volume, stress related test tools), support for the various plugins. This is very important as team can really focus on the core development work rather than focusing on integration of tools to CI which may not be a priority work at the time of core development. With the right tool selection, teams can really improve the productivity.

List of Tools which are available

Bamboo
http://www.atlassian.com/software/bamboo/

CruiseControl
http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET

LuntBuild
http://luntbuild.javaforge.com/

ParaBuild
http://www.viewtier.com/products/parabuild/index.htm

Sin
http://sin.tigris.org/

TeamCity
http://www.jetbrains.com/teamcity/

Tinderbox
https://developer.mozilla.org/en/Tinderbox

Pulse
http://zutubi.com/products/pulse/

Bitten
http://bitten.edgewall.org/

BuildBot
http://trac.buildbot.net/

Gump
http://gump.apace.com/

BuildForge
http://www-01.ibm.com/software/awdtools/buildforge/enterprise/

References

Agile Story Card Writing - Some Facts

"How to write User Stories in Agile Projects"- I am very sure that this question would have a very different answer from different business analyst, product owners, technical community. Story Card should should follow the INVEST characterstics [ I- Independent, N- Negotiable, V- Valuable, E- Estimatable, S- Small, T- Testable]
You may refer to the following article by Bill Wake where he describes about the details of good Story
http://xp123.com/articles/invest-in-good-stories-and-smart-tasks/

In this post I would give a different perspective of the issues teams faces while writing user stories. Writing User Stories from the Customer's perspective and the usage by Developers is relatively easy in Application Development based projects. But in case of embedded , operating system domain teams face issues of vertically slicing the story card and usage by the development and test team.
In my opinion there are following kind of issues. For each kind of issues, I have tried to provide some solutions:

Issue 1- Lack of enough competency where the Business Analyst can really slice the stories effectively cutting across the multi layer (presentation layer, logic layer, persistance layer etc). In some teams the technical members play the role of Business Analyst. And there is no training given to these members. Sometimes it is default assumed that all the members who are technically very good can write good user stories. There is less authentic references available in industry about writing user stories in each domain. User story writing is specific to the domain. It is important that good training be provided to the members involved in writing user story.

Issue 2- There may be some stories that need some background preparation as Spike Stories so that the solution is ready. These strories can also be said as Infrastructure readiness stories. Sometimes these stories are not ready and team members get the stories which are vertically sliced across the layers. In this case the team faces issues related to the technical solution. The solution to such kind of issue is that anytime before the actual development starts, the Business Analyst or the Technical member need to check if the Spike Stories need to be played before the actual story execution. Accordingly, the actions should be taken.

Issue 3- Team members are very junior or dont have enough expertise. In such case stories may need to contain more details and in some cases, solution artefact. There may be also cases where at implementation level, the key algorithm / solution for the similiar implementation can be generalized. Such cases the Designer and the Technical Person need to provide some technical solution so that the implementation is effectively done. This issue may not be there in teams where there are senior members working or where the team size is small.


The important point is that there is no one stop solution for writing story cards. The requirement documentation process needs to be customized based on the situation and domain. If teams face issues with vertical story slicing then team can taken minimum marketable feature based approach and implement the customer requirements. What best fits to the context need to be taken. Story Card has many advantages: if the stories are vertically sliced, it can be developed faster, it gives an end to end value to the customer, customer can give feedback easily, it also helps in other agile practices like continous integration where, faster feedback can be given. However if teams have there own constraints like complexity of the domain (due to which slicing is very difficult), lack of enough competency etc., team can use Minimum Marketable Feature (MMF) approach. The team that use the feature based approach is also as much as Agile as teams using the story card based approach.

References:
http://xp123.com/articles/invest-in-good-stories-and-smart-tasks/
 

How to Plan in Agile

Planning is an essential activity for anything and when it comes to project execution, a good plan gives right direction for the execution of the work. Some questions that comes to me when teams follow agile approach:
" How much planning to do", "Should we do estimation in agile", "Should we assign work to the team member level", "Who all should get involved in the planning" etc.
In my opinion the focus in Agile team should be more towards value driven approach than a plan oriented approach. The focus and direction should be towards iterative and incremental development.

Following are some suggestions for doing overall planning in agile team. In this context I am considering a slightly bigger project (3000 person month). For smaller projects the context and the related activity can be decided accordingly. These need to be considered as only suggestions and not a prescription. During my Consultancy to various teams I have come across such similar questions, so this post. In this I would be only covering the steps and key activities.

Step 1- Project Charter and Vision : The overall vision and objective of the project is decided. Some of the activities which are conducted as part of this are as follows:
• The key features required for the customer is decided. Here the features may be at a very high level. It may not be at an implementation level.
• Some overall level estimation is done. Here the question comes that which estimation techniques can be used. The best is to use Team Velocity based approach. But I have seen that some team faces issues in deciding the team velocity due to reasons like either no historical information available, different team members, not able to identify the ideal story etc. It is even okay to estimate based on effort, lines of code, function point etc. The important point here is to be pragmatic and using a collaborative approach for estimation.
• Based on the work estimation the release dates are decided. In a typical project there can be more than 1 release. The Release Milestones are decided as part of this step.

Step 2- Release Plan : The next step is to have the Release plan in place. A Release comprises of more than 1 Iteration. To come up with the Release Plan following are some of the important activities done:
• Release planning meeting is done where all the team members attend. Before the meeting the Business Analyst / Requirement Analyst provides overview of the User Stories that are going to be taken up. The attempt should be to ensure that all the members are at the same page for doing the feature estimation.
• The overall estimation is being done keeping following key activities in mind : requirement analysis, architecture design, design analysis (if any), coding, test coding, infrastructure preparation, automation infrastructure readiness, training , testing, customer level test etc.
• The Project manager or Scrum master should give enough importance for team training, readiness of the automation, continuous integration, and competency of team. Without good preparation in this direction it may be very difficult during actual project execution.
• Sometimes the project team needs to give commitment to the client about the release dates after the release planning meeting. There may be cases where some of the user stories are not clear. How to give commitment is a challenge which most of the teams faces. While there is no right answer to this, some suggestions - we can plan for some buffer based on our experience and past history. We can make the client intimated about buffer and we can deliver the project early if we are able to do so. Please note there is a difference between Buffer and Padding. Padding (giving some additional effort for each user story level without any basis and not making it known) is wrong and it should not be done. It is like Gold Plating. Buffer is planned to ensure that any risks can be handled due to wrong customer priority, not clear story / technical readiness etc and it is intimated to the customer in advance.
• Based on the amount of work planned, the dates of Iterations are planned. The thumb rule is to keep the iteration fixed for 2 weeks and put the prioritized user stories in each of the iterations.
• The prioritization of the stories are done based on the following (in the order of priority)- customer priority, technical readiness, internal project dynamics and others.
• SCRUM is a very good management framework that can be used for the team structure and following scrum and scrum of scrum teams. This gives very good way of the project management and governance.
• Release Retrospective shall be conducted at the end of each release.

Step 3- Iteration Plan and tracking : The next step is to have the Iteration plan in place. Iteration comprises of stories that are executed and each story has a list of key tasks that are performed to realize the user story. Following are some key aspects:
• Iteration Planning Meeting is conducted where there is an overview given by the business analyst about the user stories being taken up.
• Planning Poker Method can be used for estimation. Refer to book by Agile Estimating and Planning by Mike Cohn.
• If team is not comfortable with velocity based approach then either effort or LOC based approach can be used. It is always good to have a parallel usage of Velocity for the team transitioning into agile way of estimating and over a period we can make our estimation better by using only Velocity based approach.
• User Story level estimation is being done keeping following aspects in mind- story analysis, story design, coding, test coding, TDD approach, local build , continuous integration, testing, release etc.
• Ensuring the “Done “ criteria at each story level is very important. Teams can decide the common process or criteria how a story can be considered as “Done”
• Daily SCRUM meetings should be conducted for ensuring that all team members are in sync with each other’s work and there is a collective ownership.
• At the end of each Iteration Review / Sprint Review should be conducted and the feedback should be included to make the product better. It is always better that the customer or the customer representative is available during the review.
• Sprint retrospective is conducted at the end of each iteration.

Agile Project Management and the planning should be value based and the focus should be to remove the non valued added activities continuously and focus on the value that can be delivered to the customer ,always.

References:

Our Customer don't sit with us, Can we be Agile..

Have you heard these questions during the project execution:

• My customer don't sit with me, Can I go agile?
• What is the use of doing agile when customer cannot give feedback?
• I am doing daily standup everyday, Can I be called Agile?
• My team does Daily build , Is it Continuous Integration?, Is it Agile?
• We are doing Iteration but it is not timeboxed. But we are Agile.

I have given just a few questions that we may come across during our interactions with the team. Sometimes it looks like the focus is to be called agile by doing certain practices. Sometimes the team and the middle management dont want to change to any new process and puts across points to prove that Agile is not suitable given the dynamics.

It is loud and clear- Agile is something which we cannot do, we need to be Agile. Agilility can be practiced in every team, irrespective of whatever the constraints the team faces.

Teams must note that a collection of SCRUM , XP, Lean Software Development, Crystal, Kanban etc and the usage of these practices does not mean that a team is Agile. Agility comes from the team and individual who are working on the project. Agility is all about being sensitive and providing highest focus to customer, fast , pragmatic , practical , self organized, giving continuous attention to technical excellence, giving importance to the working software, being simple, having a constant rhythm and sustainable pace, having a motivated team, welcoming change, having mechanism of stopping and correcting. These are not some silver bullet points. These are the principles behind being Agile. Teams must understand that it is not to prove to somebody we need to go Agile. Similiarly it is not to show what environment is not available due to which Agile cannot be done. In my opinion these are some of the anti patterns which we as team members should watch for. If we see any such symptoms then we should immediatly stop and do some retrospection about the reason for being Agile.

Now answer to my question "Our Customer don't sit with us, Can we be Agile..". Well the answer is "Of course, you can be". Agililty is all about following the principles. Even if customer dont sit with us, we can get iterative feedback through the 2 weeks sprint from our test team. It can also give feedback about the way the user stories are written from the development team. It can give feedback about the architecture. And so many other feedbacks. Of course, If the customer sits with an Agile Team, it has lot of advantages. The feedback can be much faster. If there is an option available, having the customer with the project team will really help in being agile and successful. But looking into the diverse projects and situation in which different projects are executed, sometimes the customer is not available directly. In this case we can be Agile by following the iterative and incremental delivery model & related practices and improve the delivery every iteration.

References:
a) The Art of Agile Development- James Shore and Shane Warden
b) The Software Project Manager's Bridge to Agility- Michele Sliger and Stacia Broderick

Build Optimization for effective CI

In this post I would like to bring some of my observation and perspective of Build Optimization to have a very effective and good Continuous Integration System.

Build Script is all about automating the tasks required for the system development.
I have noticed that Build Scripts are not given enough priority in teams as compared to production code. In fact on a lighter note, it is seen that in the context of priority the order of priority is on these lines , production code first, test code second and build script may be last. "-:)
Some of the most commonly used build script language is Ant, Maven, Make.

The way the build is organized should depend on the following factors:
a) Module Organization
b) Where (Which Operating System ) and how it should compile, test
c) Packaging Structure

Before the build script is written the member should understand the way the system is organized, how the modules are dependent to each other. This would help in really organizing the system appropriately.

Due care should be taken to ensure that there are no unnecessary steps added without any purpose. Sometimes I have seen that some build scripts perform tasks like zipping and sending the files to some other server, unzipping & compile and bring back the file to the root server and then do certain operations. When it is closely looked it was found that these operations were really not required. These tasks add to the time taken for the building the system. This is also one of the reason why Continuous Integration System in some teams takes more time to provide the overall results.

The important point is the Build Scripts should be reviewed carefully and members writing the scripts should be well trained on the system. If it is given less attention, later it effects the complete team's performance as time taken for completing the whole build through Continuous Integration System (includes compilte, inspection , test ) is very high. And later refactoring the build scripts is a mammoth task.

Reference
1)http://en.wikipedia.org/wiki/Make_(software)

Antipatterns of Daily Standup Meeting

In this post I would list down some of the anti patterns of Daily Standup which I have noticed in various teams:

• Daily Standups continuing for more than 20 minute- Some Daily standup meeting continues very long. Some discussion goes for long and even the problem solving happens during the standup. The typical thumb rule can be that if the daily standup is taking more than 10 minutes then it can be abruptly closed. The abrubt closure is an indication to the team that they need to improve. The very reason it is a standup is because if members stands for more than 10 minutes the leg starts paining. :-) One of the way by which we can solve this issue is by logically separating the team based on the kind of work, provide trainings to the team members about the objective of the standup.


• Daily Standups being used as a project status review meeting- Some daily standups are used as project status review meeting where the project managers takes feedbacks from the team members about the status of progress of work.


• Members reporting to leader about the project- Leader centric - Daily standup being a reporting to the leader is also seen as common issue. Leader or the Scrummaster should take some actions like making the team conscious that standup should not be a reporting to the leader. Also the leader can show body language like removing the eye contact with the member who is only reporting the update to the leader.


• Not good body language of the members- One kind of issue is seen where the body language of team members shows that they are not very much interested towards the updates of others. They somehow want to give their update and go back to work. Agile is all about working collaboration and having a team which collectively owns the work and code. Scrummaster should be cautious of such issues.


• Daily Standup meeting is not attended by all the members- Sometimes all members of the team don't attend the standup . It is sometimes seen that on any given day any one / some person is always missing from the standup.


• Daily Standup meeting time and place enforced to team- It is imporant that Daily Standup time, location is decided by the team and team should collaborative works towards this. The standup time and place should not be enforced to the team.

The above are few of the anti patterns which is observed. It is important that Agile Team overcomes these issues and ensure that the Daily Standup is conducted with the real objective of having collaboration and better communication between the team members.

References

• Agile Adoption Patterns-By Elssamadisy Amr


• Agile project management with Scrum- Ken Schwaber

How much process in Agile

The question of “How much process in Agile” is very difficult to answer. In fact while coaching various agile teams, I get these questions very frequently and these questions are being asked by all the roles of the team. Every time I give the same diplomatic answer “It depends” . In this post I would try to elaborate my perspective of “It depends”. The number of process steps in the agile mode of software development depends on the competency of team members (team experience and skills), maturity of the team, the business domain and the criticality, the management approach, trust of the team, team motivation and other similar factors.

• Competency of team members and maturity of team members How much requirement elicitation and requirement documentation to do, how much design modeling and how much detailed design document to be prepared and many other aspects depends on how much experience the team has and the skills the team members possess. In some teams where the team is very competent, just writing the user stories from the user perspective is enough for the further coding and testing and for the final release. Whereas in some other team where the team competency is not that high it needs further detailing of the user’s perspective and it needs to further have the design details until the class level details (in oo based approach). In a team which has very good skills on the design, there may be no need to detail down the Architecture further. Just having a brainstorming on the overall design modeling is enough. Team can further come up with the solution by themselves.But if the team lacks in skills on design patterns, refactoring skills to make design better then there is high probability that the design which is getting developed lack the overall perspective. The amount of documents and the amount of process depends on the competency of the team. So team should spend time in improving the competency.
• Business Domain and the Criticality  The amount of review, coverage and the code inspection to be done in a mission critical system is quite different from an application system. If we need to achieve zero defects quality, then it needs high level of attention. Well, as mention in the first point, it has a very high dependency on the competency of the members which the team has. I see a pattern in various teams where there is a kind of acceptance for the level of quality, also termed as , AQL (Acceptable level of quality). For a mission critical system , every 1 defects is very costly and it can even pose danger to lives of people. So zero defects is not a choice but the requirement in this context. Another example is an online shopping system where a minor defect may not cause that much harm as the former. The level of process stems also changes based on this. Also it highly depends on the AQL.
• Trust in the Team Trust among the team members is an important factor for deciding the process steps. Some examples- Team conducts manual logical code reviews, whether records for the code reviews should be maintained. It depends on the team trust and how responsible the team is in ensuring that the team agreement is followed with the right objective. Every step in the process is added to ensure repeatability. If the team members can ensure without the records then it is good. Processes in team becomes less heavy if every team member trust on each other and completes the activities as agreed upon.
• Some mandatory practices that is recommended for the Agile Teams Simple Design, Automated Testing, Automated Inspection, Continuous Integration , Collective Code Ownership, Iterative Releases, Customer Demo are some of the practices which Agile Teams should follow. In my opinion team also should not do very detailed design if it is not required. Each team to use common sense based approach to decide this rather than written rule. If rules are there it is good but there need to be enough flexibility to change the rules.
• Not a right idea to remove the existing process, governance completely It is recommended that teams change the existing process which is working well and take a step by step approach towards adopting agile. Governance is something which is here to stay irrespective which process model we follow. Governance helps in the management visibility and attention. It is good to have governance as it ensures that there is enough mechanism in an organization to ensure that all projects deliver successfully. Agility is all about people and the amount of process to be followed also depends on the people who are doing it. Teams should use the common sense based approach based on the team dynamics, business situation and domain for defining the process. Nobody can give any silver bullet solution for this. Only best practices can be referred.
• References

Agile Software Development - The Cooperative Game - Alistair Cockborn ( Alistair Cockburn and Jim Higgsmith Series Editors

Agile Iterations Test Challenges - Test Automation

In this series “Agile Iterations and Test Challenges”, I would try to bring my perspective of some of the test challenges with respect to Automation during the Iterative releases.

Unlike in the V Model and the waterfall model where the testing time is dedicated 2-3 months for a typical was 50-100 kloc projects, in Agile Iterative life cycle this is split into each of the iterations. Each Iteration is tested by the test professionals and is certified about the quality of the release. This work is done in collaboration with the development team. During the iterations the test team participates actively in the brainstorming and provides active contribution towards the various customer scenarios from the test perspective. I have seen that in most of the organization the test staff is relatively less staffed. The typical ratio of Tester to developer is around 1:3 or 1:4 or may be less. The tester tests at each of the user story level. Sometimes the pace at which the developer completes the story is much faster than the tester can complete the story testing since the number of tester is less and also due to poor infrastructure. The developer testing is mostly at unit testing level. For the functional testing mostly the team relies on the testers. Some of the practices of automation like have a scalable automation framework, good test suite organization, test naming convention; test logging etc. is slightly weak from the test perspective. The Iteration level automation is weak and sometimes not complete. Since the Iterative releases are being given at every 2-3 weeks once and the testing also happens parallel. Due to lack of good automation, I have seen test sometimes is not able to cope up with the pace of the releases. Due to this the quality of the software sometimes suffers.

• Test Automation Framework Scalability- Refer to my earlier post “Top Reasons why Continuous Integration practice fail or result not seen “where I have given some aspects that should be considered for the test automation framework. It is observed that the Test framework either focuses on either the tool part or the scripting part. The Test Frameworks does not ensure that the SUT system architecture is taken into consideration while designing the overall automation framework. This creates issues, where the frameworks are not scalable and there are issues related to overall test execution later. Test Teams must ensure that the Automation framework considers all the aspects of testing like the system under test architecture, test inputs, test outputs, SUT precondition program state, SUT post condition program state, environmental inputs and outputs, component dependency, APIs , capturing the results, analysis of the results etc, the various monitoring tools. In the following sample system the automation framework should be divided into each of the respective areas like GUI, APIs.

• Test Suite Organization, Naming Convention - It is seen that the test suites are not organized well based on the features and the functionality. The unit testing, functional suites, performance, reliability, volume testing suites should be rightly organized at each of the feature level. Sometimes the test cases are run as monolithic systems where all the test cases are run at one go with no organizations. There is lack of appropriate naming convention due to which it is not clear which functionality is failing.
  
  Some of these issues is due to lack of enough competency & awareness and others due to readiness for the iterative testing. It is important as agile teams to plug these gaps.

References

1. Agile Testing- A Practical Guide for Testers and Agile Teams- Lisa Crispin, Janet Gregory

Agility is all about people

"If you put fences around people, you get sheep"- William McKnight, President and Chairman (1949-66), 3M.

"What is Agile?", " How can we do Agile?", "Are you Agile?", "Measure whether the team is Agile", "What practices we should follow to be become really agile", "Is this team 100% Agile or 50% Agile"..
These are few questions which are frequently asked by various stakeholders of the organizations /(s) with which I have been associated with. I am sure you also would have heard such kind of questions in your day to day execution of project..
Well, Let me put my perspective of Agility..

Agile means quick , active, lively in literal meaning. Agility is the pace at which we respond to the customer and provide what the customer need. The team need to be really quality conscious and have practices & processes that can help in faster response to ensure this.

To be Agile , it is important that the people associated with the project is highly motivated. They should have the shared dream to be successful. Team should take pride in working together and executing the project. Team should be able to define the rules of the project and follow them well. Team should decide when to meet for the daily standups. Team should collectively decide the appraisals of each other through the 360 degree feedback. There should be collective ownership for the delivery and all should work towards ensuring that the common goal is met. There need to be sufficient flexibility in the team for changing the rules of the project as the situation demands. The Scrummaster or the relevant role should support the team in following the rules and protecting the team well.

All the practices of Agile (eg. XP, SCRUM) gives very high focus to the agility of people. In fact all the practices of Agile works effectively only if the people executing it executes with the real objective in mind. I have seen tasks getting completed just because somebody wanted it to be completed. I have also seen tasks getting completed where the whole team had a vested interest to complete and they strived hard to complete. The difference is very visible. The quality of the second one really shows. In my opinion there is no alternative to have a highly motivated and energized team to have a good agile team. If any team want to see their customers really happy and deliver what they want, they must have a motivated and energized team to execute the project. Unlike other industry, software development is highly people centric. So, of course, it is different and it should be handled differently. We must encourage team to open up and build up a very positive environment of Trust where the members feel motivated, energized and feel like doing anything for achieving the common goal. Real Agility can only be achieved by this. The practices of Agile are just a means to achieve this. There can be more and better practices which can be found. But we need to keep people as our centre of focus.

Building a highly self organized, motivated and energized team does not mean that Business is not the focus. Our team exists due to the Business and we need to do everything possible to ensure that our business is successful. A self organized and motivated team can achieve and deliver more business..

References
1. The Art of Agile Development- James Shore & Shane Warden
2. The Software Project Manager's Bridge to Agility- Michele Sliger and Stacia Broderick

Top Reasons why Continuous Integration practice fail or result not seen

In this post I would try to bring my perspective of what are the top reasons where Continuous Integration fails or in many cases the benefits are not seen immediately. Many cases I have seen that team looses hope and feel that it is an overhead which Agile practices has brought in. One of the reason may be that the Team / Team member or the management is looking for a quick solution without deep thinking about how Continuous Integration practice can be done really well.

Among many of my readings I have found that the article by Martin Fowler is very useful. Thanks to Martin for his writing:
http://martinfowler.com/articles/continuousIntegration.html

Reason 1- Every team situation, competency , technology, domain and dynamics is different. Just copying the CI Design or roadmap from some other team is a wrong idea. Every team should have their own vision about how do they want the CI to look like. What all they want to run in the Continuous Integration System. Some teams try to run all the tools of inspection, all test cases together and starting with real continuous integration immediately without the infrastructure in place. That may not be a good idea in my opinion. It is always better to have a step by step approach. We can start with periodic build, incremental build and slowly move into the continuous integration approach. But if the team really have the competency then they can start immediately. It is really required to do this analysis.

Reason 2- Assuming that all the team members are very good in test coding skills is a wrong idea. Many times it is found that the test suite organization skills are poor in team members and there is no systematic training. The test code writing skills are not good in some cases due to which the test cases does not work properly. This is one of the reason where CI does not work effectively and there are frequent build failures.

Reason 3- User stories are not granular enough. During the user stories documentation I have seen that the stories has not been written with an overall perspective of granularity so that the developers can really check-in the code frequently and also if the story card can be realized effectively in short period. This is not quite easy and I am sure many people may have different opinion about this. One of the reason which I have seen for this is that the Business Analyst does not write the story as vertical slices from the user perspective. It does not depict the end to end functionality in the story. The requirements are sometimes written in a modular way. Due to this it is very difficult for a coder to checkin at frequent intervals. One pattern is observed that this issue is actually not known and we expect Continuous Integration to work.

Reason 4- CI should be actually run in the clone of production environment. But in big products which are used by many customers there are many environments and the multi-combinational testing, it is required that the strategy of testing is analyzed deeply.
Following is a good example -
Try to answer this

Each user on a computer system has a password which is generally 6-8 characters long, where each character is an uppercase letter or digit. Each password must contain at least 1 digit.
How many password combination is possible.?
The answer is:
2,68,4,483,063,360 possible variations of password exists
1 Test Case Design Time = 1 minute ,60 Test Cases = 60 minutes
1 day= 480 Test Procedure , 155 years to create and execute a complete Test.

In many cases I have seen the team don't analyze the stategy and the CI is run in some combination which is actually not the right representation of the production environment. Teams can use some of the techniques like parwise testing / OAT(Orthogonal Array Testing) to analyze the right combinations. But it is important to use these techniques with good judgement about the test sufficiency and coverage. The Test Strategy is really important to understand the real customer environment, where the test suites should be run and this work should happen during the Infrastructure readiness time for CI,not during the Test Execution time.

Reason 5- The Test Frameworks does not ensure that the SUT system architecture is taken into consideration while designing the overall automation framework. This creates issues, where the frameworks are not scaleable and there are issues related to overall test execution later.
Test Teams must ensure that the Automation framework considers all the aspects of testing like the system under test architecture, test inputs, test outputs, SUT precondition programe state, SUT postcondition programme state, environmental inputs and outputs, component dependency, APIs , capturing the results, analysis of the results etc, the various monitoing tools. In the following sample system the automation framework should be divided into each of the respective areas like GUI, APIs.

Here the test framework can be designed in such a way that the GUIs can be tested independently of the APIs. Most of the functionalities performed by the system might be tested through the APIs. It is better to automate the tests through the API as it is easier and more reliable. GUIs are normally fragile and goes through changes frequently.

Since the focus of the article is continuous integration, I may not elaborate more about the Test Frameworks. Important point is that for running continuous integration systems effectively it is important to have robust test automation system and the automation framework should be scaleable and reliable.

I have found this article very useful and I would highly recommend this for further reading.
http://www.softwarequalitymethods.com/Papers/autoarch.pdf

Reason 6- I have seen sometimes that the team does not have good build scripting skills due to which there are issues. Sometimes the CI Tool chosen does not support administration console for build dependency management etc where it becomes need that there need to be good knowledge on build scripting.

Reason 7- Overall CI Framework of the system is not very clear. Here CI framework does not mean Tool. It means how the different components in the overall system is organized and how it depend on each other and whether the Continuous Integration system is also organized in similiar manner. For big system it is very important that there should be multiple Component level CI System which further builds the overall System level CI. It is also seen that the build staging (Stages to ensure faster feedback in the respective CI System) is not very clear.

These causes issues where the CI takes longer time to execute and finally the team does not use the CI for actual feedback.

Some further reading

http://martinfowler.com/articles/continuousIntegration.html
http://cruisecontrol.sourceforge.net/
http://hudson-ci.org/
http://www.jetbrains.com/teamcity/

Continuous Integration- Powerful practice focussing on the team culture not merely tool deployment

Continuous Integration is one of very powerful practice that can used for the faster and quality delivery. If Continuous Integration has to be implemented well in teams, it is important to have the deep level of software engineering practice implementation right. If teams assume the Continuous Integration can be well done only if the same is thought about during the development phase then it may not be very successful.

CI is one of the most powerful practice that can help in ensuring that we have the working software available at any given point of time. It must be understoood that continuous integration is not just a tool deployment. It is a culture adoption.
In my opinion the team should have following aspects ( just mentioned few important ones) very well implemented during the software development life cycle:

• During Requirements development ensure that the requirements / user stories are granular enough so that later developers are able to checkin frequently.
• User stories should have the right acceptance test cases identified. Further when the test cases are automated it can be ensured that the basic acceptance test cases can be run in the stage 1 of the build so that the feedback is available early.
• Test Automation is thought and designed from the scalability perspective
• Test Suite Naming and Organization is well thought about before the beginning
• Test Strategy should be well thought.
• Testers and Developers are training on coding, test coding, test naming convention, build scripting etc.
• Team is trained on CI
• There should be a good private build setup.

The team must ensure that before the actual execution of the project, the team should have all he infrastructure ready so that the time can be really productive during the actual work and focus on the functional work rather than infrastructure readiness.

Following are some the varioius aspects of Continuous Integration

• Continuous Testing
• Continuous Build
• Continuous Inspection
• Continuous Feedback
• Continuous Deployment and Of Course
• Continuous Improvement :-)

In our further blogs we will talk about how to think about system perspective while thinking about CI.

Suggested Reading:

• Continuous Integration- Improving Software Quality and Reducing Risk - Paul M Duvall, Steve Matyas, Andrew Glover

Agile Practices that can be adopted

The collection of Agile Practices of XP, SCRUM, Crystal etc can be used for faster and better delivery process. Following are some of the practices which team can think of adopting:
• Continuous Integration
• Automated Testing
• Collective Code Ownership
• Private Build
• Incremental Testing
• Iterative Delivery
• Product Backlog and Iterative Planning
• Daily Stand-up Meeting
• Simple and Evolutionary Design
• Customer Demo and Feedback
• Architecture and Iteration Modelling
• Epic and Story Card
• Pair Programming
• Test Driven Development

Adopting Agile- Some key strategy

Adopting Agile Principles and Practices is sometimes due to the hype created in the industry. Sometimes it is due to some fancy terms used by some people.

Before any organization adopting the agile practices, it is always better to understand the real meaning of agile. Just by following some practices the team cannot become agile.

During my readings I have found the book The Art of Agile Development - By James Shore, Shane Warden a very interesting read.

I will try to cover some of the pre-requisite for Agile Adoption from any team's perspective.

Strategy 1- Team agreement to follow Agile

I feel the most important point regarding the adoption of Agile strategy is the team acceptance that the agile practice deployment would help in present issues faced by the customer and better customer delivery. If the team members don’t want to use the practices and if it is forced by some other team or management to adopt agile, it probably may not work.

May be we need to have some strategy to adopt principles where there is awareness issue in team.

Strategy 2- Strong Management support and acceptance about the change

The next most important aspect is the support provided by management for this change. Management can be slightly tolerant from the perspective that even if it fails it is okay.
Also it will really help in bringing the change in the organization as it also requires some infrastructure support and othe related support

Strategy 3- Availaibility of Customer or in worst case Proxy Customer for Feedback Purpose

It is very important in Agile mode of development that customer be closer to the development team so that feedback can be given immediately about the progress of the work. In my opinion the biggest advantage of being Agile or following agile practices is to give the customer what they want.
It is better that the customer is avaialble during the development and the customer should provide feedback of the iterative development. In worst case we can have the customer representative or the proxy customer who can provide the feedback.

Strategy 4- Good understanding about the Agile Principles and Practices

Team should have a very good understanding about the Agile values , principles and practices.
Agile should not be considered as some silver bullet solution for all the existing problems. It cannot solve the problems with with magic wand. Also we cannot delete all the existing process of an organization and live in a virtual world of agility.
In fact there is nothing like this.
Every team need to identify their own way of being Agile and provide the customer the best delivery.
Teams must understand the agile values, be self organized and deeply motivated to give the best. Teams can come up with the operating rules and follow that.

Strategy 5- Team Organization

It is upto the way the team adopt agile. Agile works in teams based on the way teams adopt. Be it a small team or a big team each team must ensure that the teams are componentized well, divide the team into smaller groups and have a good structure in place. This really helps in ensuring that all aspects work well

Strategy 6- Adopt the practices of Agile with the real objective

Agile practices of XP or SCRUM is a collection of practices which complement to each other. It is always better to use the practices together.
For example if we are doing simple desing and if dont do TDD then it becomes difficult. Same way if team is doing Iterative Delivery without following Automation and Continuous Integration then it is next to impossible to give iteriative delivery in 2 weeks sprint.

All the practiecs of XP and SCRUM should be considered in clusters and adopted else there are chances of failure.

Agile Value- Customer Collaboration over Contract Negotiation

There were times when service providers used to negotiate with the customer for what services shall be provide and what not. There were all permutations and combinations done and finally both the party signs the contracts. The contracts has all the details right from what requirement going to be delivered to contract about how to handle changes. Any changes that comes later to the contract negotiation goes through a big bureaucratic process to ensure that each part project each other better. One of the important value of Agility is that there is a contract but rather than negotiating on the contract there is more and more collaboration with the customer . There are cases which we have come across where there are teams to which the customer says “ I don’t want the requirement as of now”, there is reply from the team that “ It is mentioned in the contract so we will deliver. After delivering we will take up change requirement”. This a good example of “waste”. An Agile team focus on delivering value and focus more on collaborating with the customer.

Agile Value- Working Software over Comprehensive Documentation

I remember days when the project manager used to ask the team members about the progress of the tasks and he gets the reply that it is “50% complete” or “82% complete”. Nobody knows what does “50% complete” or “82% complete means. Only the team member is aware of the status. Finally the project managers does some guess work about the completion status and projects the overall status as some x% as part of the project status report. The stakeholders are happy about the progress without knowing the underlying truth about the real progress of the project. In the Agile Model of Development if there is any need for status projection the status is always reflected in the latest build of the Continuous Integration Server. The feature which is not available as part of the build means the progress is not there. So the things are quite clear in agile mode of development.
Many times there is a myth we have observed that “Agility means no documentation”. Here we would like to bring our experiences into effect. It is to be noted that Agility is not a rule book where somebody can specify about how much to be documented at specific scenarios. Each project is different and situation also is quite unique. No experiences or scenarios can be compared. Now setting this context we would give the background. There are experienced Agile practitioners who feel that there is no need of requirement or design artefacts; they can directly sit with the customers and do the coding. There are set of experience agile practitioners who feel there should be some basic documentation of requirements and further to that the coding and testing process can start. And many such different experiences.. We can do search over the net and we can find many such experiences. There is nothing wrong in reading such experiences. But we should not get drifted with such thoughts or experiences. How much documentation or how detailed the documentation should be a call which team or organization can take based on the specific condition in which the teams are operating. We should not overdo neither should we underdo. It should be a balance. To find out the right balance is the challenge. The need of right and enough documentation depends on factors like criticality of the projects, experience and skill sets of team members, availability of right tools, maturity level of team and other factors.
In Agile Teams, the working building is always available. Continuous Integration practice is being followed by teams where integration is a non event. Each Build compiles the source code , runs the necessary test, inspection tools, other necessary tools and have the completed build multiple times a day.

Agile Values, Principles and Perspective

Agile Manifesto, which focusses on the Agile Values and Principles is well explained in the following website:
http://www.agilemanifesto.org/
In my opinion every team before embracing the Agile Practices must deeply understand the Agile Values and Principles well. This will really ensure that the team will adhere to the principles and not merely focuses on practices. We must realize that practices are just means to achieve agility.

Principles behind the Agile Manifesto
(http://www.agilemanifesto.org/principles.html)

We follow these principles:
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity--the art of maximizing the amount
of work not done--is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.