CSC/ECE 517 Fall 2011/ch6 6d sk
6d. The Agile landscape. What are the main agile development methodologies? What are their origins? How do they differ? How widely are they practiced? What evidence exists for their effectiveness?
Overview
Traditionally, software has been developed using rigid methodologies such as the Waterfall method of software development. The Waterfall methodology is a set of stages such as - Analysis, Design, Implementation, Testing and Maintenance. These stages follow a strictly linear sequence as mentioned in the last sentence. This method works perfectly well when a project is completely planned, but it has been observed that it is not always the case. The waterfall model is not capable to handle changes requested by the customers when the project has gone deep into the pipeline. In such cases, all the processes have to be followed to incorporate the requested changes, which consumes time and costs dearly to the customer.
Developers who were facing this problem more often started to realize that real world tends to have quite an impact on the development of any process or product. It was time to have a new strategy for software development which was capable enough to overcome the shortcomings of traditional software development methodologies. There are various such methodologies which provide fast and iterative development, ability to handle late changes in requirements and many more features. These methodologies are collectively called as Agile Software Development Methodologies.
The following article provides a brief introduction to Agile Software Development, the various methodologies for agile development, their comparison and some statistics to demonstrate the acceptance and the success rates of agile methodologies.
Introduction to Agile Software Development
In 1990's, software developers started to realize that a planned waterfall approach for software development had many shortcomings and did not facilitate speedy and flexible software development. There were many efforts to devise new methodologies which overcame those shortcomings. These methodologies were called the Agile Methodologies for software development. They allowed developers to adapt to the changes in demands during software development as well as incorporate future changes to the software during any stage of software development.
The agile methodologies are based on iterative and incremental development. In organizations who follow agile development, requirements for a product are formed mainly through collaboration between small teams which are cross-functional and self-organizing. It allows the process of software development to be iterative, time bound, adaptive, evolutionary and flexible. A team learns how to be Agile in the environment provided by the Agile processes. There is more importance to how a team works together, allowing teams to work in a cohesive fashion, increasing the productivity many times. The increase in productivity is such as a team can provide a working software in the first week, and then improve with subsequent iterations.
Agile Manifesto
n February 2001, a group of software developers, interested in developing new lightweight methodologies for software development, published the Manifesto for Agile Software Development. The Agile Manifesto forms the core of agile software development. The manifesto reads as follows: <ref name = "Agile Manifesto"></ref>
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
The text in bold face are the main terms used in the Manifesto. We describe each of them below:<ref name = "Agile Manifesto Wiki"></ref>
Individuals and Interactions: Agile methodologies are focused on individuals and interactions between them. They have more emphasis on self-organization and motivation and are based on the principles of co-location and pair-programming.
Working Software: A working software developed quickly will be useful rather than presenting documents to customers
Customer Collaboration: It is not possible to fully gather requirements at the beginning of a software development cycle. so it is important to keep the customer involved during the development process and receive their feedback
Responding to change: Quickly respond to changes in requirements through continuous iterative development process.
Principles of Agile Manifesto
The supporters of Agile Manifesto follow the 12 principles mentioned below:<ref name = "Agile Manifesto Principles"></ref>
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity--the art of maximizing the amount of work not done--is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
If you believe in and endorse the Agile Manifesto, you can become a signatory here.
Agile Methodologies
Extreme Programming
Extreme programming (XP) is a popular approach to agile software development and was created by Kent Beck in 1996. XP is defined by a set of five values that establish a foundation for all work performed as part of development process. These five values are communication, simplicity, feedback, courage, and respect. Each individual value serves as a driver for specific activities, actions, and tasks <ref name = "Pressman"></ref>.
- Communication - XP emphasize close, yet informal communication between customers and developers. This helps ensure effective communication between software engineers and other stakeholders.
- Simplicity - In order to achieve simplicity, XP requires developers to design only for the immediate needs of the project. This results in a simple design that can be implemented easy.
- Feedback - Feedback is received from three sources: software tests, the customer, and other software team members.
- Courage - In order to adhere to certain XP practices, Beck states that is requires courage.
- Respect - By following the other four values, an agile team will develop respect among its members and other stakeholder. As the team begins to achieve success in their delivery of software increments, they will also develop respect for the XP process itself.
XP consists of four framework activities: planning, design, coding, and testing as seen in the figure to the right. Each phase is summarized in the paragraphs that follow.
- Planning - The initial phase of the process involves the planning game. It is a requirements gathering activity that enables the members of the team to understand the business context for the software and involves listening to the customer. This leads to the creation of a set of user stories that describe the required output, features, and functionality of the software. Each story written by the customer is written on a index card and assigned a value based on its overall business value. Members of the team will then determine a cost (measured in weeks) for each story based on their estimation for how much work would be required to implement the story. The customers and developers will then work together to determine which group of stories will be implemented in the next iteration of the software.
- Design - The design phase of the process involves the creation of a simple design of the system and also provides implementation guidance for each story to be included in the iteration.
- Coding - After a preliminary design for the system is created, the team will then develop a set of unit tests for the system. These tests will serve as guidelines for the developers to focus only on getting the tests to pass and nothing more. One of the defining aspects of XP, involves the pair programming concept. XP recommends that two programmers should work together at one computer workstation to develop code. One person writes the code, while the other will watch and provide feedback. After a period of time, the two programmers will switch their roles.
- Test - In order to place an emphasize on tests, the unit tests generated during the code phase are ran anytime the code is modified to ensure no problems exist in the codebase.
Scrum
Scrum (derived from a term used in rugby) is an agile process created by Jeff Sutherland in the early 1990s. The basic principles of Scrum are consistent with the Agile Manifesto and serve to guide development. Some of the distinguishing features of Scrum include <ref name = "Pressman"></ref>:
- Work occurs in “sprints” and is derived from a “backlog” of existing requirements
- Testing and documentation are on-going as the product is constructed
- Meetings are very short and sometimes conducted without chairs
- Product demonstrations are delivered to the customer with the time-box allocated
The overall flow of the Scrum process is illustrated in the diagram to the right. Scrum emphasizes the use of a set of software process patterns that have proven effective for projects with tight timelines, changing requirements, and business criticality.
The defining characteristics of Scrum include:
- Product Backlog - A list of project features that provide business value for the customer. New requirements can be added to the backlog at any time. The project manager is responsible for accessing the backlog and updating the priority of each individual requirement.
- Sprint Backlog - Each iteration of the process (called a Sprint) involves choosing a subset of the product backlog that will be implemented.
- Sprint - A typical sprint occurs over a 30 day time-box and involves the implementation of requirements currently in the sprint backlog.
- Scrum Meetings - Every 24 hours, the project team will hold a meeting to discuss three key questions; what did you do since the last team meeting?, what obstacles are you encountering?, what do you plan to accomplish by the next team meeting?
- Release - At the end of each sprint, a working iteration of the software is release to the customer to evaluate. Any new features requested will then be entered into the product backlog for another iteration of the software.
Dynamic Systems Development Method
The Dynamic Systems Development Method (DSDM) is another agile software process that "provides a framework for building and maintaining systems which meet tight time constraints through the use of incremental prototyping in a controlled project environment." DRDM is an iterative process which follows the 80 percent rule, meaning that 80 percent of an application can be delivered in 20 percent of the time it would take to deliver the complete application <ref name = "Pressman"></ref>.
DSDM is structured into five phases:
- Feasibility study - The initial feasibility study establishes the business requirements and any constraints associated with the application. This phase is also used to determine if the application is a viable candidate for the DSDM process.
- Business Study - In this phase, the functional and information requirements are established which gives the application business value. The basic architecture of the application and maintainability requirements for the application are also defined.
- Functional Model Iteration - In the fuctional model interation phase, a set of incremental prototypes of the application are demonstrated to the customer to show functionality. By gathering feedback from the customer, the team can gather additional requirements to be included as part of the application.
- Design and Build Iteration - The project team will revisit the prototypes built during the functional model iteration to ensure that they have been designed in a way that they will provide operational business value to the customer. This phase can run concurrently with the functional model iteration phase in some cases.
- Implementation - In the implementation phase, the lastest software increment is deployed to the customer.
Crystal Family of Methodologies
The Crystal family of agile methods were create by Alistair Cockburn and Jim Highsmith. The main goal was to achieve a software development approach that puts an emphasize on "maneuverability" during "a resource-limited, cooperative game of invention and communication, with a primary goal of delivering useful, working software and a secondary goal of setting up for the next game. <ref name = "Pressman"></ref>"
A defining part of Crystal is the use of several different colors to indicate the "heaviness" of the process. A project team trying to determine which process to pick would consider its size and criticality as the major factors. In addition to multiple colors, there are four levels of potential loss as a result of a system failure <ref name = "Agile"></ref>.
These four levels from smallest to largest potential loss are:
- Comfort (C)
- Discretionary Money (D)
- Essential Money (E)
- Life (L)
The lowest potential loss would be Comfort, in this case a system failure would simply be an inconvenience for the customer. On the other end of the spectrum, Life as the name implies means that a system failure would result in loss of life.
Crystal Clear & Orange
Two of the more well known methodologies in the Crystal family are Crystal Clear & Orange. The only difference between the two is that Crystal Clear is designed for one project team while Crystal Orange is for a multi-team project.
The fundamental principles of both include <ref name = "Orange"></ref>:
- Incremental delivery
- Project tracking
- End-user participation
- Automated Regression
- Two user viewing
- Workshop
Feature Driven Development
Feature-Driven Development (FDD) is an agile process that is model-driven. Models, which are written in UML and other standardized languages, provide a way to identify units of functionality and understanding the context in which the units are used. It is entirely focused around discovering and implementing system features. A system feature is any distinct unit of functionality that is valued by the client. In most cases, features correspond to system use cases, but may also involve functionality not directly used by an external actor (for example the customer) <ref name = "Pressman"></ref>.
The FDD process involves five phases:
- Develop an Overall Model
- Build a Features List
- Plan by Feature
- Design by Feature
- Build by Feature
The first three phases, which focus on planning, are performed at the beginning of a project, while the final two phases are typically implemented by a series of short iterations. Other then the software deliverable, the only documentation includes system feature and class models.
Adaptive Software Development
Adaptive Software Development (ASD) was proposed by Jim Highsmith as an agile process model for building complex software and systems. The two core ideologies of the process are human collaboration and team self-organization.
The three phases of ASD are <ref name = "Agile"></ref>:
- Speculation - In this initial phase, the project is started by conducting adaptive cycle planning. Adaptive cycle planning uses the customer mission statement, project constraints, and basic requirements to define the set of release cycles (software increments) that wil be part of the project. Since the cycle plan will inevitably change, after completion of the first cycle, it is reviewed and adjusted to better fit the reality in which the team is working.
- Collaboration - Next in collaboration, the project team performs requirements gathering and performs joint application development (JAD). A JAD session involves developers and customer representatives meeting to discuss project features and serves as a way to enhance communication.
- Learning - The final learning phase is used to conduct quality assurance and deployment of the application. Lessons learned during the entire ASD process is then incorporated into the next iteration of the software.
Agile Unified Process
The Agile Unified Process (AUP) is a simplified version of the Rational Unified Process (RUP) created by IBM. It provides an easy to understand approach to develop applications using the defining agile concepts while still remaining true to RUP.
The four phases of RUP are <ref name = "AUP"></ref>:
- Inception - The goal of this phase is to determine the overall scope of the project, as well as obtaining project funding and determining a potential architecture of the application.
- Elaboration- In the elaboration phase, the architecture proposed during the previous phase is expanded and developed into a usable design.
- Construction- In the construction phase, the goal is to build working software on an incremental basis while ensuring the highest-priority requirements are being implemented first.
- Transition- Lastly, the transition phase is used for validation and project deployment.
Other Methodologies
Other less known agile methodologies include:
- Agile Modeling
- Lean Software Development
- Essential Unified Process
- Open Unified Process
- Velocity Tracking
Comparison of different agile methodologies
In this section, we compare the different agile methodologies mentioned in this text based on multiple aspects. In general, the agile life cycle is more or less similar in all the methodologies. It involves phases such as - Project Initiation, Planning, Requirement Elaboration, Architecture design followed by one or more releases. A Realese is the software that will be delivered to the customer. Each release is made up of timeboxes which are 1 -6 weeks long and have a fixed delivery date. We compare the Agile metthods based on the practices followed in each method for each phase of the Generic Agile Life Cycle.<ref name = "Agile"></ref><ref name = "Balagan"></ref>
Project Initiation
This stage involves setting up project and justifying the need and the business case for the project. It also involves elaborating vary high level requirements. Not all methodologies give equal importance for justifying the business case for the project, and have different names for the vision statement that mentions the goal of the project. The level of formality varies greatly from one method to another. It is important to know the practices followed by each method in order to select the right method for the project.
Formality and Vision
Scrum is the least formal of all. Requirements are documented as Backlogs and the only other documentation is the code. The "sprint Goal" is the timebox vision statement for Scrum.
In XP, the documentation involves Stories, Running code and the Tests. XP is a little more formal than Scrum. The "Metaphor" is the product vision for XP.
Crystal Orange is much more formal and has documentation in the form of Release sequences, Project Schedule, Requirements document, Use case documents, Design sketches, Object models, Running code, Test cases, User manuals, Status reports and Inter-team specifications.
DSDM also provides a high level of formality. It has documents such as Feasibility report which is the project vision, Outline plan, Business Area Definition, Requirements List, System Architecture Definition, Development Plan, Functional Model, Functional Prototype, Design Prototype, Tested system, Delivered system, Implementation Plan, Development Risk Analysis Report, Review Records, Test records, User documentation and Project Review Document.
FDD does not cover all the stages of the life cycle, but still provides feature lists and running code.
ASD and AUD too provide documentation in the form of Requirements, Running code, Test cases, etc.
Requirements Gathering
In Scrum, a Product Backlog List is created which is maintained by the Product Owner.
In XP, requirements are mentioned in the form of Stories by the customer. Projects which use XP generally have a collection of such Story cards. Scrum and XP do not involve a separate stage for Elaborating Requirements.
In FDD, a feature list is created from an existing requirements document.
In Crystal Orange and DSDM have a separate Elaboration phase in workshops are conducted to identify high level requirements which are documented in a Requirments document.
ASD provides Joint Application Development (JAD) sessions where the customers and developers discuss the features to be included in the product.
Project Planning
Having a project plan is always beneficial to track the project and take to completion in desirable time. The methods mentioned in this article have different rules for preparing a project plan. Some methods such as DSDM make it mandatory to have a project plan while others do not give much emphasis on having one. All the agile methods have a similar approach toward project planning. Releases and Timeboxes are basic elements of a project plan in all the methods. The plan includes multiple Releases with ussually 3 Timeboxes in each Release, except for DSDM which has only one Release.
Estimating the time required for to develop a particular component or a feature is done by different individuals. In Scrum, the Owner estimates Backlog items in days. In XP, groups of developers estimate a Story in Days, Months or Weeks. In ASD too, there is a high level of interaction between the customer and the developers through the JAD sessions. In DSDM and Crystal Orange,Project Managers and developers estimate the effort based on the size of the deliverable. DSDM uses Function Points while Crystal Orange uses number of Classes in the Object model.
Roles and Responsibilities
Team Size and Visibility
Projects following Agile methods have small, cross-functional teams based on the principles of self-organization and co-location. The team size varies from one method to another, and usually contains 3-16 members. The number of teams per project also varies depending on the method followed by the project.
XP usually has a single team per project with 3-16 members in the team. Members of the team include the Customer, Programmers, Testers. Tracker and a Coach, Consultant and the Manager who is the Big Boss of the project.
Scrum has 1-4 or more teams per project. Each team can have between 5-9 members. Members of the team usually include Scrum Master, Experienced Engineer, Junior Engineer, Testers, etc. The Project Manager, Scrum Master and the Owner form the project roles.
The size of a team for Crystal Orange can be anywhere between 4-8. The number of teams in a project varies and can go up to 40. Usually, 10 teams are assigned to a single project. The team consists of a Business Analyst, Designer, Programmers, UI Designer, Testers, etc. The main Project Roles include Sponsor, Project Manager, Architect, Technical Facilitator and Design Mentor.
In DSDM, each project can be made up of 1-6 teams, with each team having 1-6 members. Team typically includes Team Leader, Ambassador User, Senior Developer, Developer, Scribe and the main project roles are Visionary, Executive Sponsor, Project Manager, Technical Co-ordinator, Facilitator.
Due to small team sizes, developers can be stressed due the pressure to meet the deadlines. In most of the methods, this is handled by the 40 hour week rule which takes care of the work-life balance of the developers. Also, developers have a more visibility to their work and progress because of practices such as Pair programming (XP), Peer Reviews (DSDM), User Reviews (XP, Scrum, DSDM), Daily meetings (XP, Scrum), which drives fast development and achieves better quality.
Customer Involvement
All agile methods have the customer involved in the process of software development. The role of customer and the way in which developers interact with the customer may vary, but the customer is involved none the less. XP does it by directly involving the customer in the project team. Scrum has the customer prepare the Backlog list with the help of developers. Crystal Orange and DSDM facilitate workshops for the customers to identify requirements. ASD has JAD sessions with the customers.
Project Manager
The role of Project Manager in Agile methods is somewhat different than that in traditional methods. In Agile methods, the Project Manager merely guides the team, but does not have much control on how the team chooses to perform its tasks. The names for this role are different in different agile methods, but essentially all of them perform similar tasks in the respective agile methods. The following definition of a Project Manager is constructed by combining description of the Project Manager, Scrum Master, Coach, and Tracker from XP, DSDM, Crystal Orange and Scrum and taken from here:
Gather information from all stakeholders and integrate into workable plan; log estimates; advise programmers on their estimates using historical figures; log task assignments; sustain high rate of progress; make decisions; remove impediments; build and sustain team culture; ensure team sticks to process; liaise between management and team; manage customer relationships; fend off “feature creep” and other project hazards; get agreement on the prioritisation of requirements and detailed content of timeboxes; track progress against the timebox goals/plan; collect information without disturbing the process; update plans; communicate progress; ensure that the lessons are learnt; log defects; stay calm.
Controlling the Project
Release Management
XP and Scrum provide flexibility in managing the releases. In both methods, the customer can announce the project "done" at any time. This is not the case with other methods.
Communication
Emphasis is on communication within a team and within teams in all the Agile methods. Intra team communication is achieved by co-location, Daily Stand ups (XP), Scrum meetings (Scrum), Big charts (XP), Backlog Graph (Scrum), Information Radiators (Crystal Orange). Teams communicate with each other in an open area, there Daily Scrum of Scrums meetings (Scrum), Big charts, Inter-team spec (Crystal Orange). Reports are generated in the form of charts, frequent delivery reports and Status reports (Crystal Orange).
Change Management
Time is given importance over functionality. Therefore if the functionality is too ambitious for completion in the current timebox, it is discarded. The customers have to make trade offs on what they want. Big changes that are demanded outside of the timebox are traded off with the customer by making necessary changes to Project Plan and reestimating. Scrum does not allow big changes within a timebox, while XP handles it as a new Story. DSDM allows Objective Setting meetings inside a timebox.
Quality Management
All agile methods involve quality checking by continuously testing the implemented functionality. XP has stated Quality Goals and Peer Reviews are made mandatory in XP. It also has a XUnit framework to automate Unit Test cases. DSDM too has Peer Reviews. Continuous Integration Testing is common in XP and Scrum User Acceptance Testing is carried out in parallel to development in XP and DSDM.
Development
Agile methods vary when it comes to following coding standards. XP, DSDM and Crystal Orange define a set of Coding Standards, but only DSDM and XP make it mandatory to follow them. For designing the architecture, DSDM has a mandatory architecture phase, which is optional in XP and Crystal Orange. XP, Scrum and Crystal Orange have a simple design model such as a whiteboard, while DSDM and Crystal Orange use Common Object Model, Context Diagrams, Data Models, etc.
This section uses many terms which you will come across while following Agile methods. It is not possible to provide definition for each term in this article, simply because there are too many of them. The article here provides definitions for all the terms mentioned in this write-up.
Adoption and Effectiveness of Agile Methodologies
Agile methodologies have made a foray into the software industry. Developers and organizations have started to realize the benefits of iterative and incremental approach to software development. Various surveys performed over the years show an increasing trend in adoption of agile methods. One such survey conducted by Forrester in 2006 shows that 17% North American and European enterprises use agile methods and 29% are aware of them. The number increased to 35% enterprises use Agile methods.<ref name = "Forrester06"></ref><ref name = "Forrester09"></ref> The results of a survey carried out in 2007 are mentioned here.
There are also studies conducted that provide the evidence that Agile methods work in the real world and it is indeed possible to achieve better success rates than traditional methods. The results of the DDJ 2007 Project Success Survey show that the success rate is highest among the projects that follow agile methods as compared to other methodologies. The figure beside shows 72% success rate for Agile versus 63% for Traditional methods. The success rates for projects that follow Data Warehouse and Offshoring methodologies is even lesser.
Another aspect of agile development that is hugely responsible for the ongoing success of agile methods is the short feedback cycle. Most of the enterprises which were surveyed in the above mentioned surveys agreed that short feedback loops have played a major role in achieving high success rates. Short loops facilitate continuous Improvement Process in which the developers are constantly learning and this helps them acquire deep knowledge about the product. This leads to a better quality software and higher levels of success.<ref name = "Feedback"></ref>
Conclusion
Agile Development is a set of practices and guidelines for software development. They overcome the shortcomings of the traditional software development methods and provide flexibility to assimilate changes in requirements in late stages of software development. However varied they are in the practices involved, all the methods described here follow the Generic Agile Life Cycle. Surveys show the increasing popularity of Agile methods and the increase in the project success rates due to their adoption.
References
<references> <ref name = "Agile Manifesto">Agile Manifesto </ref> <ref name = "Agile Manifesto Wiki">Agile Manifesto on Wikipedia</ref> <ref name = "Agile Manifesto Principles">Manifesto Principles</ref> <ref name = "Pressman">R. S. Pressman, Software Engineering: A Practitioner's Approach, 7/e. McGraw-Hill, 2010.</ref> <ref name = "Scrum">Scrum Process Model Image</ref> <ref name = "DSDM">What is DSDM?</ref> <ref name = "Crystal">Crystal Methods</ref> <ref name = "Agile">P. Abrahamsson, O. Salo, J. Ronkainen, and J. Warsta, Agile software development methods: Review and analysis. VTT Publications, 2002.</ref> <ref name = "Orange">P. K. Gorakavi, What You Should Know about Crystal Orange Methodology, ASAPM, 2010</ref> <ref name = "AUP">S. W. Ambler, Agile Unified Process, accessed November 17, 2011.</ref> <ref name = "Balagan">An Agile Comparison</ref> <ref name = "Forrester06">C. Schwaber, The Truth About Agile Processes, Forrester, August 2007 </ref> <ref name = "Forrester09">D. West and T. Grant, [Agile Development: Mainstream Adoption Has Changed Agility http://www.osp.ru/netcat_files/18/10/h_d8eddd303b6cf0c38c23601c4363bee4], January 2010</ref> <ref name = "Feedback">T. Tuttle, Agile Fundamentals: Follow The Knowledge, March 2011</ref> <ref name = "Success">DDJ's 2007 Project Success Survey</ref> </references>