CSC/ECE 517 Fall 2011/ch18 6d na

---

Overview

Traditional software development methodologies like waterfall methodology are notorious for their lack of flexibility and slowness. That’s why today traditional development methodologies are often replaced by agile development methodologies, which accelerate the process of software development and allow quicker project finalization. Agile software development methodologies give a chance to step away from bureaucracy and strike a balance between laissez-faire [1] style and immersion into paper work.

Introduction to Agile Software Development

Agile Software Development is a concept, a philosophy and a methodology which evolved in the 90's as an answer to the long-growing frustrations of the waterfall Software Development Life Cycle [SDLC] [2]concepts. The term promotes an iterative approach to software development using shorter and lightweight development cycles and some deliverable.

After extensive research, we have found that like other resources, the term Agile as it relates to modern software development, was being used during the 1990's in various published articles both on the internet as well as ComputerWorld/InfoWorld. The papers were based on people looking for a new approach to software process. While the ideas were not new, they gained enough steam for people to pay close attention especially while search engines on the internet became very popular. There is another alliance called the Agile Manifesto, which in our opinion, is a disorganized web site that has some claims to doctrines and philosophies, all borrowed and bastardized to look like an original concept. The web producer put a picture from a meeting in SnowBird to resemble the framers of the constitution and the Declaration of Independence. They then hijacked the term 'Agile' in early 2001 when a bunch of people who had been heavily involved in the Agile concepts got together to exchange ideas and came up with the Manifesto for Agile Software Development.

The Agile workshop was organized, by Jim Highsmith [3] and Bob Martin[4]. They contacted people who they felt were actively involved in software development communities with these similar ideas and got seventeen of them together for the Snowbird workshop. Initially, they wanted to get together and build better understanding of each others' approaches. Robert Martin was keen to get some statement, a manifesto that could be used to rally the industry behind these kinds of techniques. We also decided we wanted to choose a name to act as an umbrella name for the various approaches. One next step that did follow, with the active involvement of many of these authors, was the formation of the agile alliance. This group is a non-profit group intended to promote and research agile methods. Amongst other things it sponsors an annual conference in the US.

Agile Manifesto

Manifesto for Agile Software development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

working software over comprehensive documentation

customer collaboration over contract negotiation

responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Agile planning activities

In non-iterative methodologies like waterfall, there is large upfront design, aiming to predict accurately how much work is involved in each project activity. This leads to a significant investment early in the project, when it is uncertain that the designed functionality is actually desired. Any agile approach to large-scale development has to avoid the reintroduction of the big design up front. One solution is to incorporate levels of planning incorporate a view of ‘the whole.' Agile planning activities for large-scale development efforts should rely on these five levels.

Level 1 - Product Visioning: The highest-level view that the stakeholder can paint of the future is the product vision. In this vision, they explain what an organization or product should look like after project completion. They indicate what parts of the system need to change and what efforts can be used to achieve this goal.

Level 2 - Product Roadmap: The era of large-scale projects that deliver results in years is behind us. Customers demand faster frequent changes, and delivery is measured in weeks or months. The higher frequency and smaller time-frames force a product owner into thinking in steps - into thinking of a road towards the final product. A product roadmap is created and communicated to fellow delivery people to provide a map so concept is more of a reality.

Level 3 - Release Planning: In small projects, the product backlog can provide adequate project overview. The size, duration and deliverables are easily recognized, and there is no need to synchronize deliverables or teams. All of this changes when applying agile concepts to programs. The first time when routing activities and allocating them to teams occurs during release planning.

Level 4 - Iteration Planning: For each iteration within the release, a planning session occurs to append detail and increase accuracy. Before or during the session, detail is added to the features by breaking them down into tasks. The actual capacity of the individual teams is known with more certainty than during the release planning session. The combination of these increased accuracies helps the team commit to delivering a number of features during the iteration with a high degree of certainty.

Level 5 - Daily Plan: The stand-up meeting is part of everyday life for agile teams. This daily meeting is not often seen as a planning session, but it certainly is. The people look a day ahead, have learned from the earlier days in the iteration, and tell each other what they plan on doing. Issues are raised, possibly addressed, and the success of delivering the desired features within the iteration can be determined after the meeting.

Agile Methodologies

Scrum

Scrum is a lightweight management framework with broad applicability for managing iterative and incremental projects of all types. Ken Schwaber[5], Mike Beedle[6], Jeff Sutherland[7] and others contributed significantly to the evolution of Scrum over the last decade and a half. Over the last few years in particular, Scrum has garnered increasing popularity in the software community due to its simplicity, proven success and improved productivity, and its ability to act as a wrapper for various engineering practices promoted by other agile methodologies.[8]

Extreme Programming

XP, originally devised by Kent Beck[9], has emerged as one of the more popular and controversial agile methods. XP is a disciplined approach to delivering high-quality software quickly and continuously. It promotes high customer involvement, rapid feedback loops, continuous testing, continuous planning, and close teamwork to deliver working software at very frequent intervals, typically every 1-3 weeks. The original XP recipe is based on four simple values – simplicity, communication, feedback, and courage – and twelve supporting practices: Planning Game, Small Releases, Customer Acceptance Tests, Simple Design, Pair Programming, Test-Driven Development, Refactoring, Continuous Integration, Collective Code Ownership, Coding Standards, Metaphor and Sustainable Pace.[10]

Dynamic Systems Development Method (DSDM)

DSDM, dating back to 1994, grew out of the need to provide an industry standard project delivery framework for what was referred to as Rapid Application Development (RAD) at the time. While RAD was extremely popular in the early 1990’s, the RAD approach to software delivery evolved in a fairly unstructured manner. As a result, the DSDM Consortium was created and convened in 1994, with the goal of devising and promoting a common industry framework for rapid software delivery. Since 1994, the DSDM methodology has evolved and matured to provide a comprehensive foundation for planning, managing, executing, and scaling agile and iterative software development projects. DSDM is based on nine key principles that primarily revolve around business needs/value, active user involvement, empowered teams, frequent delivery, integrated testing, and stakeholder collaboration. DSDM specifically calls out “fitness for business purpose” as the primary criteria for delivery and acceptance of a system, focusing on the useful 80% of the system that can be deployed in 20% of the time.[11]

Crystal

The Crystal methodology is one of the most lightweight, adaptable approaches to software development. Crystal is actually comprised of a family of methodologies (Crystal Clear, Crystal Yellow, Crystal Orange, etc.) whose unique characteristics are driven by several factors such as team size, system criticality, and project priorities. This Crystal family addresses the realization that each project may require a slightly tailored set of policies, practices, and processes in order to meet the project’s unique characteristics.[12]

Lean Software Development

Lean Software Development is an iterative methodology originally developed by Mary and Tom Poppendieck. Lean Software Development owes much of its principles and practices to the Lean Enterprise movement and the practices of companies like Toyota. Lean Software Development focuses the team on delivering value to the customer, and on the efficiency of the "Value Stream," the mechanisms that deliver that value. The main principles of Lean include Eliminating Waste, Amplifying Learning, Deciding as Late as Possible, Delivering as Fast as Possible, Empowering the Team, Building Integrity In, and Seeing the Whole. Lean eliminates waste by selecting only the truly valuable features for a system, prioritizing those selected, and delivering them in small batches. It emphasizes the speed and efficiency of development workflow, and relies on rapid and reliable feedback between programmers and customers.[13]

Feature Driven Development (FDD)

FDD was originally developed by Jeff De Luca, with contributions by M.A. Rajashima, Lim Bak Wee, Paul Szego, Jon Kern and Stephen Palmer. The first incarnations of FDD occurred as a result of collaboration between De Luca and OOD thought-leader Peter Coad. FDD is a model-driven, short-iteration process. It begins with establishing an overall model shape. Then it continues with a series of two-week "design by feature, build by feature" iterations. The features are small, "useful in the eyes of the client" results. FDD designs the rest of the development process around feature delivery using the following eight practices: Domain Object Modeling, Developing by Feature, Component/Class Ownership, Feature Teams, Inspections, Configuration Management, Regular Builds, Visibility of Progress and Results.[14]

Other Methodologies

Kanban [15] as applied to software development is a pull-based planning and execution method. Rather than planning work items up front and pushing them into the work queue of a team, the team signals when they are ready for more work and pulls it into their queue. Kanban historically uses cards to signal the need for an item. For software development teams, these cards are kept on a Kanban Board which is organized into columns and rows. The columns represent the different states of a work item, from initial planning through customer acceptance. The specific columns a team uses should meet the needs of the team and be tailored to their context. The rows on the Kanban Board represent work items. Work items are sometimes grouped within areas, such as feature sets and category types. Kanban focuses on maximizing the throughput of a team. One of the ways it achieves this goal is through the application of Work-in-Process (WIP) limits in each of the states of a work item. Under a Kanban (or Lean) approach, queues or inventories of work in any state are seen as waste. The WIP limits enable a team to focus on the optimal flow of work items through the system, minimizing any associated waste. Kanban allows teams to achieve process optimizations while respecting and maintaining a sustainable pace.

Comparison of different agile methodologies

Limitations of Agile Methodologies

The biggest limitation of agile methodologies is how they handle larger teams. Cockburn and Highsmith both conclude that “Agile development is more difficult for larger teams…as size grows coordinating interfaces become a dominant issue,”. Both Larry Constantine and Martin Fowler also believe that agile with face-to-face communication breaks down and becomes more difficult and complex with developers more than 20. In contrast, heavyweight and plan-driven methods scale better to large projects.

Agile methodologies have a strong emphasis on customer involvement. The customer is considered as part of the development team throughout the whole development of the software. The Standish Group research topped this by providing the second most important factor for a project success is user involvement according to IT executive mangers opinion. According to Boehm, “Agile methods work best when such customers operate in dedicated mode with the development team, and when their tacit knowledge is sufficient for the full span of the application”. Again these methods risk tacit knowledge shortfall. If you have one customer participant then unless they are committed, knowledgeable, collaborative and empowered then there is some chance that you would have a unified set of requirements. However if you have many customers you would have different viewpoints and conflicts between them. This risk could be reduced in plan-driven methods by using documentation, planning, architecture reviews and independent expert project reviews to compensate for on-site customer negligence.

Another factor of agile methodologies that could cause problems is the interpretation of the agile manifesto principle, “Working Software over Comprehensive Documentation”. Boehm[16] questions the applicability of agiles’ emphasis on simplicity. Based on XP’s concept of YAGNI precept: “You Aren’t Going to Need It,” believes that doing extra work to get rid of architectural features that do not support the current version. This might cause misconceptions for the developers. This approach can be useful when the future requirements are highly unpredictable. However where future requirements are predictable, Boehm states “this practice not only throws away valuable architecture support for them, it also creates problems with the customers who want developers to believe that their priorities and evolution requirements are worth accommodating”.[17]

Adoption and Effectiveness of Agile Methodologies

Small team - Agile teams are restricted in size for several reasons: The team has to self-organize, implying an efficient order emerging from temporary chaos. Obviously this process would be too long for larger teams.

− Team members have to be able to communicate spontaneously with each other and with other stakeholders (i.e. without setting up meetings, sending emails, etc.).

− All team members participate in the day-to-day management (tasks, changes, impediments, etc.).

− Team members need to understand what others are doing (cross-functional perspective, team ownership), help each other with ease, and collaborate without centralized control.

− The team has to be co-located (this limitation will be examined later). As you can see, agile is a highly participative style of software development. Therefore, the number of participants significantly affects the efficiency of the process. The daily scrum of the Scrum agile methodology perfectly illustrates this limitation. The daily scrum is a 15 minutes meeting involving the whole team. The ScrumMaster (Scrum word for team leader) goes around the table and each team member mentions the status of tasks and other issues such as impediments or changes. I managed Scrum teams of sizes varying from 3 to 8, and I must admit that in my opinion 8 is about the upper limit. Beyond this size, communication becomes inefficient. Assuming that large projects tend to require large teams, this restriction naturally extends to project size.

Collocated team

Agile emphasizes that face-to-face, spontaneous conversation is the best form of communication. While we can certainly agree on the benefits of this form of communication, it severely limits agile applicability. Moreover, this agile principle extends beyond the development team since other stakeholders such as business analysts are required to be collocated. What does it mean in practice? Imagine that a team member has a question concerninga use case. She should be able to get up, walk 10 meters, ask the business analyst orkey user for clarification, and get back to work. Consequently, office space has to bephysically arranged according to agile projects so that all stakeholders involved in dailyactivities are located at the same place (let's say within a minute of walking distance). We can easily think of a number of situations where this limitation prevents using agile:

− Office space organized by departments: In most organizations, offices are organized by departments such as IT, marketing, accounting, sales, and so on. Moving people around according to agile projects is unrealistic. Even if it was possible, it would negatively affects other parts of the business.

− Distributed environment: Developers are often distributed throughout the organization, whether in the same branch or in different branches (or working from home). Just like for the previous point, moving these people is unrealistic.

− Subcontracting: Many organizations partly or completely outsource software development. Assuming that some roles such as business analysts or key users would be located at the company, this situation doesn't comply with the agile collocation principle.

Team ownership vs. individual accountability

Agile development stresses the importance of team ownership in order to improve teamwork and therefore overall results. Team ownership is a very appealing concept, but how can we implement it since an organization's performance-reward system assesses individual performance and rewards individuals, not teams?

If we rely exclusively on individual accountability, we tend to generate selfish behavior that can affect teamwork. If we rely exclusively on team assessment, we overlook that individuals perform differently in a given team, creating opportunities for underperforming team members to get away with it and lessening incentives to perform in a superior way. Obviously we have to find a way to take both these perspectives into account. Actually this problem can be solved elegantly by defining two levels of performance- reward:

− Team level: The team is perceived as a single entity from management's point of view. Management assesses teams' performance and allocates rewards to teams in the form of points.

− Individual level: Team leaders (or whatever the title is) evaluate teams members rewarding them with points. In order to have a short feedback loop, assessment should be done frequently (every month for example) using a lightweight system incurring very little administrative overhead.

Thanks to this kind of system, we encourage teamwork but we still take individual contribution into account, effectively reaching a balance between team ownership and individual accountability.[18]

Conclusion

Agile software development stresses rapid iterations, small and frequent releases, and evolving requirements facilitated by direct user involvement in the development process. Serena’s application lifecycle management tools provide a framework to visualize scope, orchestrate mundane and repetitive development tasks, and enforce process. Unlike agile-specific products offered by agile-only vendors, Serena products are methodologyneutral and can be applied equally well to agile as well as more traditional serial development processes, so they can support all the development activities within an enterprise.

Any team could be agile, regardless of the team size, but size is an issue because more people make communication harder. There is much experience from small teams. There is less for larger teams, for which scale-up strategies need to be applied.

• Experience is important for an Agile project to succeed, but experience with actually building systems is much more important than experience with Agile methods. It was estimated that 25%-33% of the project personnel must be “competent and experienced”, but the necessary percentage might even be as low as 10% if the teams practice pair programming due to the fact that they mentor each other.

• Reliable and safety-critical projects can be conducted using Agile Methods. The key is that performance requirements are made explicit early, and proper levels of testing are planned. It is easier to address critical issues using Agile Methods since the customer gives requirements, makes important things explicit early and provides continual input.

• Agile Methods require less formal training than traditional methods. Pair programming helps minimize what is needed in terms of training, because people mentor each other. This is more important than regular training that can many times be completed as self-training. Training material is available in particular for XP, Crystal, Scrum, and FDD.

• The three most important success factors are culture, people, and communication. Agile Methods need cultural support otherwise they will not succeed. Competent team members are crucial. Agile Methods use fewer, but more competent people. Physically co-located teams and pair programming support rapid communication. Close interaction with the customer and frequent customer feedback are critical success factors.

• Early warning signs can be spotted in Agile projects, e.g. low morale expressed during the daily meeting. Other signs are production of “useless documentation” and delays of planned iterations.

• Refactoring should be done frequently and of reasonably sized code, keeping the scope down and local. Large-scale refactoring is not a problem, and is more feasible using Agile Methods. Big architectural changes do not need to be risky if a set of automated tests is maintained.

• Documentation should be assigned a cost and its extent be determined by the customer. Many organizations demand more than is needed. The goal should be to communicate effectively and documentation should be the last option.

References

1. http://en.wikipedia.org/wiki/Laissez-faire
2. http://en.wikipedia.org/wiki/Systems_development_life-cycle
3. http://en.wikipedia.org/wiki/Jim_Highsmith
4. http://en.wikipedia.org/wiki/Robert_Cecil_Martin
5. http://en.wikipedia.org/wiki/Ken_Schwaber
6. http://en.wikipedia.org/wiki/Mike_Beedle
7. http://en.wikipedia.org/wiki/Jeff_Sutherland
8. http://wiki.pentaho.com/display/OpenScrum/05.+Scrum+Methodology
9. http://en.wikipedia.org/wiki/Kent_Beck
10. http://www.extremeprogramming.org/
11. http://www.freetutes.com/systemanalysis/sa2-dynamic-system-development-method.html
12. http://weblogs.java.net/blog/chet/archive/2008/01/crystal_methodo.html
13. http://www.projectperfect.com.au/info_lean_development.php
14. http://www.nebulon.com/fdd/
15. http://applicationengineering.blogspot.com/2009/06/kanban-agile.html
16. http://en.wikipedia.org/wiki/Barry_Boehm
17. http://www4.in.tum.de/publ/papers/XP02.Limitations.pdf
18. http://upetd.up.ac.za/thesis/available/etd-02212011-125417/unrestricted/dissertation.pdf