Collective Ownership and Continuous Integration

Question:Two of the important Agile methodologies that we have not said much about in this course are collective ownership and continuous integration. What are the best definitions or examples you can find? Are these widely practiced? Are they controversial? Are there adherents of other design methodologies who consider them harmful? Investigate, through the Web and through the ACM DL.

Collective Ownership

What is Collective Ownership?

Collective ownership is extreme programming concept that “code does not belong to a single person; rather it belongs to the project or to the whole team”. This concept is useful in a sense that it overcomes the individual competition and helps to build better team dynamics. Also, as the system grows larger or more complex, it is difficult for one person to be in charge of it. In such situations, collective ownership yields benefits over traditional single-person single-subsystem ownership model.

Definitions

Definition: Collective code ownership means that everybody is responsible for all the code; this, in turn, means that everybody is allowed to change any part of the code. Pair programming contributes to this practice: by working in different pairs, all the programmers get to see all the parts of the code. A major advantage claimed for collective ownership is that it speeds up the development process, because if an error occurs in the code any programmer may fix it.

Source: http://en.wikipedia.org/wiki/Extreme_Programming_Practices#Collective_code_ownership

Comment: This definition introduces concept of collective code ownership in simple terms and highlights the advantage of incorporating it into development practice.

Definition:

ExtremeProgramming considers code to belong to the project, not to an individual engineer. As engineers develop required functionality, they may browse into and modify any class. They are responsible for keeping all the UnitTests running (and writing new ones for new functionality). They take on the same integrity-preserving duties as the class owner in a CodeOwnership situation.

Source: http://c2.com/cgi/wiki?CollectiveCodeOwnership

Comment: Collective code ownership defined in terms of integrity of whole system.

Definition: The team should be evaluated on the progress according to the project plan. Everyone shares in the development, accountability, and ownership of the project. Collective ownership is an integral component of the XP development process. (See figure)

Source: http://www.extremeprogramming.org/rules/collective.html

Comment: Collective code ownership defined as a component of XP (Extreme Programming).

Collective code ownership as part of XP process:

Examples

Real-world scenario and truck factor

Reference: http://www.xpexchange.net/english/intro/collectiveCodeOwnership.html

Often in traditional software projects each subsystem belongs to one engineer. This engineer is then also responsible for the detailed design, the implementation and also for the maintenance of this subsystem. In other words: He is the "owner" of the code; this includes all the related tasks. This seems logical as every developer has a different skill-set which the other developer may not possess. But there are some severe disadvantages of this approach. There is for instance the "truck" factor. Imagine what happens when a team member gets hit by a truck in the middle of the road. If this is the only engineer on a subsystem, then the truck factor for this subsystem is 1: the risk of loosing skills and knowledge in this area is 100%, if that person gets hit.

It is not necessary that the engineer gets hit by a truck. It is more likely, that an engineer is on vacation or on a training, or that he is sick. In all these cases: Who will take care of the sub system the engineer was responsible for?

The Traditional Solution: Backup Engineer

In practice a frequent solution is to assign a backup engineer, that is another engineer from the team, to the sub system. The objective is, that there is more than one person, who has the skills and the knowledge to work on the sub system. However, since this backup engineer works on his system for 100% of his programming time, he hardly gets to learn what the specialist is producing and hence in case of emergency, cannot be as productive with other system as possible

The XP Answer

Distributing skills and knowledge as much as necessary is the best way to reduce the truck factor. XP chooses a pragmatic approach: all members of the project team are collectively responsible for all results of their work, including source code and sub systems.

Pair programming is one example of how this is achieved. This alone is not sufficient, but changing the programming partner in short time frames (several times a day if possible) is a requirement. Though a beginner does not turn into an expert by this approach, as the pairs match several times a day, the skills and knowledge will be fairly evenly distributed within the team within a short time frame. The truck factor is significantly reduced.

The are other XP practices reflecting the collective (code) ownership. The planning game is play within the team. Daily standup meetings serve the synchronization within the team.

From the programmers point of view this has the advantage that he does not have to work all the time on the same sub system. Instead work becomes more interesting if one can work on different sub systems. Generally this also leads to an increase in the versatility of the programmer, as he can acquire skills and knowledge in multiple areas, making it easier when seeking new job opportunities.

Ward Cunningham’s interview on “wiki as a model of collective code/text ownership”

Reference: http://www.artima.com/intv/ownership.html

Ward Cunningham in an interview with Bill Venners explains how he employed collective code ownership during development of first wiki. He explains how the idea of collective ownership can be applied to wikis to collaborate efforts of many people in producing wiki articles. He also compares collective code ownership to collective text ownership.

Penetration of collective code ownership

According to 2nd Annual “The state of Agile Development” Survey was conducted by VersionOne Inc. and sponsored by APLN, results compiled:

Survey statistics: Collective ownership practice is employed by 36% of organizations

Source: http://www.versionone.com/pdf/StateOfAgileDevelopmet2_FullDataReport.pdf

This global survey was held in June and July of 2007 and was completed by almost 1700 people from 71 countries.

Controversy

Reference: http://c2.com/cgi/wiki?ContinuousIntegration

Collective ownership has faced criticism by its opponents’. Some of the claims and arguments are given below:

Claim 1:Having lots of people working on the same class can only lead to chaos, unreliability, and the warm heat death of the universe.

Argument: That turns out not to be the case. You have complete UnitTests to test that class, and all the other classes with which it may collaborate. No code is released unless all the Unit Tests work.

Claim 2: The guy who edits my class may not be as good as I am.

Argument: Partner with him. Both of you will learn something. Besides, what do you mean by "my class"? Most developers think they are better - how is this possible?

Continuous Integration

What is Continuous Integration

Continuous Integration is a powerful agile development practice where integration and build is performed continuously, or as frequently as possible. It encourages developers to integrate their code often to mainstream so that unexpected conflicts do not go undetected for longer periods of time. This concept has originated from extreme programming as one of its twelve original practices and has emerged as a core concept in agile development.

Need for continuous integration

In traditional code development environment, whenever programmer needs to make changes to the system, s/he checks out the relevant files from central version control system, works independently on his checked out version of the code and before submission, runs the new code with latest build available.

However, though s/he may be able to build and run it correctly on his/her own system, some other conflicts may go undetected until central repository code undergoes a build. Such central repository builds are typically once a week in traditional development scenario. So failures persist in system for longer periods of time and are harder to detect and fix. Even if there are no errors, integrated code is available only after a week for release.

One solution to the problem is to have daily builds so that conflicts and failures are detected early and hence can be fixed as soon as they are detected. This maintains the sanity of central repository code. Typically, iterative development projects that make use of prototypes benefit from this approach as daily releases are possible.

Continuous integration is a powerful technique that extends this idea further by having developers integrate and release code into the central code repository every few hours, whenever possible. Only one person or one pair of programmers (in case of pair programming) integrates at a time so that compatibility problems are reduced to a minimum. This solves the problem of “Integration hell” [1] where there are lot of problems integrating code from different developers.

Figure indicating continuous integration process:

Source: http://www.think-box.co.uk/blog/2006/02/ten-minute-build-continuous.html

Definitions of Continuous Integration

There are a lot of definitions, of continuous integration available on the web, that help to understand the concept clearly. We have attempted to mention below the most significant ones below:

Definition:

“Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

Source: http://www.martinfowler.com/articles/continuousIntegration.html

Author: Martin Fowler, renowned author, speaker on software architecture and one of the leaders of agile development movement.

Comment: This is a much quoted, comprehensive definition that introduces continuous integration and explains its advantage over other approaches.

Definition:

Continuous Integration (CI) is the practice of periodically executing Automated Build Scripts (ABS), preferably in short intervals. There are two flavors of this practice. The first flavor, mostly applied by early XP practitioners states that the system is built by one developer at any given time. In this flavor of CI Automated Build Scripts are still required, but kicking off the build happens manually. Since this version relies heavily on the entire team to be co-located, it does not scale well in distributed agile projects.

The second flavor of CI relies on a computer background process to poll the code repository in short intervals and automatically trigger the build process if it detects any changes since the last build. In this version of CI, the developer is only responsible to build locally and check-in the code. This model is better suited for highly scalable agile teams because of its reliance on automated asynchronous processes to trigger the build. This gives distributed teams an early feedback on the status of the system and therefore it enables dislocated team members and managers to have the same quick feedback as the team itself.

Source: http://www.jacoozi.com/blog/?p=18

Comment: This definition explains the ways in which continuous integration can be implemented using automated build scripts to achieve frequent integration.

Definition:

“The macro process of object-oriented development is one of "continuous integration." ... At regular intervals, the process of "continuous integration" yields executable releases that grow in functionality at every release. ... It is through these milestones that management can measure progress and quality, and hence anticipate, identify, and then actively attach risks on an ongoing basis.”

Source: Grady Booch, Object-Oriented Analysis and Design with Applications, 2nd edition.

Comment: This definition was introduced by Grady Booch in the context of Object Oriented analysis and design. It illustrates that Continuous integration is not limited to tools that achieve it but rather it is one of the best software development practices.

Definition:

“Continuous integration allows that team to measure their progress in compile-able working code. It gives the team instant feedback on the health of the build and the state of code repository. Setting up continuous integration also sets the team up for reaping the value of automated testing once they grow enough to include that in the agile arsenal.”

Source: http://www.commandinformation.com/blog/?p=30

Comment: This definition points out how continuous integration can act as a means to achieve fast positive feedback cycles during software development.

Definition:

Continuous integration basically means that all changes to the software are integrated with the rest of the software as soon as the developer submits the change to the central repository. At that point, or at very near intervals, the whole system is run through unit and integration tests to see that it is still healthy. If a defect arises, either through a mistaken submission, or the submission of something that breaks something else, the system alerts the developer, and possibly relevant management.

Source: http://www.agileadvice.com/archives/2005/07/why_should_busi.html

Comment: This definition describes the how continuous integration works in response to changes to software.

Definition:

“Continuous integration is

A central repository for all members of a team, containing:
- the latest code (at least)
- the latest executables
An automated process for building and testing all project assets that:
- can be run many times a day
- is self sufficient

It is best to think of continuous integration as a mindset or statement of intent that allows you to reduce risk by frequently integrating incremental software development changes. Basically, it represents the realization and refinement of a common software development best practice: the daily build and smoke test.”

Source: http://www.ibm.com/developerworks/rational/library/sep05/lee/

Author: Kevin A. Lee, Technical Consultant, IBM

Comment: Continuous integration defined in terms of resources needed for it.

Examples

Real-world scenario: Building small feature using Continuous integration

Reference: http://martinfowler.com/articles/continuousIntegration.html#BuildingAFeatureWithContinuousIntegration

Martin Fowler explains what CI is and how it works by giving an example. It shows the way it is used with the development of a small feature. The example is long but comprehensive in a way that illustrates use of continuous integration in a real-world scenario. The excerpt below is adapted from reference site mentioned above.

Scenario:

Developer begins by taking a copy of the current integrated source onto his/her local development machine. S/he does this by using a source code management system by checking out a working copy from the mainline.

Now s/he takes modifies working copy, adding or changing code and automated tests. Continuous Integration assumes a high degree of tests which are automated into the software. Once done s/he carries out an automated build on his/her development machine. This takes the source code in his/her working copy, compiles and links it into an executable, and runs the automated tests. Only if it all builds and tests without errors then the overall build considered to be good.

With a good build, developer can then think about committing changes into the repository. However, other people may, and usually have, made changes to the mainline before s/he gets chance to commit. So first s/he updates his/her working copy with their changes and rebuild. If their changes clash with her/his changes, it is developer’s responsibility to fix errors if any and repeat until s/he can build a working copy that is properly synchronized with the mainline. Once s/he has made his/her own build of a properly synchronized working copy s/he can then finally commit changes into the mainline, which then updates the repository. However this commit doesn't finish his/her work. At this point s/he builds again, but this time on an integration machine based on the mainline code. Only when committed changes build successfully on the integration is his/her job done. This integration build can be executed manually or could be automated.

If a clash occurs between two developers, it is usually caught when the second developer to commit builds their updated working copy. If not the integration build should fail. Either way the error is detected rapidly. At this point the most important task is to fix it, and get the build working properly again. In a Continuous Integration environment you should never have a failed integration build stay failed for long. A good team should have many correct builds a day. Bad builds do occur from time to time, but should be quickly fixed. The result of doing this is that there is a stable piece of software that works properly and contains few bugs. Everybody develops off that shared stable base and never gets so far away from that base that it takes very long to integrate back with it. Less time is spent trying to find bugs because they show up quickly.

Use of tools in implementation of Continuous integration:

Example tool: CruiseControl

References:

http://www.ibm.com/developerworks/rational/library/sep05/lee/

http://cruisecontrol.sourceforge.net/index.html

The figure below shows infrastructure for a typical CruiseControl implementation

The process that CruiseControl follows is: To monitor the SCM repository for changes; using Rational ClearCase (version control system), it looks for check-ins performed on a branch or stream (usually the project's integration stream). If changes are found, then, on a schedule, CruiseControl builds the application via existing Ant or Maven scripts, runs JUnit Test suite, and reports on the results of build. It can send these results automatically to a set of users via email and/or hold them centrally on a build-results Web site. Optionally, CruiseControl can also deploy an application and carry out any additional scripted task as required

Video presentation on Continuous integration implementation

Source: http://www.integratebutton.com/

In this video, Paul Duvall walks through an example of integrating button using continuous integration technology.

Penetration of Continuous Integration in Industry

According to 2nd Annual “The state of Agile Development” Survey was conducted by VersionOne Inc. and sponsored by APLN, results compiled:

Survey statistics:

Organizations that ---

Currently use continuous integration tools: 57%
Plan to use continuous integration tools: 17%
Would like to use continuous integration tools: 20%
Don’t need continuous integration tools: 6%

50% of organizations already employ continuous integration in agile practices.

This global survey was held in June and July of 2007 and was completed by almost 1700 people from 71 countries.

Link: http://www.versionone.com/pdf/StateOfAgileDevelopmet2_FullDataReport.pdf

This shows that the continuous integration concept is picking up well in the industry.

Controversy over Continuous integration

Opponents’ claims on continuous integration:

Reference: http://c2.com/cgi/wiki?ContinuousIntegration

"Opponent claim: What if you release from your machine and I release from mine and we step on each other's changes?

Proponent argument: It hurts when you do that. Don't do that. Consider having a SingleReleasePoint. Go to it; load your changes, test, and release. This makes releasing single-threaded, which makes it quite simple to detect conflicts. Single release machines are good. Although we allow developers to version and release code that they have tested, we don't update a config map and create a package until the code has been tested on a single machine, integrated with all of the other changes. That way, if all is not well, we can recover by reloading the previous map. The new map is created on the same single machine”. -- Matt Stephenson

Conclusion

Continuous integration and collective ownership originated from extreme programming practice of agile methodology and are gradually being embraced by many firms in software industry. Continuous integration encourages developers to have integrated builds as frequently as possible in a day to provide for high accuracy, low compatibility problems and early integration. Collective ownership, on the other hand, is philosophy that code belongs to project team on a whole and not to an individual. This attitude helps in team-building and alleviate problems arising out of intra-team competition. Though there are some drawbacks to each of these techniques, they are found to be very useful especially in agile development environments and iterative software development projects.