CSC/ECE 517 Spring 2014/ch1a 1p fy

Data migration is the process of transferring data between storage types, formats, or computer systems. It is a key consideration for any system implementation, upgrade, or consolidation. Data migration is usually performed programmatically to achieve an automated migration, freeing up human resources from tedious tasks. Data migration occurs for a variety of reasons, including: Server or storage equipment replacements or upgrades; Website consolidation; Server maintenance; and Data center relocation.<ref>[ http://www.techopedia.com/definition/1180/data-migration/# Janssen C, Data migration, ]</ref>

Data Migration

Categories

Data is stored on various media in files or databases, and is generated and consumed by software applications which in turn support business processes. The need to transfer and convert data can be driven by multiple business requirements and the approach taken to the migration depends on those requirements. This page is mainly focus on data migration within or between databases.

Data Migration in Ruby on Rails

Using Migrations

Migrations are a convenient way to alter your database schema over time in a consistent and easy way. They use a Ruby DSL so that you don't have to write SQL by hand, allowing your schema and changes to be database independent. When creating or updating data in a migration it is often tempting to use one of your models. After all, they exist to provide easy access to the underlying data. Here is an example on how to use Migrations to migrate data in your rails application.

A developer has a User model and a Profile model with the relationship has_one, which means each user has one profile. User table has a column named phone and profile table also has a column named phone. What the developer want to do is moving all the data in the phone column in user table to the phone column in profile table and then delete the phone column in user table.

class DataMigration < ActiveRecord::Migration
 def self.up
   User.find_each do |user|
 user.profile.update_attributes(:phone => user.phone) unless user.phone.blank?
   end
   remove_column :users, :phone
 end

 def self.down
  add_column :users, :phone, :string
 end
end

Using gems

There are several gems that support data migration in ROR, including gem data_migrate

Data migrations are stored in db/data. They act like schema migrations, except they should be reserved for data migrations. For instance, if you realize you need to titleize all yours titles, this is the place to do it. Running any of the provided rake tasks also creates a data schema table to mirror the usual schema migrations table to track all the goodness.

Data migrations can be created at the same time as schema migrations, or independently. Database (db:) tasks have been added and extended to run on data migrations only, or in conjunction with the schema migration. For instance, rake db:migrate:with_data will run both schema and data migrations in the proper order.<ref name="data-migrate gem by Ajvargo">data-migrate gem by Ajvargo</ref>

Advantages

Agile software methodologies offer flexibility, promoting evolutionary development. As the product is developed, customer feedback and testing are incorporated simultaneously. <ref name="Waterfall: Evaluating The Pros And Cons">Agile Vs. Waterfall: Evaluating The Pros And Cons</ref> In this manner, agile software development is especially beneficial when a customer's requirements are changing or unclear. Furthermore, as stated in the agile manifesto, agile methodologies promote teamwork and cohesion. Agile methodologies clearly state their believe that people is the main driver to the projects success, rather than the process and the tool. Agile Methodologies encourage individuals and their interaction through several practices and principles;

• Motivated individuals and Self organized team: Planning Game, Collective ownership.
• Face to Face Communication: Pair programming, Open space, On-site customer, Planning game.
• Sustainable Base.

Disadvantages

The changing nature of agile software development makes things like schedule and budgets somewhat hard to predict. <ref name="Waterfall: Evaluating The Pros And Cons"></ref> While the collaborative nature of agile methodologies can be a benefit, it is also a hindrance, since team members are often involved for the life of the project. This requires designers to still attend to the project's changing specifications after initial design has been completed.

• Requires too much cultural change to adopt.
  
• Go against "Human Psychology": many human factors goes against Agile methodologies, such as: People prefer to follow a process, No milestones, People self-interest, mini-dictatorships, Knowledge monopolies, Resource management vanish, customer dissatisfaction.
  
• The cost efficiency of pair programming is questionable.
  

Prototype Development

The prototype development process can be used to provide a working model of a product or some aspect of the product for testing and feedback. This is often done during the early stages of development, and it provides the customer a means of proofing the product. <ref name="Software Prototyping and Agile Development">Software Prototyping and Agile Development</ref> As the prototype often changes and evolves during the process, it is essentially used to figure out exactly how a component should work.

• Individuals could become singularly responsible for specific areas of the project in individual ownership. This can pose a problem when any one individual leaves the project. The degree to which this may affect the project varies, depending on how well the individual has documented the program and on how much time there is to get somebody else trained before the individual leaves. When everyone has a stake in knowing every piece of code, as in a collective ownership scenario, this danger is less.
• Another problem is that modules that are too insular may become ‘miniprojects’ consisting of just a few developers each. Each miniproject may have its own set of coding styles, designs, politics, feature wars, and preferred technologies. Integrating these miniprojects into one collective, consistent product might prove to be impossible, depending how much they have been allowed to diverge. The problem worsens when the project scales up, particularly when not all the developers will fit into one room. In collective ownership the likelihood of very divergent coding practices amongst different groups is less likely.
• When only specific individuals are familiar with certain portions of the codebase, others on the team can be held up waiting for changes to be made. In addition, the amount of work needed for different sections of a project can change during the course of development. If individual ownership of each section is promoted too much, work loads can become lopsided

Advantages

Software prototyping essentially provides a means for agile software development because it opens a discussion about what customers want. Customers can use the prototype, see what they like and dislike, and this reduces the cost of making the change later in the project. As customers and software developers often have different vocabularies and interpretations of the requirements, a prototype allows customers and other users to provide more meaningful feedback.

Disadvantages

Because prototypes often do not depict the entire project, their use can lead to misinterpretation. The limited prototype can distract users from the big picture, which can lead to overlooking other solutions. In the case of customers, they may see a limited prototype as an insufficient substitute for the product as a whole, and they may not provide the beneficial feedback the prototype was intended for. Another problem is that too much time, effort, and money can be invested in a prototype that is intended to be thrown away. <ref name="Software Prototyping Wiki">Software Prototyping Wiki</ref>

An Empirical Study in Agile Development

The paper Empirical studies of agile software development: A systematic review<ref name="Empirical studies of agile software development: A systematic review">Empirical studies of agile software development: A systematic review</ref>, published in 2008, sought to determine how exactly agile software methodologies were being used in the real world. As agile was not a widely known concept until 2001, they analyzed published findings concerning agile software development between 2001 and 2005. During that time, a survey of American and European countries showed that only 14% of companies were using agile methodologies, while 49% were interesting in adopting them. During an analysis of some 36 papers, it was uncovered that customers often praised agile methodologies for letting them feel that they had complete control over the development process. In addition, customers reported that daily meetings kept them up to date on the product's status, and developers were often more clear about what their development objectives were. However, in some cases, the collaborative nature of agile methodologies proved difficult in outsourced projects, as some customers were required to become acclimatizes the working processes of outside developers.

From a developer's standpoint, many papers reported that developers felt more comfortable using an agile methodology like XP or paired programming. In particular, many developers reported that paired programming seemed to make the process quicker; however, they also reported that this process seemed more tiring, as it required intense concentration. Furthermore, Scrums were favored by developers, and they even led to a reduction in overtime in certain cases.

As for productivity, for the 36 papers studied, 42% reported an increase in productivity over traditional development methods. The majority of the productivity gain was often seen during the first iteration of the software. In addition, many of the papers reported reduced errors and better overall quality for products developed using agile methodologies.

People at Microsoft research lab have done extensive research on the application of agile development on large scale projects.According to the Andrew Begel and Nachiappan Nagappan dominant agile methodologies used are Scrum,XP and Crystal with in Microsoft. According to the survey, the top benefit of following agile development was improved communication and coordination among team members, quick releases and flexibility of design-quicker response to changes. Some of the downside of the agile development were coordination with other teams, lose sight of big picture and dev/test integration.

Development in waterfall and agile methodology

Acoording to The Chaos Manifesto, the graph below shows the specific results reported from a study conducted based on projects executed from 2002 to 2012. In 2002, agile projects made up less than 2% of over all projects and less than 5% of new application development projects. Today, agile projects account for almost 9% of all projects and 29% of new application development projects. The increase in project success rates can directly tie back to projects resolved through agile process.

When it comes down to which model is better, neither the agile method, prototype or waterfall is inherently better than the other. Each method does have its uses. Waterfall tends to be best for static projects, where it's not likely that many changes will be made throughout the development process. Agile methodology would be a better option for smaller projects where changes are likely to made during the development process. One can consider taking best of all i.e., taking best aspects of waterfall, prototype and agile combining them in order to make the best possible software development process.

Summary

The following table gives a higher level view of the variations between traditional (waterfall), prototype and agile development.

Traditional (Waterfall)	Prototype	Agile
Complete solution	Prototype versions	Functional modules
Linear development process	Milestones and integration	Short iterations
Lock-down change	Calibration , prototyping and integration	Experimentation, improvement and re-prioritization
Users specify all requirements at the start	User requests are prototyped and later intergrated	User requests embedded throughout the process

The features, advantages and disadvantages of the three models are tabulated as follows:

Traditional (Waterfall)	Prototype	Agile
Structured and sequential	Incremental approach with “milestones”	Iterative and incremental
Process-oriented	Progress-oriented	People-Oriented
Command-Control culture	Prototyping culture	Collaboration culture
Heavy documentation	Moderate documentation as versions	Low documentation
Scope/requirement decided upfront	Scope/requirement versioned and integrated	Scope/requirement easy to adapt and change
Change demands high cost	Change demands considerable cost	Change demands least cost
Less re-work due to static specification	Moderate work in integrating the versions.	More re-work due to dynamic specification

See also

• Survey of Agile Development Methodologies
• Waterfall Model Waterfall
• Waterfall modules
• Extreme Programming
• Crystal
• Scrum
• Choosing Your Software Development Process: Agile or Plan-driven?
• Taken from Empirical Findings in Agile Methods

References

<references />