CSC/ECE 517 Fall 2013/oss mapFeeds

Background

The Sahana Software Foundation is dedicated to the mission of saving lives by providing information management solutions that enable organizations and communities to better prepare for and respond to disasters. It is an open source software platform which provides a range of solutions for Disaster Management practitioners to help them reduce the impact disasters have on our communities through tracking the needs of the affected populations & coordinating the responding agencies & their resources.It can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. These features are designed to help Disaster Management practitioners to better mitigate, prepare for, respond to and recover from disasters more effectively and efficiently.

Sahana Eden is designed to be rapidly configured and customized to support the diverse business processes used within Disaster Management. Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts. The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

Core modules and outlines

Organization Registry

Many diverse organizations are involved in Disaster Management, from responding to disasters to strengthening communities to providing support to people in need. Sahana Eden’s Organization Registry can track what organizations are active in different contexts, providing opportunities for collaboration and coordination.

Project Tracking

By telling you Who’s Doing What, Where, and When, Sahana Eden provides a valuable tool to help organizations responding to disasters know where the greatest needs are and coordinate with others who are engaged in similar work.

Human Resources

The most important part of Disaster Management is the people.Whether they are community volunteers or staff working for different organizations, the Human Resources module can help manage the people involved. It will track where they are, what skills they have and help ensure that everyone is effectively engaged with the work that needs to be done. Sahana Eden can also be used to provide a contact list to ensure that the right people can be contacted at the right time.

Inventory

Whether organizations are supplying basic essentials of life to people affected by natural disasters or giving communities the tools they need to restore their livelihoods, Sahana Eden can be used to manage inventories of items and match requests for items with warehouses and other facilities which have them available. Operationally, Sahana Eden can be used to record and automate transactions for sending and receiving shipments. Sahana Eden can support multiple Catalogs of Items as well as providing alternative items to ensure more effective use of supplies. Sahana Eden can be configured to load multiple catalogs including a generic list of items and/or the IFRC Emergency Item Catalog.

Assets

A wide range of assets are needed to respond to disasters, including vehicles to transport people and relief items, radio equipment to provide communication where telecommunication infrastructure has been destroyed, and generators to provide backup power. Sahana Eden is able to manage assets, track where they are, who they have been assigned to, and what condition they are in. This ensures that assets are used effectively and efficiently.

Assessments

Sahana Eden can be used to collect and analyze information from assessments to help organizations more effectively plan their disaster management activities. Different assessment templates can easily be designed and imported into Sahana Eden to support assessments for different organizations in different contexts. Data can either be entered into an interactive web form or imported via an Excel template.

Scenarios and Events

To help organizations better plan for disasters, Sahana Eden can be used to plan for different scenarios, including recording what human resources, assets, facilities and tasks will be needed to effectively respond. When an incident occurs events can be created from a scenario template to allocate the resources and alert people of the need to respond.

Map

Sahana Eden has fully integrated mapping functionality which allows any location-based data to be visualized on a map. This information can also be searched using a map-based boundary selection. Maps provide situational awareness which is essential when either planning to prepare for or respond to a disaster. Sahana Eden supports many standard formats for overlaying data on maps from other sources and Geographical Information Systems (GIS), for example natural hazard risks, population or weather.

Shelter Management

When disasters are widespread and result in population displacement, understanding and tracking the landscape of shelters - and the people in them - is a critical activity. The Shelter Registry provides functionality to list and track information on shelters and on the people arriving and departing. Shelter details include location, services provided, responsible organization and contacts, demographics, and needs. In addition, individual person data includes name, age, relatives, status, health, and many other details to provide a clear understanding of population demographics within the site.

Messaging

In the complex domain of Disaster Management, communication is critical. Sahana Eden provides support for messages to be sent by Email, SMS, Twitter and Google Talk. Distribution Groups can be set up to allow messages to be easily sent to many people at once. Users are able to search for specific information and subscribe to receive update messages when new information is added.

Technical Overview

Because Sahana Eden needs to be accessible to users at remote locations, including the public, a browser-based solution was essential. The system also needs to be able to be used on offline laptops, so it needs to run on a lightweight stack.

Python was selected as a suitable high level language allowing the rapid customization of code required for each individual circumstance yet has a large number of powerful libraries available including for Geospatial Information Systems (GIS).

Sahana Eden Framework

The Sahana Eden Software Platform has been built around a Rapid Application Development (RAD) Framework. This provides a high level of automation to ensure that new solutions can be quickly and effectively developed. Once a database table is defined, the Sahana Eden Framework automatically generates HTML pages to handle CRUD (Create, Read, Update, Delete) as well as Search, Map and Pivot Reports. Web Services are available to import and export in XML, CSV, JSON and EXtensible Stylesheet Language (XSL) transforms are supported to produce other data standards.

Sahana Eden Architecture

Web Server: Apache. Other web servers can also be used,such as Cherokee.

Application: Sahana Eden

Web Application Framework: Web2Py

Programming Language: Python & JavaScript

Database: MySQL,PostgreSQL, or MySQL, PostgreSQL, and SQLite are supported. Other databases should be usable without major additional work since Web2Py supplies many connectors.

Operating System: Linux (Debian recommended),Windows and Mac OS X are possible, but only recommended for single-user environments.

Installation

Sahana Eden can be installed on any environment which can run Python, including Linux, Windows and OSX. The system supports a number of different databases and has been widely tested on MySQL, PostgreSQL and SQLite. A webserver is optional, but for production installations we have experience of both Apache/mod_wsgi and Cherokee/uwsgi.

Installation scripts and detailed instructions are available on the Wiki: http://eden.sahanafoundation.org/wiki/InstallationGuidelines

Web2py

This is an MVC environment (like Rails & Django.). Web2Py can work at several different levels of abstraction. The Sahana Eden framework (S3) is built on Web2Py's Auth/Crud classes in tools.py (with some remnants of the older T2), however sometimes we need more control therefore need to drop down a level or two. Can execute a controller to be able to access its functions using: execfile("applications/eden/controllers/mycontroller.py", globals())

S3

The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD in all the sahana applications. The Sahana api docs can be found at http://eden.sahanafoundation.org/wiki/DeveloperGuidelines

Social Media

Social Media is playing a important role in Disaster and Emergency Management, for such things as:

- Sending warnings and emergency instructions to the public.

- Receiving help requests and incident reports from the public.

- Engaging with the public, encouraging preparedness, providing information.

- Detecting incidents not directly reported. f

Map Feeds

This project allows the user to add layers of Geo-tagged data from Twitter onto the Map. It searches for tweets with specific keywords (that can be configured by user on Sahana web page). It then plots the location of these tweets on the Map. This is the work that we have done as a part of our project implementation. The following steps are involved in this:

After setting up Eden in the system twitterSearch library(https://pypi.python.org/pypi/TwitterSearch/) is to be installed which helps to fetch the tweets with specific keywords

Searching Tweets

- Configuring twitter account:

Sign into https://dev.twitter.com and create an app. (Select an application Type: Client). Put the Callback URL as blank and create your own Twitter application. After creating an app on twitter account, you will be able to see a tab called OAuth tool. Open eden/models/000_config.py file. Under Twitter Settings, add consumer_key and consumer_secret obtained from the twitter app's OAuth tool. Edit the following lines in 000_config.py

    settings.msg.twitter_oauth_consumer_key = "<consumer key from twitter app>"
    settings.msg.twiiter_oauth_consumer_secret = "<consumer_secret from twitter app>"

- Add keywords to be searched:

Go to a page eden/msg/twitter_search. You can add keywords / search queries records under this page. It will ask you for a PIN to enter before saving the keywords/queries. The pin is obtained from Twitter app that was created.

- Populating database with search results:

After saving the keywords/ queries, click on Results which triggers the search and msg_twitter_search_results table in the database gets populated.

Processing Tweets

After fetching the twitter feeds, the Geo-tagged tweets are processed. They are parsed to fetch user information, tweet data and location information in terms of latitude and longitude. All this information then goes into the msg_twitter_search_results table.

Plotting tweets on Map

To display tweets on map, a new layer named "Twitter" is added under the overlays section on the map page. When a user selects this option, the location ids from the msg_twitter_search_results are fetched and a marker is displayed on those locations. Clicking on the marker image displays additional tweet information such as content, time and user info.

Our Tasks

1) After fetching the twitter feeds, process the geo-tagged data in tweets and store in a different table with a foreign key reference in twitter search result table.

2) To display the tweets on a map, for which a new layer “Twitter” is added under the overlays section on map page.

3) To display the twitter search result in “Summary View”. The Summary view provides a consolidate view of the data and provides tabs for Map, Table and Charts.

4) When the tweets are displayed on Map, the data associated with the tweets needs to be displayed.

Steps to Run and Test the Project

Go to application URL : https://sahanaeden.herokuapp.com/eden/default/index. It will bring you to the application`s main page. We need to test the Map feeds functionality for which follow the following steps:

1) Login (A user has been created with email ID : sarangmetkar@gmail.com and password : sarang).

2) Go to URL https://sahanaeden.herokuapp.com/eden/msg/index which is the index page for Messaging feature.

3) A twitter account is configured to fetch the tweets using Twitter API. On the left you will find a heading called Twitter Search. There are two actions you can perform viz. Search Queries and Result.

4) Search Queries - It shows the list of keywords for which you want to search your Twitter account.You can add new search query. For adding new tweets, enter the keyword and select Search after Save check box.

5) Result - It shows a list of tweets consisting of the searched tweets in a table. To view the tweets on Map go to following URL: https://sahanaeden.herokuapp.com/eden/msg/twitter_result/map.

6) You can also view the searched tweets results by clicking on 'Map' menu in the top. URL is : https://sahanaeden.herokuapp.com/eden/gis/index. You can see the different layers, one of which is 'Tweets'. When you select the 'Tweets' option, it shows all the tweets on the Map.

7) For viewing the content in 'Summary View' we have to change the template to 'CRMT' which is done at configuration level. Please refer below to understand more about 'Summary View'.

Summary View

In order to view the twitter search results in Summary View, we need to make configuration level changes. The application running on Heroku has the 'default' theme and the Summary View can be obtained only when the 'CRMT' theme is selected. A screen shot of 'Summary View' has been added below for testing.

Summary View Table

Summary View Map

Future Scope

1. Major limitation is that the twitter search keyword is hard coded.We need to make the code access the msg_twitter_search table and use the keywords stored by the user through the GUI to search and display twitter feeds on the GIS map.

2. Display twitter icon on map instead of the normal location bubble.

3. Work on the pop-up contents. Ideally this would include any photo or video linked in the message (or a URL if it's to a page of text). Actually doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).

4. Create a cron job, to search tweets periodically and not only at user's request.

References

http://sahanafoundation.org/

http://eden.sahanafoundation.org/

http://eden.sahanafoundation.org/wiki/InstallationGuidelines

http://eden.sahanafoundation.org/wiki/S3

http://eden.sahanafoundation.org/wiki/Book

http://en.flossmanuals.net/sahana-eden/why-sahana-eden/

https://pypi.python.org/pypi/TwitterSearch/