Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2015 M1502 WSRA

2015-05-04T23:14:26Z

Achen4: /* Implementation */

==Implement Rust Websocket==
This project concentrates on implementing Rust WebSocket API for Mozilla's web browser engine, Servo

==Introduction==
Servo<ref> https://github.com/servo/servo </ref> is a Web Browser engine written in [https://github.com/rust-lang/rust Rust]. It is an experimental project build that targets new generation of hardware: mobile devices, multi-core processors and high-performance GPUs to obtain power efficiency and maximum parallelism. Implementing WebSocket API for the Servo engine would allow a persistent single TCP socket connection to be established between the client and server that will provide bi-directional, full duplex, messages to be instantly exchanged with little overhead resulting in a very low latency connection and supporting interactive, dynamic applications.

===Rust===
Rust <ref> https://github.com/rust-lang/rust </ref> is a Systems programming language built in Rust itself that is fast, memory safe and multithreaded, but does not employ a garbage collector or otherwise impose significant runtime overhead. Rust is able to provide both control over hardware and safety which is not the case with other programming languages like C, C++, Python that provide only either control or safety but not both.

===WebSocket===
WebSockets is a protocol that provides [http://en.wikipedia.org/wiki/Duplex_(telecommunications)#FULL-DUPLEX full-duplex] channel for a TCP connection and makes it possible to open an interactive communication session between the user's browser and a server. With WebSockets, you can send messages to a server and receive event-driven responses without having to request the server for a reply. The [http://dev.w3.org/html5/websockets/ WebSocket] specification defines an API to establish a "socket" connection between a web browser and a server. This establishment involves a handshake following which there is a persistent connection between the client and the server and both parties can start sending data asynchronously.

===Cargo and Crate===
Cargo <ref>http://doc.crates.io/guide.html</ref> is a application level package manager that allows Rust projects to declare their various dependencies. Cargo resembles the Bundler in Rails that is used to run Rails app, install required Gems mentioned in the Gemfile. Gemfile correspond to <code> Cargo.toml </code> file and Gem correspond to <code> Crates </code> 

Cargo introduces two metadata files with various bits of project information, fetches and builds project's dependencies, invokes rustc or another build tool with the correct parameters to build the project.

==Project Description==

The goal of this project is to implement WebSocket functionality in to Servo using the Rust WebSocket project. Once this is completed, tests following the HTML spec [https://html.spec.whatwg.org/multipage/comms.html#network] for websockets should pass, as the same spec will be driving development. The project will be completed when the following conditions are true, and all tests have passed.
* The client can establish a websocket connection
* The client can send objects to the server through the websocket
* The client can receive objects from the server through the websocket
* The client can successfully close the Websocket

If Servo, as the client, is able to successfully perform all of these tasks, then our project will be completed.

==Requirement Analysis==
The requirements for the implementation is well detailed in the HTML spec shown here [https://html.spec.whatwg.org/multipage/comms.html#feedback-from-the-protocol]. The majority of the requirements have to do with the feedback for the protocol. The entire reading can be seen there.

Outside of the HTML spec, a couple other requirements are necessary. First, Servo must successfully compile with the websocket components included. There are a number of warnings currently with compiling Servo not related to Websocket, but no new ones should be introduced.

Second, we should pass a set of tests already included in Servo's tests/wpt/web-platform-tests/websockets directory. These tests determine if the current websocket implementation fulfills all the requirements for Servo's use.

Finally, we should successfully issue pull requests for each feature as it is completed.

==Implementation==
The full implementation specs are outlined here: https://github.com/servo/servo/wiki/WebSocket-student-project

In general the websocket has 3 functions:
*Open() - Initiates and connects the websocket to the server. Once the connection is open, hands receives.
*Send() - Sends application data to the server
*Close() - Closes the websocket connection

In addition the websocket has the following attributes:
*readyState - indicates the current state of the websocket. The websocket can be in one of the five states below:
**Connecting - an initial connection handshake has been started but the server has not responded to the handshake yet
**Open - the connection has been established (server has responded to the initial connection handshake)
**Closing - an initial close connection handshake has been started but the server has not responded to the handshake yet
**Closed - the connection has been closed (server has acknowledged the closing handshake and responded)
*Extensions - Extensions are being used by the websocket (see full specs link above)
*Protocol - Protocols being used by the websocket (see full specs link above)
*bufferedAmount - returns how many bytes of application data has been buffered in the websocket thus far (unsent data)

==Design Patterns==
[http://en.wikipedia.org/wiki/Factory_%28object-oriented_programming%29 Factory Pattern] 
For every web application application running on the web browser that requires a websocket for communicating with the server, a new thread needs to be spawned that creates a client end of the socket associated with the web app. Spawning this thread can be considered to be similar to instantiating an object for which Factory Pattern can be employed. 
[http://en.wikipedia.org/wiki/Thread_pool_pattern Thread Pool Pattern] 
Servo's primary objective was to provide concurrency. This concurrency can be attained by scheduling tasks which can be run in parallel. Threads help execute these tasks in parallel. Thread Pool Pattern helps manage these thread. For example, one thread could be to accept new requests for websockets, as a result of which new thread would be spawned (client side of websocket) for that application.

==Proposed Test Cases==

The test cases for websocket are already included in servo, under the Servo directory tests/wpt/web-platform-tests/websockets. To break down how each should be tested, here are the proposed scenarios:
* The send function should send some object, then wait for confirmation of sending.
* The constructor should make a websocket connection to the URL and either open the connection or close it based on the response coming in.
* The receive function should trigger on receiving an object, then send confirmation.

All of these are tested thoroughly in the directory given at the beginning of this section.

==Further Readings==
[https://html.spec.whatwg.org/multipage/comms.html#network HTML Spec for WebSockets] 
[https://github.com/servo/servo/wiki/WebSocket-student-project Mozilla Websocket Project Page] 
[https://github.com/servo/servo Servo github] 
[http://cyderize.github.io/rust-websocket/ Rust Websocket] 

==References==
<references/>

CSC/ECE 517 Spring 2015 M1502 WSRA

2015-04-01T23:57:00Z

Achen4: /* Implementation */

==Implement Rust Websocket==
This project concentrates on implementing Rust WebSocket API for Mozilla's web browser engine, Servo

==Introduction==
Servo<ref> https://github.com/servo/servo </ref> is a Web Browser engine written in [https://github.com/rust-lang/rust Rust]. It is an experimental project build that targets new generation of hardware: mobile devices, multi-core processors and high-performance GPUs to obtain power efficiency and maximum parallelism. Implementing WebSocket API for the Servo engine would allow a persistent single TCP socket connection to be established between the client and server that will provide bi-directional, full duplex, messages to be instantly exchanged with little overhead resulting in a very low latency connection and supporting interactive, dynamic applications.

===Rust===
Rust <ref> https://github.com/rust-lang/rust </ref> is a Systems programming language built in Rust itself that is fast, memory safe and multithreaded, but does not employ a garbage collector or otherwise impose significant runtime overhead. Rust is able to provide both control over hardware and safety which is not the case with other programming languages like C, C++, Python that provide only either control or safety but not both.

===WebSocket===
WebSockets is a protocol that provides [http://en.wikipedia.org/wiki/Duplex_(telecommunications)#FULL-DUPLEX full-duplex] channel for a TCP connection and makes it possible to open an interactive communication session between the user's browser and a server. With WebSockets, you can send messages to a server and receive event-driven responses without having to request the server for a reply. The [http://dev.w3.org/html5/websockets/ WebSocket] specification defines an API to establish a "socket" connection between a web browser and a server. This establishment involves a handshake following which there is a persistent connection between the client and the server and both parties can start sending data asynchronously.

===Cargo and Crate===
Cargo <ref>http://doc.crates.io/guide.html</ref> is a application level package manager that allows Rust projects to declare their various dependencies. Cargo resembles the Bundler in Rails that is used to run Rails app, install required Gems mentioned in the Gemfile. Gemfile correspond to <code> Cargo.toml </code> file and Gem correspond to <code> Crates </code> 

Cargo introduces two metadata files with various bits of project information, fetches and builds project's dependencies, invokes rustc or another build tool with the correct parameters to build the project.

==Project Description==
==Requirement Analysis==
==Implementation==
'''When the WebSocket connection is established, the user agent must queue a task to run these steps:'''
*If the WebSocket object's client-specified protocols was not an empty list, but the subprotocol in use is the null value, then fail the WebSocket connection, set the readyState attribute's value to CLOSING, and abort these steps.
*Change the readyState attribute's value to OPEN.
*Change the extensions attribute's value to the extensions in use, if is not the null value.
*Change the protocol attribute's value to the subprotocol in use, if is not the null value.
*Act as if the user agent had received a set-cookie-string consisting of the cookies set during the server's opening handshake, for the URL url given to the WebSocket() constructor.
*Fire a simple event named open at the WebSocket object.

'''When a WebSocket message has been received with type type and data data, the user agent must queue a task to follow these steps:'''
*If the readyState attribute's value is not OPEN, then abort these steps.
*Let event be a newly created trusted event that uses the MessageEvent interface, with the event type message, which does not bubble, is not cancelable, and has no default action.
*Initialise event's origin attribute to the Unicode serialisation of the origin of the URL that was passed to the WebSocket object's constructor.
*If type indicates that the data is Text, then initialise event's data attribute to data.
*If type indicates that the data is Binary, and binaryType is set to "blob", then initialise event's data attribute to a new Blob object that represents data as its raw data.
*If type indicates that the data is Binary, and binaryType is set to "arraybuffer", then initialise event's data attribute to a new ArrayBuffer object whose contents are data.
*Dispatch event at the WebSocket object.

==Architecture==
==Component Design==
==Data Design==
==Design Patterns==
[http://en.wikipedia.org/wiki/Factory_%28object-oriented_programming%29 Factory Pattern] 
For every web application application running on the web browser that requires a websocket for communicating with the server, a new thread needs to be spawned that creates a client end of the socket associated with the web app. Spawning this thread can be considered to be similar to instantiating an object for which Factory Pattern can be employed. 
[http://en.wikipedia.org/wiki/Thread_pool_pattern Thread Pool Pattern] 
Servo's primary objective was to provide concurrency. This concurrency can be attained by scheduling tasks which can be run in parallel. Threads help execute these tasks in parallel. Thread Pool Pattern helps manage these thread. For example, one thread could be to accept new requests for websockets, as a result of which new thread would be spawned (client side of websocket) for that application.

==Proposed Test Cases==
==Further Readings==
==References==
<references/>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-23T21:38:27Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1077/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields in a hash and append them to the results variable

<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

===JSONP Formatted Response===
The following code shows the format of the final response. The return value is a hash that contains 2 fields:
*gislocations - a hash of the search results. The fields of this hash must be defined in the GeoNamesSearchCombo.js (see lines 252-268 of the pull request for an example).
*totalResultsCount - a integer count of how many results are in the gislocations hash

Also note that unlike JSON, JSONP requires the return value to be wrapped inside a callback function.

<pre>
returnVal = {}
returnVal["gislocations"] = results
returnVal["totalResultsCount"] = count

#Autocomplete caller expects JSONP response. Callback wrapper.
return callback_func+'('+json.dumps(returnVal)+')'
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T23:10:16Z

Achen4: /* JSONP Formatted Response */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields in a hash and append them to the results variable

<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

===JSONP Formatted Response===
The following code shows the format of the final response. The return value is a hash that contains 2 fields:
*gislocations - a hash of the search results. The fields of this hash must be defined in the GeoNamesSearchCombo.js (see lines 252-268 of the pull request for an example).
*totalResultsCount - a integer count of how many results are in the gislocations hash

Also note that unlike JSON, JSONP requires the return value to be wrapped inside a callback function.

<pre>
returnVal = {}
returnVal["gislocations"] = results
returnVal["totalResultsCount"] = count

#Autocomplete caller expects JSONP response. Callback wrapper.
return callback_func+'('+json.dumps(returnVal)+')'
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T23:03:49Z

Achen4: /* JSONP Formatted Response */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields in a hash and append them to the results variable

<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

===JSONP Formatted Response===

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T23:03:39Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields in a hash and append them to the results variable

<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

==JSONP Formatted Response==

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T23:02:59Z

Achen4: /* Searching the Internal database */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields in a hash and append them to the results variable

<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T23:00:21Z

Achen4: /* Searching the Internal database */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
'''The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.'''

This code gets the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next the code performs the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

To parallel the level codes returned from geonames.org, the internal database translates the level field from the gis_location table into equivalent codes from geonames.org. See http://www.geonames.org/export/codes.html for a complete list of feature codes.
<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"
</pre>

Lastly, we gather the id, fcode, name, lat, and lng fields and add them to results
<pre>
result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:56:06Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search.
The implementation of the search_gis_locations() function can be broken down into two sections: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:53:48Z

Achen4: /* Goals of the project */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
*Extend upon the Geonames searchCombo feature embedded in Sahana map module
*Search the internal gis_locations table for a partial match of the entered string in the search field
*Partial matches should populate in the autocomplete drop down
*When user selects a result from the drop down, zoom to that location
*Default to geonames.org when internal search returns no results

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:51:27Z

Achen4: /* Goals of the project */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
The focus of this project is to extend upon the Geonames searchCombo feature embedded in Sahana map module. Specifically the location linked to the search term in gis_location table should be zoomed to and if the search in internal database returns no results, the search redirects to Geonames.org

==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:48:26Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:48:11Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
==Implementation==
The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:47:40Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
==Implementation==
'''Notes:'''
*Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
*For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T22:46:35Z

Achen4: /* Implementation */

==About Sahana==
[http://eden.sahanafoundation.org/ Sahana Eden] is an Open Source Humanitarian Platform which can be used to provide solutions for Disaster Management, Development, and Environmental Management sectors. Being open source, it is easily customisable, extensible and free. It is supported by the [http://sahanafoundation.org/ Sahana Software Foundation]. Sahana Eden was first developed in Sri Lanka as a response to the Indian Ocean Tsunami in 2005. The code for the Sahana Eden project is hosted at [https://github.com/flavour/eden Github] and it is published under the [http://en.wikipedia.org/wiki/MIT_License MIT License]. The demo version of Sahana Eden can be found [http://demo.eden.sahanafoundation.org/eden/ here].

Eden is a flexible humanitarian platform with a rich feature set which can be rapidly customized to adapt to existing processes and integrate with existing systems to provide effective solutions for critical humanitarian needs management either prior to or during a crisis. Sahana Eden contains a number of different modules like 'Organization Registry', 'Project Tracking', 'Human Resources', 'Inventory', 'Assets', 'Assessments', 'Scenarios & Events', 'Mapping', 'Messaging' which can be configured to provide a wide range of functionality. We are contributing to Sahana Eden as a part of our Object-Oriented Design and Development's Open-Source Software (OSS) Project. In this Wiki Page, we would be explaining the goals of our project and how we implemented them.

==Goals of the project==
==Implementation==
Any groups wishing to extend, understand, or review the zoom to location feature enhancement should see this github pull request: https://github.com/flavour/eden/pull/1075/files
For tracking purposes, the dialog for this project can be found on this google groups thread: https://groups.google.com/forum/#!topic/sahana-eden/vS54iJEDqvQ

The implementation of the zoom to location feature is contained entirely in the search_gis_locations() function in controllers/gis.py and static/scripts/gis/GeoExt/ux/GeoNamesSearchCombo.js. The only significant change to the GeoNamesSearchCombo.js was the url option which makes the search box call the search_gis_locations() method instead of querying the geonames.org website for a search location (see lines 220-221 on the pull request page).

The search_gis_locations() function primarily uses an adapter design. It allows the GeoNamesSearchCombo.js to query the internal database and geonames.org for data during a user search. The implementation of the search_gis_locations() function can be broken down into two section: Searching the Internal database, and Searching Geonames.

===Searching the Internal database===
The following code searches the 'name' column in the gis_location database for items "starting with" the string entered in the search field of the map page on Eden.

Here we are getting the parameters from the url. The user string is stored in a parameter called "name_startsWith". Because eden expects a JSONP response, a callback function is needed. The name of the call back function is stored in the "callback_func" parameter of the url. We also select the "gis_location" table to get ready for the query.

<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
</pre>

Next we perform the query on the "name" column of the gis_location table. Note that the "%" is a wildcard and allows us to do a partial match with the string the user entered into the search field. The rows variable selects the fields of our interest. For the search, we only care about the entry id, zoom level (aka. level), name, latitude, and longitude.

We also count and store the number of results from the search since it's a required field for the search box.
<pre>
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}
</pre>

<pre>
#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching Geonames===

If the initial search for the user query on the internal 'Locations' table, then a search is done on [http://www.geonames.org/ Geonames] as a fallback. This fallback search is implemented within the 'gis' controller using 'urllib2' an extensible Python module for opening URLs. The base URL for the Geonames search is http://ws.geonames.org/searchJSON?. The searchJSON action expects the Geonames username, the prefix of the location names to be searched, number of rows in the JSON result, etc. as parameters. The following code performs the HTTP GET request and loads the JSON response in a dictionary.

<pre>
username = settings.get_gis_geonames_username()
maxrows = "20"
lang = "en"
charset = "UTF8"
nameStartsWith = user_str
geonames_base_url = "http://ws.geonames.org/searchJSON?"
url = "%susername=%s&maxRows=%s&lang=%s&charset=%s&name_startsWith=%s" % (geonames_base_url,username,maxrows,lang,charset,nameStartsWith)
response = urllib2.urlopen(url)
dictResponse = json.loads(response.read().decode(response.info().getparam('charset') or 'utf-8'))
response.close()
</pre>

The JSON object in the response has two keys 'totalResultsCount' and 'geonames'. The object corresponding to the 'geonames' key is an array of the Geonames search results, each of which is a dictionary in itself. Of all the Keys present in a single result dictionary, the relevant ones are 'id', 'fcode', 'name', 'lat' and 'lng'. The relevant keys are extracted and a new dictionary is created for each search result. The array of new dictionaries is then returned as a response. The following code performs the decoding of the JSON object, creating new dictionary objects with the relevant keys and then collecting them as a single array.

<pre>
results = []
if dictResponse["totalResultsCount"] != 0:
geonamesResults = dictResponse["geonames"]
for geonamesResult in geonamesResults:
result = {}
result = {"id" : int(geonamesResult["geonameId"]), "fcode" : str(geonamesResult["fcode"]),
"name" : str(geonamesResult["name"]),"lat" : float(geonamesResult["lat"]),
"lng" : float(geonamesResult["lng"])}
results.append(result)
</pre>

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T21:25:45Z

Achen4: /* Searching Internal database */

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T19:32:42Z

Achen4: /* Searching Internal database */

==About Sahana==
==Goals of the project==
==Implementation==
===Searching Internal database===
<pre>
#Get vars from url
user_str = get_vars["name_startsWith"]
callback_func = request.vars["callback"]
atable = db.gis_location
query = atable.name.lower().like(user_str + '%')
rows = db(query).select(atable.id,
atable.level,
atable.name,
atable.lat,
atable.lon
)
results = []
count = 0
for row in rows:
count += 1
result = {}

#Convert the level colum into the ADM codes geonames returns
#fcode = row["gis_location.level"]
level = row["gis_location.level"]
if level=="L0": #Country
fcode = "PCL" #Zoom 5
elif level=="L1": #State/Province
fcode = "ADM1"
elif level=="L2": #County/District
fcode = "ADM2"
elif level=="L3": #Village/Suburb
fcode = "ADM3"
else: #City/Town/Village
fcode = "ADM4"

result = {"id" : row["gis_location.id"],
"fcode" : fcode,
"name" : row["gis_location.name"],
"lat" : row["gis_location.lat"],
"lng" : row["gis_location.lon"]}
results.append(result)
</pre>

===Searching geonames===

CSC/ECE 517 Spring 2015/oss S1504 AAC

2015-03-21T19:17:37Z

Achen4: Created page with "==About Sahana== ==Goals of the project== ==Implementation== ===Searching Internal database=== ===Searching geonames==="

==About Sahana==
==Goals of the project==
==Implementation==
===Searching Internal database===
===Searching geonames===

CSC/ECE 517 Spring 2015

2015-03-21T19:16:23Z

Achen4: /* Writing Assignment 2 */

==Writing Assignment 1==
*[[CSC/ECE 517 Spring 2015/ch1a 17 WL]]
*[[CSC/ECE 517 Spring 2015/ch1a 5 ZX]]
*[[CSC/ECE 517 Spring 2015/ch1a 6 TZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 4 RW]]
*[[CSC/ECE 517 Spring 2015/ch1a 7 SA]]
*[[CSC/ECE 517 Spring 2015/ch1a 9 RA]]
*[[CSC/ECE 517 Spring 2015/ch1a 14 RI]]
*[[CSC/ECE 517 Spring 2015/ch1a 1 DZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 20 HA]]
*[[CSC/ECE 517 Spring 2015/ch1a 3 RF]]
*[[CSC/ECE 517 Spring 2015/ch1a 12 LS]]
*[[CSC/ECE 517 Spring 2015/ch1a 13 MA]]
*[[CSC/ECE 517 Spring 2015/ch1a 2 WA]]
*[[CSC/ECE 517 Spring 2015/ch1b 21 QW]]
*[[CSC/ECE 517 Spring 2015/ch1b 23 MS]]
*[[CSC/ECE 517 Spring 2015/ch1b 10 GL]]
*[[CSC/ECE 517 Spring 2015/ch1b 27 VC]]
*[[CSC/ECE 517 Spring 2015/ch1b 22 SF]]
*[[CSC/ECE 517 Spring 2015/ch1b 15 SH]]
*[[CSC/ECE 517 Spring 2015/ch1b 18 AS]]

==Writing Assignment 2==
*[[CSC/ECE 517 Fall 2014/oss E1502 wwj]]
*[[CSC/ECE 517 Fall 2014/oss E1508 MRS]]
*[[CSC/ECE 517 Spring 2015/oss E1504 IMV]]
*[[CSC/ECE 517 Spring 2015/oss E1505 xzl]]
*[[CSC/ECE 517 Spring 2015/oss E1509 lds]]
*[[CSC/ECE 517 Spring 2015/oss E1510 FLP]]
*[[CSC/ECE 517 Spring 2015/oss E1506 SYZ]]
*[[CSC/ECE 517 Spring 2015/oss S1504 AAC]]

CSC/ECE 517 Spring 2015

2015-03-21T19:15:30Z

Achen4:

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T03:01:57Z

Achen4:

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>. While Amazon is a fast paced, growing cloud storage service, there exists many alternatives to Amazon S3. A brief overview of the cost to performance comparison is shown below.

==Alternatives to S3==
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png |frame |center|Source: http://www.cloudberrylab.com/blog/amazon-s3-azure-and-google-cloud-prices-compare/]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref>[https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/ Barb Darrow. Amazon Web Services tops list of most reliable public clouds ]</ref>

[[File:ServiceCompare.png |frame |center|Source: https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/]]

==Design==

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

===Objects===

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

===Buckets===

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

===Keys and Metadata===

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

===Regions===

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

===Access Permissions===

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

===Data Protection===

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

==Ruby and Amazon S3==

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the [http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release| AWS Ruby Development Blog].

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:55:34Z

Achen4: /* Background */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>. While Amazon is a fast paced, growing cloud storage service, there exists many alternatives to Amazon S3. A brief overview of the cost to performance comparison is shown below.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png |frame |center|Source: http://www.cloudberrylab.com/blog/amazon-s3-azure-and-google-cloud-prices-compare/]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref>[https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/ Barb Darrow. Amazon Web Services tops list of most reliable public clouds ]</ref>

[[File:ServiceCompare.png |frame |center|Source: https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/]]

==Design==

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

===Objects===

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

===Buckets===

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

===Keys and Metadata===

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

===Regions===

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

===Access Permissions===

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

===Data Protection===

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

==Ruby and Amazon S3==

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the [http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release| AWS Ruby Development Blog].

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:51:17Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png |frame |center|Source: http://www.cloudberrylab.com/blog/amazon-s3-azure-and-google-cloud-prices-compare/]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref>[https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/ Barb Darrow. Amazon Web Services tops list of most reliable public clouds ]</ref>

[[File:ServiceCompare.png |frame |center|Source: https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:48:42Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png |Source: http://www.cloudberrylab.com/blog/amazon-s3-azure-and-google-cloud-prices-compare/]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref>[https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/ Barb Darrow. Amazon Web Services tops list of most reliable public clouds ]</ref>

[[File:ServiceCompare.png |Source: https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:45:03Z

Achen4:

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref>[https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/ Barb Darrow. Amazon Web Services tops list of most reliable public clouds ]</ref>

[[File:ServiceCompare.png]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:43:16Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref> Barb Darrow. Amazon Web Services tops list of most reliable public clouds https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/</ref>

[[File:ServiceCompare.png]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:42:23Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours. <ref> Barb Darrow. Amazon Web Services tops list of most reliable public clouds [https://gigaom.com/2015/01/07/amazon-web-services-tops-list-of-most-reliable-public-clouds/]</ref>

[[File:ServiceCompare.png]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:40:55Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists alternatives to AmazonS3:
:*[https://cloud.google.com/storage/| Google Cloud Storage]
:*[http://azure.microsoft.com/en-us/ |MSAzure]

As with any online cloud storage service, the cost of storage is always a concern. The following table shows a cost comparison between them:

[[File:PriceCompare.png]]

While all three services are fairly comparable in price, there are some developer support considerations for selecting a service. For instance, if an app is focused around the [https://www.android.com/| Android OS], using the google cloud storage provides you with more tightly coupled APIs for your app. The overhead of development may be much less compared to using S3 to rails APIs for a Google app. Similarly, [http://www.windowsphone.com/en-us/features| Windows Phone OS] would provide more native support for Windows Phone apps and likewise kindle fire apps would have more native support on [https://developer.amazon.com/public/apis| Fire OS].

The reliability of a cloud storage service like S3 plays a heavy role in the selection process. Less downtime translates to a more robust and reliable app. In 2014, Amazon S3 registered only 23 outages totaling 2.69 hours of downtime whereas Google registered 8 outages totaling 14.23 hours of downtime. Microsoft Azure totaled 141 outages summing 10.97 hours.

[[File:ServiceCompare.png]]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

File:ServiceCompare.png

2015-02-14T02:40:16Z

Achen4:

File:PriceCompare.png

2015-02-14T02:17:46Z

Achen4: uploaded a new version of "File:PriceCompare.png"

File:PriceCompare.png

2015-02-14T02:15:27Z

Achen4:

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:08:49Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===
There currently exists many alternatives to AmazonS3:
**[https://cloud.google.com/storage/| Google Cloud Storage]
**[http://www.rackspace.com/cloud/files| Rackspace Cloud Storage]
**[http://azure.microsoft.com/en-us/ |MSAzure]

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:05:09Z

Achen4: /* Alternatives to S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Alternatives to S3===

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T02:04:55Z

Achen4: /* Background */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

==Alternatives to S3==

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:57:34Z

Achen4: /* Upload a file to Amazon S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following license <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:57:08Z

Achen4: /* Upload a file to Amazon S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
This example is a combination of many examples shown above. A bucket is created and filled with a local file. The program also generates a url for the upload and allows the option of deleting the local copy of the object at the user's discretion.
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:47:18Z

Achen4: /* Download an object to a folder on S3 */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket my-new-bucket.
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:47:04Z

Achen4: /* Creating an object */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file hello.txt with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket "my-new-bucket".
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:46:40Z

Achen4: /* Download an object to a folder */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder on S3===
An app may choose to retrieve objects from a server and save these to the S3 server, where they can be accessed later. This example shows how to download the poetry.pdf object to the bucket "my-new-bucket".
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:43:02Z

Achen4: /* Generating object download urls */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. In the example, the expiration of secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:42:27Z

Achen4: /* Generating object download urls */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
Generating download links for objects on the S3 server can make it easier for you to share your objects with other users. This example shows how to create a download link for the hello.txt object we created earlier.
Note that generating a link requires you to specify the bucket it resides in. Links can be public, as in the case of hello.txt below, or be set to expire on a timer through the expires_in symbol. Note that the expiration of the secret_plans.txt is 1 hour (60s * 60).
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:34:57Z

Achen4: /* Examples */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:33:24Z

Achen4: /* Examples */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
This example shows you how to delete the object "goodbye.txt". You must specify the bucket as the second parameter of the S3Object.delete function.
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:27:20Z

Achen4: /* Change an object's ACL (access control list) */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
Securing user data from view by any web user is important if you're keeping sensitive information about users. This example shows you how to open the hello.txt object created earlier in my-new-bucket to the public (anyone can access it). This example also shows how to restrict the secret_plans.txt so that no one can access them.
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:21:43Z

Achen4: /* Creating an object */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
Object creation is important to any program. If your app stores data via objects, you can use the S3 server to host them. In this example we create a text file "hello.txt" with content "Hello World!" and save it to my-new-bucket. Note that the :content_type is required for the ruby method S3Object.store.
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:19:35Z

Achen4: /* Forced removal of non-empty buckets */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
If you want to forcibly remove a bucket and dump all of its contents, you can force a deletion as shown below.
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:18:36Z

Achen4: /* Deleting a bucket */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
Bucket removal may be necessary when you're trying to reduce the cost of maintaining your data on the S3 servers. This code allows you to delete buckets but only if they're empty.
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:17:19Z

Achen4: /* Listing a bucket's contents */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
For a known bucket (see example for listing all buckets you own), you can also query the server for all objects inside. This code shows you how.
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
:'''Note:''''' The target bucket must be empty!'' 
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:15:49Z

Achen4: /* Listing all buckets you own */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
Buckets hold data about the objects you upload. To access any object, you must access the bucket first. This example shows you how to query the S3 server for a list of all the buckets you own.
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
:'''Note:''''' The target bucket must be empty!'' 
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>

CSC/ECE 517 Spring 2015/ch1a 7 SA

2015-02-14T01:13:29Z

Achen4: /* Creating a connection to S3 server */

[[File:S3.gif|frame|Source: http://www.w7cloud.com/7-reasons-to-use-amazon-s3-cloud-computing-online-storage/|right]] Amazon Simple Storage Service (Amazon S3) is a remote, scalable, secure, and cost efficient storage space service provided by Amazon. Users are able to access their storage on Amazon S3 from the web via REST <ref>[http://en.wikipedia.org/wiki/Representational_state_transfer Wikipedia: REST]</ref> HTTP <ref>[http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP]</ref>, or SOAP <ref>[http://en.wikipedia.org/wiki/SOAP SOAP]</ref> making their data accessible from virtually anywhere in the world. Amazon S3 implements redundancy across multiple devices on multiple facilities in order to safeguard against application failure ,data loss and minimization of downtime <ref>[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html Amazon S3]</ref>. Some of the most prominent users of Amazon S3 include: Netflix, SmugMug, Wetransfer, Pinterest, and NASDAQ <ref>[http://aws.amazon.com/s3/ Amazon S3 Homepage]</ref>.

[https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit Writing Assignment 1a]

==Background==

Amazon S3 launched in March of 2006 in the United States <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=830816 Amazon Press Release]</ref> and in Europe in November of 2007 <ref>[http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1072982 Amazon Press Release]</ref>. Since its inception, Amazon S3 has reported tremendous growth. Beginning in July of 2006, S3 hosted 800 million objects; April of 2007, 5 billion objects; October of 2007, 10 billion; Jan 2008, 14 billion <ref>[http://www.allthingsdistributed.com/2008/03/happy_birthday_amazon_s3.html Happy birthday Amazon S3]</ref>; October 2008, 29 billion <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-now/ amazon s3 now]</ref>; March 2009, 52 billion <ref>[https://aws.amazon.com/blogs/aws/celebrating-s3s-third-birthday-with-an-upload-promotion/ Upload promotion]</ref>; August 2009, 64 billion <ref>[http://www.eweek.com/c/a/Cloud-Computing/Amazons-Head-Start-in-the-Cloud-Pays-Off-584083 Amazons Head Start in the Cloud]</ref>. In April of 2013, S3 now hosts more than 2 trillion objects and on average 1.1 million requests every second! <ref>[https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ Two trillion objects]</ref>.

===Design===

S3 is an example of an object storage and is not like a traditional hierarchical file system. S3 exposes a simple feature set to improve robustness and all data in S3 is accessed in the terms of objects and buckets.

====Objects====

Objects are the basic units of storage in Amazon S3. Each object is composed of object data and metadata. S3 supports a size of up to 5 Terabytes per object. Each object has an associated metadata that is used to identify the object. Metadata is a set of name-value pairs that describe the object like date modified. Custom data about the object can be stored in metadata by the user. Every object is identified by a user defined key and is versioned by default. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html S3 Introduction]</ref>. An object consists of the following - Key, Version ID, Value, Metadata, Subresources and Access Control Information. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html Using Objects]</ref>

====Buckets====

A bucket is a container for objects and every object must be part of a bucket. Any number of objects can be part of a Bucket. Buckets can be configured to be hosted in a particular region (US, EU, Asia Pacific etc.) in order to optimize latency. S3 limits the number of buckets per account to 100. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html Using Bucket]</ref>

====Keys and Metadata====

An user specifies a key on object creation which is used to uniquely identify the object in the bucket. Keys for the objects can be at most 1024 bytes long.

There are two kinds of metadata for an object - System metadata and Object metadata. System metadata is used by S3 for object management. For eg. - Data, Content-Type etc. are stored as System metadata. Object metadata is optional and can be used by the user to add additional metadata to the objects during object creation. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html Using Metadata]</ref>

====Regions====

Regions allow a user to specify the geographical region where the buckets will be stored. This can be used to optimize latency and minimizing costs.
S3 supports the following regions - US Standard, US West (Oregon) region, US West (N. California) region, EU (Ireland) region, EU (Frankfurt) region, Asia Pacific (Singapore) region, Asia Pacific (Tokyo) region, Asia Pacific (Sydney) region, South America (Sao Paulo) region <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions Regions]</ref>

====Versioning====

All objects in S3 are versioned by default and it can be used to retrieve and restore every version of an object in a bucket. Every change to an object(create, modify, delete) results in a separate version of the object which can be later used for restoring or recovery. Versioning is done at the bucket level and not for individual objects. It can be turned off or on per bucket but a versioned-enabled bucket cannot be turned to an unversioned bucket. Versioning can only be paused in these cases. <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html Versioning]</ref>

====Access Permissions====

All resources(buckets,objects etc) are private in Amazon S3 by default. Only the resource owner can access the resource and can grant access to other users to accesss the resource. There are two types of access policies in S3 - Resource-based and user policies. Resource-based policies are attached to a particular resource and user policies are assigned to a particular user.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html Access Control]</ref>

====Data Protection====

Objects are redundantly stored on multiple devices across multiple facilities within a region for durability. To improve durability, write requests do not return success before storing the data across multiple facilities. Also checksums are used to verify data integrity. If any corruption is detected, it is repaired using redundant data.<ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html Data Durability]</ref>

===Ruby and Amazon S3===

Amazon Web Services (AWS) provides an SDK <ref>[http://rubygems.org/gems/aws-sdk| download]</ref> that works with Ruby for many amazon webservices, including Amazon S3. Developers new to the Amazon AWS SDK should begin with version 2 as it includes many built in features such as waiters, automatically paginated responses, and a streamlined plugin style architecture. Version 2 of the SDK has 2 "packages", also referred to as "gems" <ref>[http://en.wikipedia.org/wiki/RubyGems Wikipedia: RubyGems]</ref>:

:* '''aws-sdk-core''' - provides a direct mapping to the AWS APIs including automatic response paging, waiters, parameter validation, and Ruby type support
:* '''aws-sdk-resources''' - provides an object-oriented abstraction over low-level interfaces in the core to reduce the complexity of utilizing core interfaces; resource objects reference other objects such as an Amazon S3 instance and the attributes and actions as instance variables and methods.

It should also be noted that there exists a Version 1 of the aws sdk that lacks some "convenience features" otherwise available in version 2 of the sdk. For more information see the <ref>[http://ruby.awsblog.com/post/Tx2OMCYFEZX2I6A/AWS-SDK-for-Ruby-V2-Preview-Release|AWS Ruby Development Blog]</ref>

==Examples==
The following section contains examples of ruby interfacing with S3. Sources and documentation for the code are provided. Please observe the copyrights if you choose to use any or all of the posted code.

'''Note:''' There are 3 key classes in AWS SDK <ref>[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingTheMPRubyAPI.html Using the Ruby API]</ref> -

:* '''AWS::S3''' - Denotes an interface to Amazon S3 for the Ruby SDK. It has the ''#buckets'' instance method for creating new buckets or accessing existing buckets.
:* '''AWS::S3::Bucket''' - Denotes an Amazon S3 Bucket. It provides the ''#objects'' instance method to access existing objects and also other methods to get information about a bucket.
:* '''AWS::S3::S3Object''' - Denotes an Amazon S3 Object. It provides the method that gives information about the object and also setting access permissions, copying, deleting and uploading objects.

===Creating a connection to S3 server===

Connecting to the S3 server is the essential starting point of accessing your data. The follow example shows how to connect to the server via. SSL <ref>Wikipedia: SSL http://en.wikipedia.org/wiki/SSL</ref>
<pre>
AWS::S3::Base.establish_connection!(
:server => 'objects.example.com',
:use_ssl => true,
:access_key_id => 'my-access-key',
:secret_access_key => 'my-secret-key'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing all buckets you own===
<pre>
AWS::S3::Service.buckets.each do |bucket|
puts "#{bucket.name}\t#{bucket.creation_date}"
end
</pre>

'''Expected output:'''
<pre>
mybuckat1 2011-04-21T18:05:39.000Z
mybuckat2 2011-04-21T18:05:48.000Z
mybuckat3 2011-04-21T18:07:18.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Listing a bucket's contents===
<pre>
new_bucket = AWS::S3::Bucket.find('my-new-bucket')
new_bucket.each do |object|
puts "#{object.key}\t#{object.about<ref>['content-length']}\t#{object.about<ref>['last-modified']}"
end
</pre>
'''Expected output'''
<pre>
file1.filex 251262 2011-08-08T21:35:48.000Z
file2.filex 262518 2011-08-08T21:38:01.000Z
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting a bucket===
:'''Note:''''' The target bucket must be empty!'' 
<pre>
AWS::S3::Bucket.delete('my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Forced removal of non-empty buckets===
<pre>
AWS::S3::Bucket.delete('my-new-bucket', :force => true)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Creating an object===
<pre>
AWS::S3::S3Object.store(
'hello.txt',
'Hello World!',
'my-new-bucket',
:content_type => 'text/plain'
)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Change an object's ACL (access control list)===
<pre>
policy = AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket')
policy.grants = <ref>[ AWS::S3::ACL::Grant.grant(:public_read) ]
AWS::S3::S3Object.acl('hello.txt', 'my-new-bucket', policy)

policy = AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket')
policy.grants = <ref>[]
AWS::S3::S3Object.acl('secret_plans.txt', 'my-new-bucket', policy)
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Download an object to a folder===
:'''Note:''' ''This downloads the object poetry.pdf and saves it in /home/larry/documents/''
<pre>
open('/home/larry/documents/poetry.pdf', 'w') do |file|
AWS::S3::S3Object.stream('poetry.pdf', 'my-new-bucket') do |chunk|
file.write(chunk)
end
end
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Deleting an object===
:'''Note:''''' This deletes the object goodbye.txt''
<pre>
AWS::S3::S3Object.delete('goodbye.txt', 'my-new-bucket')
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Generating object download urls===
<pre>
puts AWS::S3::S3Object.url_for(
'hello.txt',
'my-new-bucket',
:authenticated => false
)

puts AWS::S3::S3Object.url_for(
'secret_plans.txt',
'my-new-bucket',
:expires_in => 60 * 60
)
</pre>
'''Expected Output:'''
<pre>
http://objects.dreamhost.com/my-bucket-name/hello.txt
http://objects.dreamhost.com/my-bucket-name/secret_plans.txt?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
</pre>
:Source: http://ceph.com/docs/master/radosgw/s3/ruby/

===Upload a file to Amazon S3===
:As per the Apache License v 2.0, the follow code is reproducible and redistributable with the following <ref>[http://aws.amazon.com/apache-2-0/ license]</ref>.
<pre>
# Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

require 'aws-sdk'

(bucket_name, file_name) = ARGV
unless bucket_name && file_name
puts "Usage: upload_file.rb <BUCKET_NAME> <FILE_NAME>"
exit 1
end

# get an instance of the S3 interface using the default configuration
s3 = AWS::S3.new

# create a bucket
b = s3.buckets.create(bucket_name)

# upload a file
basename = File.basename(file_name)
o = b.objects[basename]
o.write(:file => file_name)

puts "Uploaded #{file_name} to:"
puts o.public_url

# generate a presigned URL
puts "\nUse this URL to download the file:"
puts o.url_for(:read)

puts "(press any key to delete the object)"
$stdin.getc

o.delete
</pre>

See the following link for the documentation for AWS SDK - <ref>[http://docs.aws.amazon.com/AWSRubySDK/latest/_index.html AWS SDK for Ruby]</ref>

==References==
<references/>