Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:41:52Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. DataNode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
'''WebSocketClient'''
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''': Method to get the CountDownLatch.
'''5. sendMessage''': Method to send message to server

'''WebSocketServer'''
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

'''HeartBeatHandler'''
'''1. sendMessage''': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge.
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter].
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality.
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used.
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI.

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:17:20Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. DataNode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
'''WebSocketClient'''
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

'''WebSocketServer'''
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

'''HeartBeatHandler'''
'''1. sendMessage''': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge.
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter].
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality.
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used.
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI.

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:15:25Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
'''WebSocketClient'''
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

'''WebSocketServer'''
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

'''HeartBeatHandler'''
'''1. sendMessage''': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge.
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter].
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality.
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used.
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI.

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:13:41Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
'''WebSocketClient'''
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

'''WebSocketServer'''
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

'''HeartBeatHandler'''
'''1. sendMessage''': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:11:15Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

======'''WebSocketServer'''======
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

======'''HeartBeatHandler'''======
'''1. sendMessage''': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T04:10:40Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

======'''WebSocketServer'''======
'''1. onOpen''': An event listener to be called when connection has been opened.
'''2. onClose''': An event listener to be called when connection has been closed
'''3. onMessage''': An event listener to be called when message is received .

======'''HeartBeatHandler'''======
'''1. sendMessage'': Method to instantiate a WebSocketClient to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:59:36Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

======'''WebSocketServer'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''': 
'''5. sendMessage''': Method to send message to server.

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:55:33Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:55:09Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''': Method to send message to server

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
==='''Test cases'''===
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:52:04Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server.
'''4. getLatch''':
'''5. sendMessage''':
=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:51:09Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 
======'''WebSocketClient'''======
'''1. onOpen''': An event listener to be called when connection to server is established.
'''2. onClose''': An event listener to be called when connection to server is closed.
'''3. onText''': An event listener to be called when message is received from server
'''4. getLatch''':
'''5. sendMessage''':
=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 
3. Verifying the correctness of websocket behavior for agent Heartbeats received with different contents. We could have Heartbeats having a variety of data such as node status, component's status on a node, volume of requests processed over a past time period on a node etc.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:34:33Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current [https://en.wikipedia.org/wiki/Polling_(computer_science) polling mechanism] with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using Ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern Messaging Pattern].

Request–response is a message exchange pattern in which a requestor sends a request message to a replier system which receives and processes the request, ultimately returning a message in response. This is a simple, but powerful messaging pattern which allows two applications to have a two-way conversation with one another over a channel. In our project, Ambari-Server's HeartBeat handler acts as a requestor which sends the message to Ambari UI via WebSocket protocol. In response, Ambari UI acknowledges the received information and sends new requests if any to the Ambari-Server. Moreover, same messaging pattern is followed for bidirectional communication between Ambari-Agent and Ambari-Server.

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 

=='''Testing Framework'''==

A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. A whole prototype of an in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.
 
Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify / expect" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). Hence, tests are implemented in a result-oriented way and are not bound to the method implementation in anyway.
 
Following is a brief overview of the test cases that we look to add in the Ambari project to test out our functionality:

1. Testing if the websocket response to Ambari Web is triggered when a Heartbeat is received and persisted in the backend PostgreSQL DB.
 
2. Mocking an agent Heartbeat and asserting that the same status is reflected on the front-end Ambari Web.
 

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

1. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
2. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
3. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
4. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
5. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-14T03:16:27Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-Web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'Heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( Heartbeats ) from Ambari agents. Process their statuses and send Heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebServerClient.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

File:Ambari-WebServerClient.png

2015-11-14T03:12:52Z

Rpsingh3:

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-13T10:48:26Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
 Note: This is continuation of our OSS project 

==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T08:11:37Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====
====='''Class Diagram'''=====
[[File:Ambari-UML.png]]

1. New class is added for both Web Socket client and Server.
2. One method is added in the existing class "HeartBeatHandler" 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

 Note: This is continuation of our OSS project 

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

File:Ambari-UML.png

2015-11-12T08:08:03Z

Rpsingh3:

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T08:02:26Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====

[[File:Ambari-ClassDiagram.png]]

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

 Note: This is continuation of our OSS project 

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T08:01:55Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====

[[File:Ambari-ClassDiagram.png]]

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

 Note: This is continuation of our OSS project 

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T07:59:17Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. For this project, we will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern].

===='''UML Diagrams'''====

[[File:Ambari-ClassDiagram.png]]

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

File:Ambari-ClassDiagram.png

2015-11-12T07:57:28Z

Rpsingh3: uploaded a new version of "File:Ambari-ClassDiagram.png"

File:Ambari-ClassDiagram.png

2015-11-12T07:55:28Z

Rpsingh3:

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T07:44:28Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. We will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern], in this project.

===='''UML Diagrams'''====

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T07:43:15Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
==='''Project Goals'''===
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

==='''Project Design'''===
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

===='''Major Components'''====
Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

===='''Architectural design of WebSocket client-server in Apache Ambari'''====
[[File:Ambari-WebSocket.png]]

===='''Flow Diagram'''====
[[File:Ambari-Flow.png]]

==='''Implementation'''===
===='''Design Pattern'''====
We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes. We will be using [https://en.wikipedia.org/wiki/Request%E2%80%93response request-response], one of the [https://en.wikipedia.org/wiki/Messaging_pattern message exchange pattern], in this project.

===='''UML Diagrams'''====

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Benefits and Challenges'''==
'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==

2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: We are required to submit only one pull request and that too at the end of the final project. 

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-12T07:06:16Z

Rpsingh3:

=='''Introduction'''==

Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''Project Description'''==
1. '''Project Goals'''
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Flow Diagram'''==
[[File:Ambari-Flow.png]]

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Steps to build and test Ambari-WebSocket'''==
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: We are required to submit only one pull request and that too at the end of the final project. 

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

File:Ambari-WebSocket.png

2015-11-12T06:51:35Z

Rpsingh3:

CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari

2015-11-10T05:06:12Z

Rpsingh3: Created page with "=='''A1550 - Web Socket Implementation in Apache Ambari'''== Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show curr..."

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

==='''Steps to build and test Ambari-WebSocket'''===
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: We are required to submit only one pull request and that too at the end of the final project. 

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015

2015-11-10T05:01:23Z

Rpsingh3:

==Writing Assignment 2==
*[[CSC/ECE_517_Fall_2015/sample_page]]
*[[CSC/ECE_517_Fall_2015/ossE1558BGJ]]
*[[CSC/ECE 517 Fall 2015/oss/M1502/AAAASS]]
*[[CSC/ECE 517 Fall 2015/oss/M1503/IntegrateXMLParser]]
*[[CSC/ECE_517_Fall_2015/ossE1568BZHXJS]]
*[[CSC/ECE_517_Fall_2015/ossE1572VGA]]
*[[CSC/ECE_517_Fall_2015/oss_E1573_sap]]
*[[CSC/ECE 517 Fall 2015/oss E1559 rrz]]
*[[CSC/ECE 517 Fall 2015/oss E1570 avr]]
*[[CSC/ECE 517 Fall 2015/oss E1556 CHM]]
*[[CSC/ECE 517 Fall 2015/oss M1504 JJD]]
*[[CSC/ECE 517 Fall 2015/oss E1562 APS]]
*[[CSC/ECE 517 Fall 2015/oss M1501 GSN]]
*[[CSC/ECE 517 Fall 2015/oss M1501 GSN]]
*[[CSC/ECE 517 Fall 2015/oss E1550 KMM]]
*[[CSC/ECE 517 Fall 2015/oss E1551 RGS]]
*[[CSC/ECE 517 Fall 2015/oss E1555 GMR]]
*[[CSC/ECE 517 Fall 2015/oss E1552 NFR]]
*[[CSC/ECE 517 Fall 2015/oss E1565 AAJ]]
*[[CSC/ECE 517 Fall 2015/oss E1561 WZL]]
*[[CSC/ECE 517 Fall 2015/oss E1553 AAJ]]
*[[CSC/ECE 517 Fall 2015/oss E1554 AAR]]
*[[CSC/ECE 517 Fall 2015/oss E1569 JNR]]
*[[CSC/ECE 517 Fall 2015/oss E1560 PSV]]
*[[CSC/ECE 517 Fall 2015/oss M1505 MSV]]
*[[CSC/ECE 517 Fall 2015/oss E1557 GXM]]
*[[CSC/ECE 517 Fall 2015/oss E1566 ARB]]
*[[CSC/ECE 517 Fall 2015/oss E1567 APT]]
*[[CSC/ECE 517 Fall 2015/oss E1574 BKS]]
*[[CSC/ECE 517 Fall 2015/ossA1550RAN]]
*[[CSC/ECE 517 Fall 2015/oss E1571]]

==Final Project Design Document==
*[[CSC/ECE 517 Fall 2015 E1577 MayYellowRoverJump]]
*[[CSC/ECE 517 Fall 2015 E1586 AnonymousChatBetweenAuthorAndReviewer]]
*[[CSC/ECE 517 Fall 2015 E1582 Create integration tests for the instructor interface using capybara and rspec]]
*[[CSC/ECE 517 Fall 2015 E1576 Refactoring submitted content (hyperlinks and files)]]
*[[CSC/ECE 517 Fall 2015 E1590 Integration testing for Team creation]]
*[[CSC/ECE 517 Fall 2015 E1585 Use Ajax for Add Participants, Add TA ,Edit Questionnaires Screens]]
*[[CSC/ECE 517 Fall 2015 E1581 Integration testing for student interface]]
*[[CSC/ECE 517 Fall 2015 E1583 Fix the CSS used for Menu Item]]
*[[CSC/ECE 517 Fall 2015 E1591 Integration testing for peer review]]
*[[CSC/ECE 517 Fall 2015 E1589 Automating production setup and deployment]]
*[[CSC/ECE 517 Fall 2015 M1502 Improve HTTP monitoring devtool support]]
*[[CSC/ECE 517 Fall 2015 M1503 Integrate xml5ever XML parser]]
*[[CSC/ECE 517 Fall 2015 M1504 Implement support for missing XMLHttpRequest APIs]]
*[[CSC/ECE 517 Fall 2015 M1505 Add conformance tests to unicode-bidi and fix conformance bugs]]
*[[CSC/ECE 517 Fall 2015 M1501 Report CSS errors to the devtools, both stored and live]]
*[[CSC/ECE 517 Fall 2015 E1579 Instructor account creation over the web]]
*[[CSC/ECE 517 Fall 2015 A1550 Web Socket Implementation in Apache Ambari]]

CSC/ECE 517 A1550 Web Socket Implementation in Apache Ambari

2015-11-10T04:58:36Z

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-08T19:03:39Z

Rpsingh3:

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-07T01:19:46Z

Rpsingh3:

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

==='''Steps to build and test Ambari-WebSocket'''===
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: We are required to submit only one pull request and that too at the end of the final project. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-07T01:13:31Z

Rpsingh3: Add steps to build ambari

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

==='''Steps to build and test Ambari-WebSocket'''===
1. Clone the repository to your local system 
git clone https://github.com/nisarg64/ambari

2. Got to ambari directory and checkout the branch "migration" using below command 
git checkout migration

3. You will need nodeJs, maven etc to build the ambari. Check the requirements as described in [https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development Ambari PreConfigs]

4. After having the required configurations, run below command to build 
mvn clean package -DskipTests -Drat.ignoreErrors=true

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: Pull request is yet to be submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-04T02:09:57Z

Rpsingh3:

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket client-sever files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in [https://en.wikipedia.org/wiki/Apache_Maven#Project_Object_Model POM] files.

Note: Pull request is yet to be submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-04T01:12:27Z

Rpsingh3:

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket sever-client files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in pom files.

Note: Pull request is yet to be submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-04T01:10:52Z

Rpsingh3:

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the [https://en.wikipedia.org/wiki/Law_of_Demeter Law of Demeter]
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses Jetty which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket sever-client files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in pom files.

Note: Pull request is yet to be submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-04T01:09:32Z

Rpsingh3:

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the Law of Demeter
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses Jetty which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

'''Major code changes:'''
Addition of websocket sever-client files [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-0d2947581a9630d9b41c77e5f0fd23b6 WebSocketServer.java] and [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-4ac64b666141c53eb339c0375455d311 WebSocketClient.java] 
Changes in heartbeat handler - [https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b#diff-1b3d5840e204ec59cec0cc42c9de12ea HeartBeatHandler.java] 
Version changes in pom files.

Note: Pull request is yet to be submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf

CSC/ECE 517 Fall 2015/ossA1550RAN

2015-11-04T00:37:06Z

Rpsingh3: Added links to main topics

=='''A1550 - Web Socket Implementation in Apache Ambari'''==
Ambari-Web uses simple ajax polling mechanism to fetch data from Ambari-Server. Constant polling is done to show current service status, alerts, service graphs, etc on Ambari-Web. With this mechanism, the performance of Ambari-Server can be affected on a large size cluster with multiple active browser sessions due to continuous heavy requests being made.

[https://en.wikipedia.org/wiki/WebSocket WebSocket] is a protocol providing full-duplex communication channels over a single TCP connection. Implementing Web-Socket between Ambari-Web and Ambari-Server will be helpful to address this scenario.

=='''What is Apache Ambari'''==
Apache Ambari is a software project of the [https://en.wikipedia.org/wiki/Apache_Software_Foundation Apache Software Foundation], is aimed at making [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] management web UI backed by its [https://en.wikipedia.org/wiki/Representational_state_transfer RESTful] APIs. Ambari was a sub-project of [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] but is now a [https://en.wikipedia.org/wiki/Apache_Software_Foundation#Projects top-level] project in its own right.

An Apache Hadoop cluster consists on a group or nodes/machines. Each node acts as an Ambari Agent that runs various service components (e.g. Datanode for HDFS). One of the agents acts as an Ambari Server that takes care of the task allocation, management, gathering information about the services status and other information from the agents and giving the information to Ambari UI for display. More information about Agent-Server-Web flow is available in the following sections.

=='''Current Implementation'''==
Ambari is a multi-module Maven project. The relevant modules for our project from an implementation perspective are Ambari Web and Ambari Server. Ambari Agent is another module, important functionally.

Ambari Agent is a component that resides on every node of the cluster. Its responsibility is to collect the node's overall health which comprises of the health metrics of all the different services running on that node. Ambari Agent sends these statuses to the Ambari Server in the form of periodic messages which is termed as 'heartbeat' in the Ambari nomenclature. In return, Ambari agents receive Heartbeat responses from Ambari Server. Hearbeat responses could contain commands for the agent to perform.

 Ambari Web presents a user interface to monitor and manage a Hadoop cluster. Web module is written in JavaScript using ember.js framework. During our code study, we observed this module following the [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] design methodology. The whole motivation of this module is to make it convenient to the end-user to manage the concerned Hadoop cluster.

 Ambari Server is the most central component of an Ambari ecosystem. Ambari Server exposes [https://en.wikipedia.org/wiki/Application_programming_interface APIs]] to its clients such as Ambari Web, which monitors health of a cluster. Ambari Server clients can query for node health statuses. Ambari Server persists the nodes' statuses in a [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL] database in the backend. Internally, Ambari Server has several components which help it to manage its responsibilities. Ambari Server's important responsibilities are as follows:
 1. Collect health statuses ( heartbeats ) from Ambari agents. Process their statuses and send heartbeat responses to Ambari agents. 
 2. Handle requests from Ambari Server's clients: One such use case is installing a new service on an Ambari agent. This request could be received from Ambari Web and it is the server's responsibility to send appropriate instructions to the particular agent to get the new service up and running on the agent. 

=='''Testing Framework'''==
The codebase is very well supported with unit tests. Following are a few important characteristics of the testing process that we observed during our code study:

1. A popular mocking framework, [https://en.wikipedia.org/wiki/Mockito Mockito] is used to mock responses from dependent downstream services. An in-memory cluster is also mocked using Mockito. Mocking frameworks let us have our test cases fast and independent, which is very well depicted in this project.

2. Every controller method is tested in and out. Tests are quite extensive and cover almost all the code branches, testing each and every feature thoroughly.

3. Only the results/responses obtained from controller methods are asserted. We do not find any significant "verify() / expect()" calls. ("verify/expect" calls are used to assert if a given method was invoked during the test method execution). This is quite a good feature followed as the tests are implemented in a result-oriented way and are not bound to the method implementation in anyway. This also goes to show the stability of the tests as if the method implementation undergoes changes in future, it won't fail our tests.

=='''Project Goals, Benefits and Challenges'''==
'''Goals:'''
1. Understand architecture of Ambari-Server and Ambari-web
2. Replace the current pull-based mechanism with the push-based mechanism via Web-Socket
3. Write test cases for the Web Socket implementation to test the functionalities of WebSocket Client and WebSocket Server

'''Benefits:'''
1. Web Socket will allow Ambari-Server to perform robustly in a large cluster setup with multiple browser sessions and when continuous heavy requests are being made

'''Challenges:'''
1. Ambari server uses [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 8.x version, while support for Web Socket was made available after [https://en.wikipedia.org/wiki/Jetty_%28web_server%29 Jetty] 9.x
2. Ambari has a huge codebase with multiple modules and multiple frameworks and design patterns adopted. Understanding the flow of the project and taking care of dependencies is a huge challenge
3. For testing the code requires a cluster of around 3 nodes. Running 3 virtual machines requires a high performance machine.

=='''Learning Outcomes'''==
1. We have observed that Ambari project adopts various design patterns like [https://en.wikipedia.org/wiki/Singleton_pattern Singleton Pattern] in the Data Access Object Classes
2. We also noticed in one of the classes called HeartBeatHandler, that it obeys the Law of Demeter
3. For testing, project using the [https://en.wikipedia.org/wiki/Mockito Mockito] framework to mock the heartbeat and cluster functionality
4. For configuration management, [https://en.wikipedia.org/wiki/Puppet_%28software%29 Puppet] configuration management tool is used
5. Ambari-Web is implemented in [https://en.wikipedia.org/wiki/Ember.js Ember.js] which is an open-source Javascript framework that follows [https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC] pattern similar to Rails framework.
6. Ambari-server uses Jetty which is a web-server to handle all the HTTP requests made onto the ambari-server from the UI

=='''Github Location'''==
[https://github.com/apache/ambari Ambari Project]

Forked Repository:
[https://github.com/nisarg64/ambari Forked Repo]

Changes committed to forked repositry:
[https://github.com/nisarg64/ambari/commit/f7b9131cb7324dda39b95db72673a63ab2ac109b Code changes]

Note: No pull request is submitted. 
There are version issues because of which the build is unsuccessful after applying our changes. The main issue is because of mismatch between jetty versions - version which is supported by ambari and version after which web socket was made available. We are in touch with the other contributors and are working on how to resolve the issue.

=='''References'''==
https://ambari.apache.org/
https://en.wikipedia.org/wiki/Apache_Ambari
https://cwiki.apache.org/confluence/display/AMBARI/Ambari
https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf