Independent Study Spring 2019/Errbit: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
(Created page with "=Set up an open-source error monitoring tool instead of Airbrake= ==Useful links== [https://github.com/expertiza/expertiza/pull/1148 Github Pull Request] [https://github.com/err...")
 
No edit summary
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Set up an open-source error monitoring tool instead of Airbrake=
=Set up an open-source error monitoring tool instead of Airbrake=
==Useful links==
==Useful links==
[https://github.com/expertiza/expertiza/pull/1148 Github Pull Request]
[ Github Pull Request]


[https://github.com/errbit/errbit Errbit]
[https://github.com/errbit/errbit Errbit]


==Purpose==
==Purpose==
Logging feature has been added to Expertiza through a custom logging module written uniquely to serve several purposes such as:
Errbit has been set-up, replacing the already existing Airbrake API to serve several purposes like :
*Tracing application failure
*Errbit is a much more powerful monitoring tool
*Analysing user request flow
*Errbit is Airbrake API compliant
*Managing different logging level
*Store unresolved errors indefinitely without any extra cost.


==Message Format==
==Components==
All the messages are logged in either of the below formats:
*Errbit
 
*Expertiza
'''TST=[2018-03-10 00:50:18 -0500] SVT=[<''log_level''>] PNM=[<''program_name''>] OIP=[<''origin_ip''>] RID=[<''request_id''>] CTR=[<''controller_name''>] UID=[<''user_id''>] MSG=[''Message'']'''
*MongoDB
 
or
 
'''TST=[2018-03-10 00:50:18 -0500] SVT=[<''log_level''>] PNM=[<''program_name''>] OIP=[] RID=[] CTR=[<''controller_name''>] UID=[<''user_id''>] MSG=[[''origin_ip''] [''request_id''] ''Message'']'''
 
Above variation is because when certain messages are logged, ExpertizaLogger is unaware of origin_ip and request_ip and thus it is appended to the message instead of their respective placeholders.
 
===Meaning of each placeholder===
'''SVT''' - It denotes the severity level of the log. For eg, if the logging is done by calling logger.info, the severity level is "INFO".
 
'''PNM''' - It's the program name and is same as provided by default Rails logger.
 
'''OIP''' - It denotes the IP address of the system from where the request has been made.
 
'''RID''' - It's the ID of the request. All subsequent calls made by the client till the actual response is returned will have same request ID. For eg, when a page is first requested, all smaller page template fetched, all method flow will have same request ID. This field can be used to understand how a certain request was served.
 
'''CTR''' - It is the controller name from where the logging happened.
 
'''UID''' - It is the unity id/user id of the user making requests. This field could also be used in identifying the cause of an error.
 
'''Message''' - It is the actual message logged.
 
==Log files==
For different log levels, log messages are written in different files named expertiza_<log_level>.log
 
Currently, expertiza provides the capability of writing logs into 5 different custom log files.
 
For eg,
 
*expertiza_info.log
 
*expertiza_warn.log
 
*expertiza_error.log
 
*expertiza_fatal.log
 
*expertiza_debug.log
 
Apart from above files, <environment>.log file is used for logging messages which could not be logged by ExpertizaLogger, like, page load messages or certain load time exceptions.
 
Since Rails logger provides writing into a specific log file, to restrict least file switch, ExpertizaLogger holds different log level loggers in separate instance variables. A new logger instance is created and assigned whenever Rails removes the link to manage memory consumption.


==Components==
==The Steps taken for the setup==
===ExpertizaLogger===
There were several steps taken to set-up Errbit for Expertiza and schedule it to automatically pull changed from its forked branch and deploy:


This class provides the logger needed for logging messages for different log levels.
'''Setup Heroku'''
It provides 5 different static methods namely '''info''', '''warn''', '''error''', '''fatal''' and '''debug''' which could be called as follows:
*Made a new Heroku account with the following credentails:
  Email: expertiza-support@lists.ncsu.edu
  Password: expertiza2019@


'''ExpertizaLogger.info ''message'''''


where ''message'' could be an instance of '''LoggerMessage''' or a string.
'''Setup Errbit'''
*Fork Errbit from https://github.com/errbit/errbit and follow the steps mentioned on [https://github.com/errbit/errbit/blob/master/README.md README.md]
*When we run "rake db:seed" it will create an admin user with a random password. We can see these login credential in the console log. But instead, we can provide this username and password explicitly by just making some changes in errbit/db/seed.rb file. Eg:


===Custom LoggerMessage===
  def admin_email
      return 'expertiza-support@lists.ncsu.edu' if heroku_pr_review_app?
    "expertiza-support@lists.ncsu.edu"
  end
  def admin_pass
    return 'errbit' if heroku_pr_review_app?
    @admin_pass ||= "errbit"
  end


A LoggerMessage instance could be created as follows:


'''LoggerMessage.new(<controller_name>, <user_id>, <message>, <request object>)'''
*Seed the DB (NOTE: No bootstrap task is used on Heroku!).
  heroku run rake db:seed


Fourth parameter i.e <request object> is an optional parameter.
===Deploy Errbit on Heroku===
To deploy Errbit on Heroku, so that we could configure it to catch errors for Expertiza, the following steps were taken:


Whenever a request is made, the request object is injected into the controller and is available in the controller as "request".
*Created a new empty application on Heroku and named it errbit-expertiza2019
Similarly, <controller_name> is available in "controller_name" inside controller. For other trigger points, file name could be passed as a string.
*The app will have an associated empty git repository https://git.heroku.com/errbit-expertiza2019.git


Since the request object holds the ''<origin_ip>'' and ''<request_id>'', it is used to capture and log both into the log message.
Took the following steps in the /errbit repository on local machine:


===ExpertizaLogFormatter===
*Login to Heroku using Herkou CLI from your console
  heroku login
*Added remote
  heroku git:remote -a  errbit-expertiza2019
*As errbit needs MongoDB database backend so need to install MongoDB addon in Heroku. We can use heroku command to create an addon.
  heroku addons:add mongolab:sandbox </br>
  heroku config:add HEROKU=true </br>


It captures the message and determines if the message is an instance of LoggerMessage or not. Based on that, it creates a message by populating message placeholders as below:
To periodically clear resolved errors to free up space, With the cron add-on:Install the heroku cron addon, to clear resolved errors daily:
  heroku addons:add cron:daily


    if msg.is_a?(LoggerMessage)
Run the following to deploy.
      "TST=[#{ts}] SVT=[#{s}] PNM=[#{pg}] OIP=[#{msg.oip}] RID=[#{msg.req_id}] CTR=[#{msg.generator}] UID=[#{msg.unity_id}] MSG=[#{filter(msg.message)}]\n"
  git add .  
    else
  git commit -m “new creds”
      "TST=[#{ts}] SVT=[#{s}] PNM=[#{pg}] OIP=[] RID=[] CTR=[] UID=[] MSG=[#{filter(msg)}]\n"
  git push heroku master
    end


All the exceptions are preprocessed to remove newline characters so that it could fit in a single line as a message.


All the messages that are not logged by ExpertizaLogger, there is a separate logger definition provided in config/environments/<environment>.rb as:
'''Configure Expertiza to point to our deployed Errbit application
*Create a new app on Errbit named Expertiza
*Copy its configuration and place it in the expertiza/config/initializers/airbrake.rb file.


  config.log_formatter = proc do |s, ts, pg, msg|
  Airbrake.configure do |config|
    if msg.is_a?(LoggerMessage)
  config.host = 'https://errbit-expertiza2019.herokuapp.com'
      "TST=[#{ts}] SVT=[#{s}] PNM=[#{pg}] OIP=[#{msg.oip}] RID=[#{msg.req_id}] CTR=[#{msg.generator}] UID=[#{msg.unity_id}] MSG=[#{filter(msg.message)}]\n"
  config.project_id = 1 # required, but any positive integer works
    else
  config.project_key = '64ed97f0c8e628acefb3a7f63308a11c'
      "TST=[#{ts}] SVT=[#{s}] PNM=[#{pg}] OIP=[] RID=[] CTR=[] UID=[] MSG=[#{filter(msg)}]\n"
  # Uncomment for Rails apps
    end
  config.environment = Rails.env
  config.ignore_environments = %w(test)
   end
   end
On starting the server, the errors would now be caught by Errbit.


===Environment config===
config.log_tags = [ :remote_ip, :uuid ]
These config.log_tags entries make sure that <origin_ip> and <request_id> are prepended to messages. These are for messages which are not logged by ExpertizaLogger but by the Rails default logger.
These messages go in <environment>.log file.
==Restrictions==
The request object is not available beyond the controllers. Hence, ''<origin_ip>'' and ''<request_id>'' remains uncaptured if messages are logged from somewhere other than controllers.
==Visualization through ELK stack==
===Configuration===
Filebeat (filebeat.yml)
Changes:
- type: log
  # Change to true to enable this prospector configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - <path to log files>*.log
output.logstash:
  # The Logstash hosts
  hosts: ["<ip>:<5044:port of logstash>"]
Filebeat sends all log files to logstash.
Logstash (expertiza-logstash.conf)
  input {
        beats {
            port => "5044"
        }
  }
  filter {
    grok {
    match => { "message" => "TST=\[%{DATA:timestamp}\] SVT=\[%{DATA:sev}\] PNM=\[%{DATA:pgnm}\] OIP=\[%{DATA:originip}\] RID=\[%{DATA:requestid}\] CTR=\[%{DATA:generatedby}\] UID=\[%
  {DATA:unityid}\] MSG=\[%{GREEDYDATA:message}\]" }
    overwrite => [ "message", "timestamp" ]
  }
  date {
    match => ["timestamp", "ISO8601"]
  }
 
    }
    output {
      elasticsearch {
        hosts => [ "localhost:9200" ]
      }
    }
Logstash reads each line from the log file and parses it based on the template provided in grok match above. It then sends each message entry into Elastic search.
Kibana (config/kibana.yml)
    elasticsearch.url: "http://localhost:9200"
Kibana picks each entry from Elastic search and provides into the web interface.
===Setup===
ELK stack could be set up to visualize and analyze the messages from the log files.
Once ELK is installed, below commands could be used to run each server:
'''Clear Elastic search cache'''
curl -X DELETE 'http://localhost:9200/_all'
'''Filebeat'''
sudo rm data/registry
sudo ./filebeat -e -c filebeat.yml -d "publish"
'''Logstash'''
bin/logstash -f expertiza-logstash.conf --config.reload.automatic
'''Kibana'''
bin/kibana
===Kibana Interface===
Success message in Kibana
[[File:kibana_success_msg.png|frame|upright|center]]
Exception in kibana
[[File:kibana_exception.png|frame|upright|center]]
Tracing error in kibana


[[File:kibana_tracing_error.png|frame|upright|center]]
'''Set up Cron job to regularly pull changes from upstream/master into the forked repository and deploy on Heroku.'''


Search records using request ID in kibana
*Add the original GitHub repository as a "remote" called upstream to [https://github.com/errbit/errbit Errbit]
*Add the empty Git repository made during app creation on Heroku as a "remote" called "production".
In the .git/config, add:
 
    [remote "upstream"]¬
      url = https://github.com/errbit/errbit.git¬
      fetch = +refs/heads/*:refs/remotes/upstream/*¬
    [remote "production"]¬
      url = https://git.heroku.com/errbit-expertiza2019.git¬
      fetch = +refs/heads/*:refs/remotes/heroku/*¬
*Setup tasks to fetch from upstream and rebase and then push to Heroku.
*Task to migrate the database into production
*To deploy using these tasks, run:
  rake deploy:production DEPLOY_BRANCH=production


[[File:kibana_search_req_id.png|frame|upright|center]]
You could access the app using:
  https://errbit-expertiza2019.herokuapp.com/
  UserName: expertiza-support@lists.ncsu.edu
  Password: errbit


Searching records using unity id in kibana


[[File:kibana_search_using_unityid.png|frame|upright|center]]
===Errbit Interface===
*Deployment of Errbit on Heroku, showing successful build.
[[File:Build.png|frame|upright|center]]
*Login to Errbit using the admin credentials we specified.
[[File:ErrbitLogin.png|frame|upright|center]]
*Create a new App on Errbit to catch Errors for Expertiza
[[File:ExpertizaApp.png|frame|upright|center]]
*Errors from Expertiza are caught in this manner and displayed.
[[File:ErrorDetls.png|frame|upright|center]]
*Similar errors are grouped together.
[[File:ErrorsGroup.png|frame|upright|center]]

Latest revision as of 14:39, 8 May 2019

Set up an open-source error monitoring tool instead of Airbrake

Useful links

[ Github Pull Request]

Errbit

Purpose

Errbit has been set-up, replacing the already existing Airbrake API to serve several purposes like :

  • Errbit is a much more powerful monitoring tool
  • Errbit is Airbrake API compliant
  • Store unresolved errors indefinitely without any extra cost.

Components

  • Errbit
  • Expertiza
  • MongoDB

The Steps taken for the setup

There were several steps taken to set-up Errbit for Expertiza and schedule it to automatically pull changed from its forked branch and deploy:

Setup Heroku

  • Made a new Heroku account with the following credentails:
  Email: expertiza-support@lists.ncsu.edu
  Password: expertiza2019@


Setup Errbit

  • Fork Errbit from https://github.com/errbit/errbit and follow the steps mentioned on README.md
  • When we run "rake db:seed" it will create an admin user with a random password. We can see these login credential in the console log. But instead, we can provide this username and password explicitly by just making some changes in errbit/db/seed.rb file. Eg:
  def admin_email
     return 'expertiza-support@lists.ncsu.edu' if heroku_pr_review_app?
    "expertiza-support@lists.ncsu.edu"
  end
  def admin_pass
   return 'errbit' if heroku_pr_review_app?
   @admin_pass ||= "errbit"
  end


  • Seed the DB (NOTE: No bootstrap task is used on Heroku!).
 heroku run rake db:seed

Deploy Errbit on Heroku

To deploy Errbit on Heroku, so that we could configure it to catch errors for Expertiza, the following steps were taken:

Took the following steps in the /errbit repository on local machine:

  • Login to Heroku using Herkou CLI from your console
  heroku login
  • Added remote
  heroku git:remote -a  errbit-expertiza2019
  • As errbit needs MongoDB database backend so need to install MongoDB addon in Heroku. We can use heroku command to create an addon.
  heroku addons:add mongolab:sandbox 

  heroku config:add HEROKU=true

To periodically clear resolved errors to free up space, With the cron add-on:Install the heroku cron addon, to clear resolved errors daily: 

  heroku addons:add cron:daily

Run the following to deploy. 

  git add . 
  git commit -m “new creds” 
  git push heroku master


Configure Expertiza to point to our deployed Errbit application

  • Create a new app on Errbit named Expertiza
  • Copy its configuration and place it in the expertiza/config/initializers/airbrake.rb file.
  Airbrake.configure do |config|
 config.host = 'https://errbit-expertiza2019.herokuapp.com'
 config.project_id = 1 # required, but any positive integer works
 config.project_key = '64ed97f0c8e628acefb3a7f63308a11c'
 # Uncomment for Rails apps
  config.environment = Rails.env
  config.ignore_environments = %w(test)
 end

On starting the server, the errors would now be caught by Errbit.


Set up Cron job to regularly pull changes from upstream/master into the forked repository and deploy on Heroku.

  • Add the original GitHub repository as a "remote" called upstream to Errbit
  • Add the empty Git repository made during app creation on Heroku as a "remote" called "production".

In the .git/config, add: 

   [remote "upstream"]¬
     url = https://github.com/errbit/errbit.git¬
     fetch = +refs/heads/*:refs/remotes/upstream/*¬
   [remote "production"]¬
     url = https://git.heroku.com/errbit-expertiza2019.git¬
     fetch = +refs/heads/*:refs/remotes/heroku/*¬
  • Setup tasks to fetch from upstream and rebase and then push to Heroku.
  • Task to migrate the database into production
  • To deploy using these tasks, run:
  rake deploy:production DEPLOY_BRANCH=production

You could access the app using: 

  https://errbit-expertiza2019.herokuapp.com/
  UserName: expertiza-support@lists.ncsu.edu
  Password: errbit


Errbit Interface

  • Deployment of Errbit on Heroku, showing successful build.
  • Login to Errbit using the admin credentials we specified.
  • Create a new App on Errbit to catch Errors for Expertiza
  • Errors from Expertiza are caught in this manner and displayed.
  • Similar errors are grouped together.
Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=Independent_Study_Spring_2019/Errbit&oldid=125029"