PeerLogic Web Services: Python Web Service Template

The purpose of this page is to provide a useful template for creating a new PeerLogic web service, and provide some guided steps in how to modify it for new needs. This template is specifically made for Python 3 web applications.

Video Walkthrough

This is essentially a written alternative/supplement to a recorded video walking through the set up of the "AllReviewInterface" service. If you would like a more visual or hands-on demonstration of the information presented here, please check out the following link (or watch the video as you read through this written material): https://drive.google.com/file/d/15Q5hBXU4umpJN8gStLryo5XdNiy1xZay/view?usp=sharing.

Foreword: Necessary Tools and Permissions

Before beginning to create a web service, you will need some tools and permissions. These are:

• A way to SSH to a remote server: you can do this through the "ssh" command on Mac or Linux, or through the PuTTY application on Windows.
• A way to send API: Any will do, but I personally recommend Insomnia, as I find its interface easy to use and good for troubleshooting.
• Sudo access to the Peer Logic web server: You will need sudo access in order to modify the config files to set up the web service. Request this from CSC IT and CC Dr. Gehringer so he can agree to give you permission. In case it needs saying: don't abuse this access, or things will go poorly!

With all of these tools, we can begin to create a web service.

Template: AllReviewInterface

The template being used for this tutorial is the "AllReviewInterface" web service. This is a working web service on PeerLogic, accessible at peerlogic.csc.ncsu.edu/allreviewinterface/call_models, which has its functionality limited to a single short python file, making it easy to understand and use. The GitHub for this project can be found here: https://github.com/peerlogic/AllReviewInterface.

To begin with, clone this repository and rename it to the name of the project you intend to create. You should also (whether now or after writing the code) update the README for your project. The README for the AllReviewInterface serves as a good template for the basic information your README should provide (general summary of functionality, the port number and URL to access it from on PeerLogic, and the expected Input and Output with explanations and examples).

Using Flask

This application works through utilizing Flask, a Python module which allows your application to accept API requests. As is, you should be able to run the AllReviewInterface program and launch a Flask application. You can then communicate with this application by sending it API (through the tool outlined in the foreword). The AllReviewInterface can be tested in this way by sending its example input (as shown in its README) to "localhost:3013/call_models".

There are two important configuration details to modify within this application: the port number, and the URL route.

The port number is described on line 53 of the AllReviewInterface's "flaskapp.py", under "if __name__ == '__main__':". This determines what port the application is run on, both on your computer and on PeerLogic. Set this to a new, unused port (to see which ports are unused, refer to the configuration spreadsheet here: https://docs.google.com/spreadsheets/d/1oMYqjaVInPEE4jE7C2iQmOTc0H6BsltnuXXf0UWvuPA/edit?usp=sharing).

The URL route specifies the exact route necessary to reach particular code. In the case of AllReviewInterface, there is only one such route, "/call_models", which is specified on line 15 by "@app.route('/call_models',methods=['POST']). When a user accesses this route, the function defined immediately after it (in the template's case, "call_models()") is run. If the method specified is a POST method, the information within that POST can be accessed via flask.request.json. Rename the given route to one that better suits your use case, and feel free to create multiple routes if appropriate in order to have multiple supported methods.

Note that, to return the results of your web service, you must return a flask.Response object, containing the string which is your results. This can be seen in the template by the return statement of call_models(), on line 50.

Replace the code within this template with your own, renaming/reconfiguring the details described above, in order to create your own web service. You can also spread your code across multiple files, so long as the Flask code is contained within the given template file, and that is the file that is run on the PeerLogic server.

Setting Up a Virtual Environment

A difficult issue when running Python code is the necessity for packages. In fact, the "flask" package itself is not part of Python's standard libraries, and must be installed manually. While a user can do this easily using the "pip" command, this will not work on the PeerLogic server, as the root user will not have the necessary packages (and we don't want to download them for the root, as it might corrupt the server).

This problem can be alleviated through the use of virtual environments. A virtual environment can essentially be set up and contain all the packages necessary to run an application, without those packages needing to be installed by a user. This makes them great for sending code to be executed by others.

Create a virtual environment for your application using the command:

python3 -m venv /venv

This will create a virtual environment in the same directory as your application named "venv". Activate this virtual environment using the command:

source venv/bin/activate

(NOTE: this will only work on Unix or Mac. I would recommend installing WSL2 if on a Windows system, as virtual environments are difficult to use in Windows command line)

Now that you are in the virtual environment, attempt to run your application (for example, by using the command "python3 flaskapp.py"). If there are any non-standard packages required for your application (such as flask), you will receive an import error naming the missing package. Install that package using "pip install <package_name>", and continue doing so until you no longer receive errors.

You have now created a virtual environment containing all the packages necessary to run your application, without downloading any onto your personal Python installation. However, you cannot send a virtual environment directly onto the PeerLogic server (it won't set up correctly). Instead, you will want to create a virtual environment on the server itself. You can make this easier by creating a "requirements.txt" file using your current virtual environment. Use this command after you have installed all the necessary packages for your application:

pip freeze > requirements.txt

This will export all of the installed packages into a text file. I would recommend going into the text file and removing the version numbers from all of the packages, as this will make them easier to run on the older distribution of Python the server uses (for example, changing the line "flask==2.0.3" to just "flask").

You can exit the virtual environment when you are done using the command "deactivate".

This concludes all of the code and work you need to do to set up the source code for your web service!

Getting Code Onto The Server

Once you have completed the above, access the PeerLogic server (see the main documentation page for details on how to do this: https://expertiza.csc.ncsu.edu/index.php/PeerLogic_Web_Services) and navigate to the webservices directory. Once there, clone your repository into the directory using a command like the following:

git clone https://github.com/peerlogic/<name_of_your_repo>

(Note: if your repository is not currently part of the peerlogic group, change its ownership to the group! This will allow others to edit and upkeep your work in the future)

This will clone over your code onto the server. However, your virtual environment will not be set up. To set it up, navigate to the directory containing your requirements.txt file and use the following command:

virtualenv --python=/usr/bin/python3.6 /venv

(note: if this fails, you may need to install the virtualenv package. Do so with "pip install virtualenv" and repeat this step) (double note: this command is different from the previous because the server runs an older version of Python)

This will set up a new virtual environment. Activate the environment using the prior mentioned command. Now, you can install all of the required packages by simply using the following command:

pip install -r requirements.txt

This will automatically install all listed packages. You can then test to make sure your application is working by running "python3 <your_app.py)". If flask successfully launches, congratulations! You have set up your web service! Whenever you are done testing, you can leave the virtual environment using the command "deactivate".

Properly Configuring the Server

A final important note is how to set up the server such that your application is properly set up and run. Doing this will require the server to not simply launch your application, but to first enter its virtual environment, run the application, and then exit the environment.

Upon its daily reboot, the server will run the contents of the file runws_root.sh, located within the webservices directory. Edit this file (through a command such as "nano runws_root.sh") and add a new line for your Flask application. The syntax for this line should be very similar to the existing code to set up the AllReviewInterface application, and should look like the following:

cd /opt/webservices/<your_application_repo_name>/app/ && source venv/bin/activate && (python3 <your_app.py> &) && deactivate

To break down what this is doing, the root user will perform the following actions in order:

• navigate to where your Flask application is located
• activate the virtual environment
• run your application with the "&" character, allowing it to run in the background while the root user moves on
• deactivate the virtual environment

It is important to exit the virtual environment at the end, so as to allow the root user to continue doing what it needs to do. With this, the server will now properly run your application at every reboot.

The final step is to set up the route to call your application. Do this by editting NGINX's configuration file, located at /etc/nginx/nginx.conf (for example, using the command "sudo nano /etc/nginx/nginx.conf"). Be very careful editting this file! If you somehow break it beyond repair, a backup exists in the google drive with all the documentation.

Go down to the bottom of this file and add in a line similar to the following, substituting in the name of the URL path you set up for your application and the port you set up for your application:

location ^~ /[URL_PATH_TOBE_EXPOSED_EXTERNALLY] {
        	proxy_pass http://localhost:<PORT_FOR_APPLICATION_YOU_SET_UP>/;
}

The other existing locations within the file should be reasonable examples.

Conclusion

Congratulations! This concludes this tutorial, and this should be sufficient to let you set up and run your own Python Web Service on PeerLogic. Please check out the video walkthrough linked at the top of this page for a more visual demonstration of these steps, and if you run into any trouble, check out this document for some useful troubleshooting tips: https://docs.google.com/document/d/1oI8TTMW_WzYdsDsWXuT5_3-H8hea3Rq-YWswoLBQ81k/edit?usp=sharing.

I hope you have success with this. Do good work!