CSC/ECE 517 Spring 2023 - NTNX-3. Refactor models to keep profiles (software, compute, network, etc) as optional and use default if not specified

Background

Kubernetes An open-source container orchestration technology called Kubernetes is used to automatically deploy, scale, and manage containerized applications. Developers can use Kubernetes to distribute and control containerized applications across a dispersed network of servers or PCs. To ensure that the actual state of an application matches the desired state, it uses a declarative model to express the desired state and automatically manages the containerized components. Kubernetes can be operated on public or private cloud infrastructure as well as in-house data centers and offers a wide range of functionality for managing containerized applications, such as autonomous scaling, rolling updates, self-healing, service discovery, and load balancing.

Nutanix Database Service

A hybrid multi-cloud database-as-a-service for Microsoft SQL Server, Oracle Database, PostgreSQL, MongoDB, and MySQL, among other databases, is called Nutanix Database Service. It allows for the efficient management of hundreds to thousands of databases, the quick creation of new ones, and the automation of time-consuming administration activities like patching and backups. Users can also choose certain operating systems, database versions, and extensions to satisfy application and compliance requirements. Customers from all around the world have optimized their databases across numerous locations and sped up software development using Nutanix Database Service.

Features offered by NDB Service:

1. Nutanix NDB is a distributed NoSQL database service that is part of the Nutanix platform. Some of the key features of NDB include highly scalable architecture, distributed data storage, support for multiple data models, consistent data, fast data access, automatic sharding, real-time analytics, high availability and fault tolerance, and strong security features.
2. With its ability to scale up or down the number of nodes in a cluster, Nutanix NDB provides highly scalable architecture without any downtime. Its distributed architecture ensures high availability and fault tolerance, while its support for multiple data models makes it a versatile database service for a wide range of use cases. Additionally, NDB supports strong consistency and fast data access by caching frequently accessed data in memory, which helps reduce the number of disk reads and improves query performance.
3. NDB also provides automatic sharding, which helps ensure that your database can handle large amounts of data. You can use graph queries to analyze relationships between data in real-time, which can help you make more informed decisions. Furthermore, NDB offers high availability and fault tolerance through its distributed architecture and replication features. Lastly, NDB provides strong security features, including role-based access control, data encryption at rest, and network security features.

NDB Kubernetes Operator

The NDB Kubernetes Operator is an innovative tool created by Nutanix to streamline the management and operation of the Nutanix NDB (NoSQL database) on Kubernetes clusters.

With the NDB Kubernetes Operator, deploying and managing NDB clusters on Kubernetes has never been easier, as it eliminates the need to manually configure and manage the underlying infrastructure. Built on the Kubernetes operator framework, it offers a declarative way to manage the lifecycle of NDB clusters and other related resources.

One of the key benefits of the operator is that it simplifies the management of NDB clusters by automating common tasks, such as cluster creation, scaling, upgrading, backup, and recovery. It also offers a high degree of flexibility and customization, allowing you to configure various aspects of the cluster, such as storage, networking, and security.

Another advantage of the NDB Kubernetes Operator is its seamless integration with other Kubernetes tools and resources, such as Helm charts, Kubernetes secrets, and Kubernetes ConfigMaps. This integration makes it easy to integrate NDB into your existing Kubernetes-based infrastructure and workflows, providing a hassle-free solution for managing your database clusters.

Overall, the NDB Kubernetes Operator is a powerful and flexible tool for managing NDB clusters on Kubernetes, freeing you up to focus on your application logic rather than infrastructure management. Its automation capabilities and integration with other Kubernetes tools make it a must-have tool for developers and administrators looking to simplify and streamline their database management on Kubernetes.

Existing Architecture and Problem Statement

Problem Statement: Refactor models to keep profiles (software, compute, network, etc) as optional and use default if not specified

The NDB Kubernetes operator currently uses default compute, network and OS software profiles while provisioning the database. Refactor this module to include optional fields and only if absent, fall back to default.

NDB Architecture

Microsoft SQL Server, Oracle Database, PostgreSQL, MySQL, and MongoDB are just a few of the databases that can have high availability, scalability, and speed thanks to the distributed architecture of the Nutanix Database Service. The hyper-converged infrastructure from Nutanix, which offers a scalable and adaptable platform for handling enterprise workloads, is the foundation around which the architecture is built.

There are various layers in the architecture of the Nutanix Database Service. The Nutanix hyperconverged infrastructure is the basic layer that provides the storage, computing, and networking resources needed to run the databases. The Nutanix Acropolis operating system, which offers the essential virtualization and administration features, sits on top of this layer.

The Nutanix Era layer, which is located above the Nutanix Acropolis layer, offers the Nutanix Database Service the ability to manage databases throughout their existence. The Nutanix Era Manager, a centralized management console that offers a single point of access for controlling the databases across several clouds and data centers, is included in this tier.

The Nutanix Era Orchestrator, which is in charge of automating the provisioning, scaling, patching, and backup of the databases, is another component of the Nutanix Era layer. The Orchestrator offers a declarative approach for specifying the desired state of the databases and is built to work with a variety of databases.

The Nutanix Era Application, a web-based interface that enables database administrators and developers to quickly provision and administer the databases, is the final component of the top layer. A self-service interface for installing databases as well as a number of tools for tracking and troubleshooting database performance are offered by the Era Application.

Design & Workflow

Large amounts of data may be handled by the highly scalable, fault-tolerant, and consistent Nutanix NDB NoSQL database. It is a distributed database created to be installed over several cluster nodes. A portion of the data is stored on each node in the cluster, and the data is replicated across several nodes to guarantee high availability.

Configure your Nutanix cluster: We need to configure your Nutanix cluster to support NDB. This includes setting up the storage and network configurations, configuring the NDB nodes, and defining the replication factor.

Create a table: We need to create a table in NDB to store your data. This includes defining the schema, specifying the replication factor, and configuring any other options you need.

Write your code: We need to write your code to interact with the NDB cluster. This includes inserting and retrieving data, as well as performing more complex operations such as querying, indexing, and data aggregation.

Test your code: We need to test your code to ensure that it works as expected. This includes testing basic operations such as creating and retrieving data, as well as testing more complex operations such as queries and data aggregation.

Monitor your cluster: We need to monitor your NDB cluster to ensure that it is performing as expected. This includes monitoring resource usage, handling errors and exceptions, and optimizing performance.

Optimize your cluster: We need to optimize your NDB cluster over time to ensure that it continues to meet your needs. This includes tuning the configuration, optimizing queries, and scaling the cluster as needed.

Backup and recovery: We need to establish backup and recovery procedures to ensure that your data is protected against data loss or corruption. This includes regularly backing up your data, testing your backups, and establishing procedures for recovering data in case of a disaster.

Potential Design Patterns, Principles, and Code Refactoring strategies

The codebase could be converted into an Object Oriented fashion with classes. Further, here are some of the design patterns we could use:

Builder: This pattern could be used to create the provisioning request for a database instance in a more modular and flexible way. Rather than creating the request directly in one function, a builder class could be used to set individual properties of the request. This would make the code more maintainable and extensible, and would allow for easier testing of different combinations of request properties.

Factory: Since there are different types of database instances that can be provisioned (e.g. MySQL, Postgres, etc.), a factory pattern could be used to create the appropriate request object based on the specified database type. This would help to decouple the creation of the request object from the calling code, and would make it easier to add support for new database types in the future.

Dependency Injection: To allow for better testability, dependency injection can be used to decouple the code from its dependencies. For example, in the provided code snippet, the NDBClient is being passed into the GenerateProvisioningRequest() function. However, if the NDBClient had additional dependencies or if it were difficult to create a testable version of the NDBClient, dependency injection could be used to allow for easier testing and swapping of dependencies.

Code Refactoring: After reviewing the code base, it was discovered that the ndb_api_helpers.go file contains the code for provisioning the database. The main task of generating the request payload for provisioning the database is handled by the GenerateProvisioningRequest function. To retrieve all the profiles, this function utilizes the GetOOBProfiles function which returns a map of all the profiles. However, the current implementation of GenerateProvisioningRequest only retrieves the first element of the values within the map, which is assumed to be the default value. This means that the function doesn't verify if the user has provided a specific profile or not before assigning a default value.

To improve this behavior, we plan to iterate over all the profiles in the arrays that are inside the values of the profiles map. If the user has provided input for a specific profile, we will assign that input to the profiles variable. If not, we will use the first element of the array as the default value.

This change will allow us to properly check whether the user has provided input for a specific profile or not, and avoid the incorrect assumption that the first element of the values array is always the default value. By iterating over all the profiles, we can ensure that the correct profile is selected and assigned to the profiles variable. This will lead to more accurate and reliable database provisioning.

Modifications

\ndb-operator\api\v1alpha1\ndb_api_helpers.go =

Functions Changed

GenerateProvisioningRequest

previous working : This function generates and returns a request for provisioning a database (and a dbserver vm) on NDB and uses default compute, software, network, databaseParams profiles
enhanced working : This function generates and returns a request for provisioning a database (and a dbserver vm) on NDB and if user has provided custom profiles in "\ndb-operator\config\samples\ndb_v1alpha1_database.yaml", it will use those profiles to create the provisioning request or it will fall back to default profiles
previous code :

new code :

Explanation of the change : changed the name of GetOOBProfiles to EnrichAndGetProfiles due to added functionality of overriding default profile values with custom profiles read from YAML file after performing applicability checks

EnrichAndGetProfiles

previous working : previously this function was named GetOOBProfiles. This function used to fetch all the profiles from NDB API and return ProfilesMap with default profiles for each of the compute, software, network and dbParams profiles.
enhanced working : now this function fetches all the profiles from NDB API and populates ProfilesMap with default profiles for each of the compute, software, network and dbParams profiles. Then it calls function EnrichProfilesMap function which will populate ProfilesMap with custom profiles if there are any in the YAML file.
previous code :

new code :
File:File11png
Explanation of the change : since we only want to fall back to default profiles if there are no custom profiles mentioned in the YAML file, we are calling a new function EnrichProfilesMap which will populate ProfilesMap with the custom profiles.

EnrichProfilesMap

previous working : This function was not there previously.
enhanced working : This function checks if there are any custom profiles in the "\ndb-operator\config\samples\ndb_v1alpha1_database.yaml" file. If there any custom profiles, this function will call function PerformProfileMatchingAndEnrichProfiles to fetch them for each category (Compute, Software, Network, dbParams) and populate ProfilesMap with it.
previous code : N/A
new code :
File:File13png
Explanation of the change : Since we have added new section for custom profiles in the "\ndb-operator\config\samples\ndb_v1alpha1_database.yaml" file, we needed a function that will check if there is a section for custom profiles and delegate the task to fetch the custom profiles from the YAML file. This function fulfills that need.

PerformProfileMatchingAndEnrichProfiles

previous working : This function was not there previously.
enhanced working : Based on compute or (software, network & dbParam), generic or dbEngineSpecific profiles are used for matching the input customProfile. Furthermore, based on whether matched or not matched, delegation is performed to override the default profile values.
previous code : N/A
new code :
File:File15png
Explanation of the change : We want to check if custom profiles mentioned in the YAML file are valid or not. If the profile type is compute, this function calls another function to validate the custom profile with generic profiles. If the profile type is network/software/dbParams, this function calls another function to validate the custom profile with dbEngineSpecific profiles.

GetAppropriateProfileForType

previous working : This function was not there previously.
enhanced working : This functions gives either generic or dbEngine specific profiles based upon the profile type to be filtered upon.
previous code : N/A
new code :
File:File16png
Explanation of the change : This function is used by PerformProfileMatchingAndEnrichProfiles function to make the decision of what kind of profiles are to be matched with what type of profiles.

EnrichProfileMapForProfileType

previous working : This function was not there previously.
enhanced working : This function checks the correctness of the profile (response) passed as the parameter and overrides the profilesMap for the custom profile type
specified if the custom profile provided passes the checks. previous code : N/A
new code :
File:File17png
Explanation of the change : The custom profile mentioned in the YAML file is only valid if it exists in the list of all profiles provided by the NDB API. This function performs the task to check if the given custom profile exists in the all profiles list.

GetTopologyForProfileType

previous working : This function was not there previously.
enhanced working : Providing the least costly topology based on each profile type. previous code : N/A
new code :
File:File17png
Explanation of the change : Costlier topologies need more space on NDB Test Drive which results in "NoHostResources" error. So this function chooses the least costly topology to avoid this error.

Test Plan

Test Case Scenario 1

Test case name: Provisioning of appropriate database based on compute parameters

• • Description: This test case verifies that the appropriate database is provisioned based on the compute parameters passed as input, as expected.
  • Pre-conditions:
    • The system is set up and running
    • The compute parameters are available for input
  • Test steps:
    • Pass the compute parameters as input to the system
    • Check if the appropriate database has been provisioned based on the compute parameters
    • Verify that the database configuration and settings match the expected values based on the input parameters
  • Expected results:
    • The system provisions the appropriate database based on the compute parameters
    • The database configuration and settings match the expected values based on the input parameters
    • The test case passes successfully

Test Case Scenario 2

Test case name: Use of default setup for configuration when optional parameters are not passed

• • Description: This test case verifies that the database configured uses the default setup for configuration when optional parameters are not passed as input, as expected.
  • Pre-conditions:
    • The system is set up and running
    • No optional parameters are passed as input
  • Test steps:
    • Do not pass any optional parameters as input to the system
    • Check if the database is configured using the default setup for configuration
    • Verify that the database configuration and settings match the expected values based on the default setup
  • Expected results:
    • The system configures the database using the default setup when no optional parameters are passed
    • The database configuration and settings match the expected values based on the default setup
    • The test case passes successfully

Github

• Repo: https://github.com/karan-47/ndb-operator/tree/feature/ntnx_3

Mentors

• Prof. Edward F. Gehringer
• Krunal Jhaveri
• Manav Rajvanshi
• Krishna Saurabh Vankadaru
• Kartiki Bhandakkar

Contributors

• Karan Pradeep Gala (kgala2)
• Ashish Joshi (ajoshi24)
• Tilak Satra (trsatra)

References

[1] Nutanix. (n.d.). Nutanix Database Service. Retrieved from https://www.nutanix.com/products/database-service

[2] Kubernetes Operator Pattern https://kubernetes.io/docs/concepts/extend-kubernetes/operator

[3] NDB Operator Document - https://docs.google.com/document/d/1-VykKyIeky3n4JciIIrNgirk-Cn4pDT1behc9Yl8Nxk/

[4] Go Operator SDK - https://sdk.operatorframework.io/docs/buildingoperators/golang/tutorial/