CSC/ECE 517 Fall 2023 - NTX-3 Usability and Security

Kubernetes

Kuberneters, or K8s, is an open-source container orchestration platform that automates many functionalities that are useful to many developers and projects, including the automation of deployment, scaling, and management of containerized applications.

Kubernetes simplifies and streamlines the process of managing containerized applications through:

- Container Orchestration: This functionality allows the user to define how the containerized applications should run, ensuring that the desired state of the application is maintained.

- Automated Scaling: This allows Kubernetes to automatically scale applications based on certain metrics, which ensures that the services are always responsive and reliable.

- Self-healing: Kubernetes monitors the users applications health and will automatically replace or reschedule failed or unresponsive containers.

- Load Balancing: This functionality automatically balances loads coming to containers, ensuring an equal amount of workloads between all containers in an efficient fashion.

- Rollouts and Rollbacks: Kubernetes allows controlled updates to applications, which allow smooth rollouts of new versions as well as easy rollbacks if necessary.

These many functionalities allow many different users to create and manage many different containers in an easy and efficient way.

Nutanix Database Service

The Nutanix Database Service is a hybrid cloud database-as-a-service that allows users to efficiently manage their databases. It supports Microsoft SQL server, Oracle Database, PostgreSQL, MongoDB, and MySQL. Through its many features, users are able to provision new databases, automate tasks like patching and backups, and choose the right operating systems, database versions, and database extensions for their needs.The Nutanix Database Service allows customers from all over the world to simplify their databases on various locations, including on-premises, colocation sites, and public clouds.

The features of the NDB Service:

1. The NDB Service allows its users to manage the entire database lifecycle through provisioning, scaling, versioning, and patch automation.

2. Users can manage many databases across various locations, including on-premises, colocation sites, and public clouds from one control plane. The supported databases are Microsoft SQL Server, Oracle, PostgreSQL, MySQL, and MongoDB.

3. Using API integration coupled with infrastructure management and development tools allows users to provision the self-service database for both dev/test and production.

4. Users are able to roll out patches over either some of their databases, or their entire database, to protect against security threats. Users are also able to ensure compliance with regulatory requirements by restricting access to databases through role-based access controls.

Incorporating the NDB Kubernetes Operator empowers your organization with a seamless approach to database management. This automation streamlines your operations, fostering secure and efficient practices while boosting developer productivity. It also liberates your DBAs to focus on strategic endeavors, all while maintaining your desired control and compliance standards for a well-rounded and agile database ecosystem.

NDB Kubernetes Operator

The NDB Kubernetes Operator is an open-source tool that simplifies database administration, provisioning, and management within Kubernetes. It allows developers to provision PostgreSQL, MySQL, and MongoDB databases directly from their Kubernetes clusters, saving significant time and effort.

Developers often choose to use the Kubernetes operator to deploy both applications and databases for various reasons, such as:

1. Simplified Deployment: Kubernetes is widely used for deploying and managing applications, offering automation and scalability. However, databases like PostgreSQL, MySQL, and MongoDB are traditionally more complex to deploy and manage. The Kubernetes operator streamlines the deployment process, making it easier for developers.

2. Reduced Dependency on DBAs: Without the Kubernetes operator, developers might need to rely on database administrators (DBAs) to deploy databases. This process can be time-consuming, taking hours to days, and typically results in a one-time, manual deployment. The Kubernetes operator empowers developers to handle database provisioning independently.

3. Automation and Reproducibility: With the Kubernetes operator, developers can automate the deployment of their chosen databases, making it easy to replicate the process for various purposes, such as testing. This automation ensures consistency and saves time.

4. Flexibility with Multiple Database Engines: Developers often work with multiple database engines, and each may require a specific Kubernetes operator. The NDB Kubernetes Operator simplifies the process of finding, installing, configuring, and validating these operators, reducing the time and effort required.

5. Streamlined Database Management: Even after provisioning, databases need ongoing management tasks like backups, cloning, scaling, and patching. Using the Kubernetes operator can make these tasks more efficient, and it allows developers to handle these responsibilities without relying on separate DBAs.

The NDB Kubernetes Operator streamlines the deployment and management of databases within Kubernetes, providing developers with the automation and flexibility they need to efficiently work with various database engines while reducing their dependency on database administrators.

Problem Statement and Possible solutions

The current system faces several security and operational challenges that need urgent attention:

1. Securing Secrets: Currently, sensitive information is stored in plaintext, making it vulnerable to unauthorized access and potential breaches. We need to implement robust security measures such as Vault or External Secrets to ensure the secrets are securely managed and encrypted.

2. Access Control for NDB Resources: There is a need to establish a strict access control mechanism for NDB resources within the Kubernetes environment. This involves configuring ServiceAccounts, Roles, and related policies to restrict unauthorized access and ensure that only authorized users can interact with NDB resources.

3. Cluster Identification: The system currently uses cluster IDs as inputs, which can be confusing and error-prone. To enhance clarity and ease of use, we should change the input to cluster names and internally resolve these names to their corresponding IDs.

These challenges collectively demand immediate attention to enhance the system's security, access control, user experience, error handling, and maintainability. Enhancing the usability and security of the NDB Kubernetes Operator architecture involves implementing several key strategies and best practices. Here’s a comprehensive approach to address both aspects:

Usability Enhancements

1. Intuitive User Interface: Develop a user-friendly dashboard or command-line interface (CLI) that provides a clear overview of NDB clusters, their configurations, and status. Use descriptive language and visual aids to simplify complex operations.

2. Configuration Simplification: Implement wizards or configuration templates to guide users through the setup process. Allow for default configurations and provide detailed documentation to help users understand each configuration option.

3. Automated Scaling: Implement auto-scaling mechanisms based on workload demands. Automatically adjust the cluster size, storage, or compute resources to ensure optimal performance without user intervention.

4. Monitoring and Alerts: Integrate monitoring tools to provide real-time insights into cluster performance. Set up alerts for critical events or performance thresholds, ensuring users are promptly notified of any issues.

5. Comprehensive Documentation: Create extensive documentation, including tutorials, troubleshooting guides, and FAQs. Provide examples and use cases to assist users in understanding how to effectively use the NDB Kubernetes Operator.

Security Enhancements

1. Role-Based Access Control (RBAC): Implement RBAC within the Kubernetes cluster to restrict access based on roles and responsibilities. Define roles that limit access to NDB resources to only authorized personnel.

2. Network Policies: Configure network policies to control the communication between NDB clusters and other services within the Kubernetes cluster. Restricting network access enhances security by minimizing potential attack surfaces.

3. Secrets Management: Utilize Kubernetes Secrets or external secret management systems like Vault to securely store sensitive information such as database credentials and API keys. Avoid hardcoding secrets in configuration files.

4. Data Encryption: Enable data encryption at rest and in transit. Use encryption mechanisms provided by Kubernetes and NDB to safeguard data both within the cluster and during communication with external services.

5. Regular Security Audits: Conduct periodic security audits and vulnerability assessments to identify and address potential security weaknesses. Stay updated with security patches and follow best practices for secure configurations.

6. Authentication and Authorization: Implement strong authentication methods, such as LDAP or OIDC, to verify the identity of users accessing the system. Additionally, ensure that only authenticated users with the necessary permissions can perform sensitive operations.

7. Secure APIs: If the NDB Kubernetes Operator exposes APIs, ensure they are secured with authentication tokens and follow OAuth standards. Use API gateways to control access and monitor API usage.

By focusing on these usability and security enhancements, the NDB Kubernetes Operator can offer a seamless and secure experience to its users while ensuring the confidentiality, integrity, and availability of data and resources.

RBAC Example

Below is a code snippet as an example of implementing Role-Based Access Control (RBAC) in Kubernetes. This snippet demonstrates how to create a Role and RoleBinding in a Kubernetes cluster:

In this example:

• rbac-role.yaml defines a Role named ndb-operator-role that grants read permissions for Pods, Services, and ConfigMaps within the specified namespace.
• rbac-rolebinding.yaml establishes a RoleBinding that associates the ndb-operator-role with a specific ServiceAccount (your-service-account) within the same namespace.

Enhancements to Secrets Management

Implemented Solution from Phase II

While the earlier proposed solution was to use an external vault service like Hashicorp or AWS Vault, it was determined that these external services are generally licensed and involve a fee to use them. Instead we have implemented a solution using RBAC policies at the Kubernetes cluster level to create new users, and restrict their secrets only to their context, such that no two users in the same K8 cluster can access their secrets. To better understand the solution here is a diagram of the cluster level implementation.

As per the above diagram, there are two main contexts namely the administrator context (Usually known as “minikube” or any other name depending on the K8 cluster), and the user contexts. The administrator has the role of creating users and assigning their NDB secrets, while the users have the role to view their secrets and also depending on the access they have been granted, they can perform certain NDB operations as well.

In addition to the contexts, there are three namespaces namely the administrator namespace, the user namespaces and the ndb-operator-namespace. Each of their privileges are described as follows:

• Administrator Namespace : Via the administrator namespace, an admin has access to all NDB resources as well as the user's secrets. Thus the administrator role should only be granted to trusted individuals with regular audits to ensure integrity and privacy.

• User Namespace : The user namespace stores the secrets of the given user and are only accessible within the confines of their respective namespace. By segregating the users into different namespaces, it can be ensured that no two users can access each other’s namespaces and thus prevent one from accessing other’s secrets. Moreover, each user in their respective namespace also can be granted access to certain NDB resources by the administrator, such as a manager (can view and manage NDB resources) or a viewer (can only view the NDB resources but doesn’t have privileges to create or destroy the same).

• NDB Operator Namespace: This namespace contains most of the NDB resources that are used to clone and provision databases. A user can be granted access to these resources by the appropriate RBAC rules set by the administrator for that particular user.

Steps for recreation of the implementation

The process of creating a user, restricting access to secrets and NDB resources is demonstrated below.

1. With the K8 cluster set up and set in the default context that has access to all privileges within the cluster, a client certificate needs to be created for each user by issuing the following commands from the terminal:

                openssl genpkey -algorithm RSA -out user1-key.pem
                openssl req -new -key user1-key.pem -out user1.csr -subj “/CN=user1”

Once the certificate signing requests have been created for each user, these have to be approved by the cluster certificates. The command to perform the same is as follows -

                openssl x509 -req -in user1.csr -CA /path_to_cluster_ca_crt/ca.crt 
                -CAkey /path_to_cluster_ca_crt/ca.key -CAcreateserial -out user1.crt

2. With the certificate signing requests completed, the next step is to create the users and their respective contexts. This can be done by issuing the following commands -

                kubectl config set-credentials user1 –client-certificate=user1.crt –client-key=user1.pem
                kubectl create namespace user1-namespace
                kubectl config set-context user1-context –cluster=<cluster-name> –namespace=user1-namespace –user=user1

3. Now the cluster role bindings need to be created for each user. To do this create two YAML files for each user. One YAML file is used to define the role of the user within the cluster and another one is used to bind the same to the cluster. Note, each of these roles need to be in the created user’s namespace.

(a)user1-namespace-role.yaml

(b)user1-namespace-rolebinding.yaml

With these roles created, save the YAML files and apply the same using the commands

                kubectl apply -f user-namespace-role.yaml
                kubectl apply -f user-namespace-rolebinding.yaml

4. Now we need to create another YAML file which will allow the user to be bound to only certain NDB resources. Some of these roles are - ndb-operator-manager-role and ndb-operator-viewer-role. The functionality of these roles have been discussed in the sections above.

user1-ndb-rolebinding.yaml

Apply the cluster role using the command:

                kubectl apply -f user1-ndb-rolebinding.yaml

5. Finally we need to create the secrets for each user. To create the secret for a user, we can create the following YAML files and issue the commands to create them:

Save this file as user1-secret and apply the same using the command - kubectl apply -f user1-secret.yaml Similarly create the database secret using the YAML file :

6. With the secrets created, we need to secure them using RBAC such that only the intended user has access to the same. To do this we need to create a role using the following YAML file. Call this as user1-role.yaml

Save and apply the same using kubectl apply.

7.With the users and secrets created, we can now switch to a user context, and try accessing secrets. It will be observed that it can only access secrets using the kubectl get secrets <secret-name> command only for the secrets it has been granted access to in the previous file.

Thus with the aid of the above steps, we can now create a secret and secure the same using RBAC. Moreover, a particular user can only apply the secrets that they have been granted access to using an appropriate manifest and not any other secret that belongs to another user.

Test Plan

Secrets Management and Access Control System

To test the chosen secrets management and access control system, various users will need to be set up and configured, assigned secrets in their respective namespace and RBAC roles need to be applied.

Test	Test Description
Verify Secret Creation and Access to Other Secrets	To verify that a secret has been created and it exists in the user’s namespace, the user can issue the command in its respective context - kubectl get secrets <user-secret-name>. It should return only for secrets it has access to and not any other secret.
Verify Only Correct Secrets Are Being Applied	To verify that a given secret is correctly being applied, the user can create their respective ndb-manifest and db-manifest YAML files and try to apply the same by using kubectl apply. It will be seen that a user can only apply secrets that it has access to and if they try to use a secret they will not have access to, it would result in the ndb-operator to give an authentication failed error.

Identification of a cluster by using clusterName instead of clusterId

The earlier ndb-operator would require the cluster id of the test drive on which they would wish to either provision or clone their databases. The cluster id was in the UUID format and this would change every time a new test drive is used. However, it is observed that irrespective of the cluster id the cluster name remains constant and would be a better identifier to use in order to connect to the cluster. (a) In order to implement this, the following changes were made to the db manifest clusterId and clusterName are now optional arguments. The user must either enter the clusterId or the clusterName. In the event that the user enters both, it will use the clusterId instead of the clusterName and connect to the respective cluster.

(b) Code changes - It can be viewed in the following github link : https://github.com/nutanix-cloud-native/ndb-operator/pull/185/commits/49cb90fb1443e05317c3ba151e4f8fc2aa9a0e4e

Mentors

• Nandini Mundra <nmundra@ncsu.edu>
• Manav Rajvanshi <manav.rajvanshi@nutanix.com>
• Mazin Shaaeldin <mazin.shaaeldin@nutanix.com>

Team

• Rashmi Datta <rdatta2@ncsu.edu>
• Swastik Ghosh <sghosh25@ncsu.edu>
• Reuben Vandezande <rcvandez@ncsu.edu>

Relevant Links

GitHub repository: https://github.com/RashmiD25/ndb-operator

Pull Request: https://github.com/nutanix-cloud-native/ndb-operator/pull/185

References

[1] “Nutanix Database Service,” Nutanix, https://www.nutanix.com/products/database-service#features (accessed Oct. 23, 2023).

[2] Nutanix (2023). Ndb-operator (Version 0.0.7) [Source code]. https://github.com/nutanix-cloud-native/ndb-operator.

[3] “Introduction: Vault: HashiCorp developer,” Introduction | Vault | HashiCorp Developer, https://developer.hashicorp.com/vault/docs/what-is-vault (accessed Oct. 23, 2023).

[4] “API overview,” Overview - External Secrets Operator, https://external-secrets.io/latest/introduction/overview/ (accessed Oct. 23, 2023).

[5] Kubernetes documentation on Role-Based Access Control. https://kubernetes.io/docs/reference/access-authn-authz/rbac/

[6] Use of LLMs(e.g; bard.ai and chat.openai) for content enhancement