CSC/ECE 517 Fall 2023 - NTX-3 Usability and Security

Kubernetes

Kuberneters, or K8s, is an open-source container orchestration platform that automates many functionalities that are useful to many developers and projects, including the automation of deployment, scaling, and management of containerized applications.

Kubernetes simplifies and streamlines the process of managing containerized applications through:

- Container Orchestration: This functionality allows the user to define how the containerized applications should run, ensuring that the desired state of the application is maintained.

- Automated Scaling: This allows Kubernetes to automatically scale applications based on certain metrics, which ensures that the services are always responsive and reliable.

- Self-healing: Kubernetes monitors the users applications health and will automatically replace or reschedule failed or unresponsive containers.

- Load Balancing: This functionality automatically balances loads coming to containers, ensuring an equal amount of workloads between all containers in an efficient fashion.

- Rollouts and Rollbacks: Kubernetes allows controlled updates to applications, which allow smooth rollouts of new versions as well as easy rollbacks if necessary.

These many functionalities allow many different users to create and manage many different containers in an easy and efficient way.

Nutanix Database Service

The Nutanix Database Service is a hybrid cloud database-as-a-service that allows users to efficiently manage their databases. It supports Microsoft SQL server, Oracle Database, PostgreSQL, MongoDB, and MySQL. Through its many features, users are able to provision new databases, automate tasks like patching and backups, and choose the right operating systems, database versions, and database extensions for their needs.The Nutanix Database Service allows customers from all over the world to simplify their databases on various locations, including on-premises, colocation sites, and public clouds.

The features of the NDB Service:

1. The NDB Service allows its users to manage the entire database lifecycle through provisioning, scaling, versioning, and patch automation.

2. Users can manage many databases across various locations, including on-premises, colocation sites, and public clouds from one control plane. The supported databases are Microsoft SQL Server, Oracle, PostgreSQL, MySQL, and MongoDB.

3. Using API integration coupled with infrastructure management and development tools allows users to provision the self-service database for both dev/test and production.

4. Users are able to roll out patches over either some of their databases, or their entire database, to protect against security threats. Users are also able to ensure compliance with regulatory requirements by restricting access to databases through role-based access controls.

Incorporating the NDB Kubernetes Operator empowers your organization with a seamless approach to database management. This automation streamlines your operations, fostering secure and efficient practices while boosting developer productivity. It also liberates your DBAs to focus on strategic endeavors, all while maintaining your desired control and compliance standards for a well-rounded and agile database ecosystem.

NDB Kubernetes Operator

The NDB Kubernetes Operator is an open-source tool that simplifies database administration, provisioning, and management within Kubernetes. It allows developers to provision PostgreSQL, MySQL, and MongoDB databases directly from their Kubernetes clusters, saving significant time and effort.

Developers often choose to use the Kubernetes operator to deploy both applications and databases for various reasons, such as:

1. Simplified Deployment: Kubernetes is widely used for deploying and managing applications, offering automation and scalability. However, databases like PostgreSQL, MySQL, and MongoDB are traditionally more complex to deploy and manage. The Kubernetes operator streamlines the deployment process, making it easier for developers.

2. Reduced Dependency on DBAs: Without the Kubernetes operator, developers might need to rely on database administrators (DBAs) to deploy databases. This process can be time-consuming, taking hours to days, and typically results in a one-time, manual deployment. The Kubernetes operator empowers developers to handle database provisioning independently.

3. Automation and Reproducibility: With the Kubernetes operator, developers can automate the deployment of their chosen databases, making it easy to replicate the process for various purposes, such as testing. This automation ensures consistency and saves time.

4. Flexibility with Multiple Database Engines: Developers often work with multiple database engines, and each may require a specific Kubernetes operator. The NDB Kubernetes Operator simplifies the process of finding, installing, configuring, and validating these operators, reducing the time and effort required.

5. Streamlined Database Management: Even after provisioning, databases need ongoing management tasks like backups, cloning, scaling, and patching. Using the Kubernetes operator can make these tasks more efficient, and it allows developers to handle these responsibilities without relying on separate DBAs.

The NDB Kubernetes Operator streamlines the deployment and management of databases within Kubernetes, providing developers with the automation and flexibility they need to efficiently work with various database engines while reducing their dependency on database administrators.

Problem Statement and Possible solutions

The current system faces several security and operational challenges that need urgent attention:

1. Securing Secrets: Currently, sensitive information is stored in plaintext, making it vulnerable to unauthorized access and potential breaches. We need to implement robust security measures such as Vault or External Secrets to ensure the secrets are securely managed and encrypted.

2. Access Control for NDB Resources: There is a need to establish a strict access control mechanism for NDB resources within the Kubernetes environment. This involves configuring ServiceAccounts, Roles, and related policies to restrict unauthorized access and ensure that only authorized users can interact with NDB resources.

3. Cluster Identification: The system currently uses cluster IDs as inputs, which can be confusing and error-prone. To enhance clarity and ease of use, we should change the input to cluster names and internally resolve these names to their corresponding IDs.

4. Logging Failed API Responses: In the event of failed interactions with the NDB API, it's crucial to log detailed error messages alongside the response status. This enhanced logging will provide valuable insights into the nature of failures, aiding in quicker issue resolution and system improvement.

5. Refactoring NDBClient and API: The current architecture might lack flexibility and scalability. To address this, we need to refactor the NDBClient and possibly the NDB API to adhere to an interface. This abstraction will allow for easier integration of new features, improved testing, and overall system modularity.

These challenges collectively demand immediate attention to enhance the system's security, access control, user experience, error handling, and maintainability. Enhancing the usability and security of the NDB Kubernetes Operator architecture involves implementing several key strategies and best practices. Here’s a comprehensive approach to address both aspects:

Usability Enhancements

1. Intuitive User Interface: Develop a user-friendly dashboard or command-line interface (CLI) that provides a clear overview of NDB clusters, their configurations, and status. Use descriptive language and visual aids to simplify complex operations.

2. Configuration Simplification: Implement wizards or configuration templates to guide users through the setup process. Allow for default configurations and provide detailed documentation to help users understand each configuration option.

3. Automated Scaling: Implement auto-scaling mechanisms based on workload demands. Automatically adjust the cluster size, storage, or compute resources to ensure optimal performance without user intervention.

4. Monitoring and Alerts: Integrate monitoring tools to provide real-time insights into cluster performance. Set up alerts for critical events or performance thresholds, ensuring users are promptly notified of any issues.

5. Comprehensive Documentation: Create extensive documentation, including tutorials, troubleshooting guides, and FAQs. Provide examples and use cases to assist users in understanding how to effectively use the NDB Kubernetes Operator.

Security Enhancements

1. Role-Based Access Control (RBAC): Implement RBAC within the Kubernetes cluster to restrict access based on roles and responsibilities. Define roles that limit access to NDB resources to only authorized personnel.

2. Network Policies: Configure network policies to control the communication between NDB clusters and other services within the Kubernetes cluster. Restricting network access enhances security by minimizing potential attack surfaces.

3. Secrets Management: Utilize Kubernetes Secrets or external secret management systems like Vault to securely store sensitive information such as database credentials and API keys. Avoid hardcoding secrets in configuration files.

4. Data Encryption: Enable data encryption at rest and in transit. Use encryption mechanisms provided by Kubernetes and NDB to safeguard data both within the cluster and during communication with external services.

5. Regular Security Audits: Conduct periodic security audits and vulnerability assessments to identify and address potential security weaknesses. Stay updated with security patches and follow best practices for secure configurations.

6. Authentication and Authorization: Implement strong authentication methods, such as LDAP or OIDC, to verify the identity of users accessing the system. Additionally, ensure that only authenticated users with the necessary permissions can perform sensitive operations.

7. Secure APIs: If the NDB Kubernetes Operator exposes APIs, ensure they are secured with authentication tokens and follow OAuth standards. Use API gateways to control access and monitor API usage.

By focusing on these usability and security enhancements, the NDB Kubernetes Operator can offer a seamless and secure experience to its users while ensuring the confidentiality, integrity, and availability of data and resources.

RBAC Example

Below is a code snippet as an example of implementing Role-Based Access Control (RBAC) in Kubernetes. This snippet demonstrates how to create a Role and RoleBinding in a Kubernetes cluster:

In this example:

• rbac-role.yaml defines a Role named ndb-operator-role that grants read permissions for Pods, Services, and ConfigMaps within the specified namespace.
• rbac-rolebinding.yaml establishes a RoleBinding that associates the ndb-operator-role with a specific ServiceAccount (your-service-account) within the same namespace.

Enhancements to Secrets Management

Implemented Solution

While the earlier proposed solution was to use an external vault service like Hashicorp or AWS Vault, it was determined that these external services are generally licensed and involve a fee to use them. Instead we have implemented a solution using RBAC policies at the Kubernetes cluster level to create new users, and restrict their secrets only to their context, such that no two users in the same K8 cluster can access their secrets. To better understand the solution here is a diagram of the cluster level implementation.

As per the above diagram, there are two main contexts namely the administrator context (Usually known as “minikube” or any other name depending on the K8 cluster), and the user contexts. The administrator has the role of creating users and assigning their NDB secrets, while the users have the role to view their secrets and also depending on the access they have been granted, they can perform certain NDB operations as well.

In addition to the contexts, there are three namespaces namely the administrator namespace, the user namespaces and the ndb-operator-namespace. Each of their privileges are described as follows:

• Administrator Namespace : Via the administrator namespace, an admin has access to all NDB resources as well as the user's secrets. Thus the administrator role should only be granted to trusted individuals with regular audits to ensure integrity and privacy.

• User Namespace : The user namespace stores the secrets of the given user and are only accessible within the confines of their respective namespace. By segregating the users into different namespaces, it can be ensured that no two users can access each other’s namespaces and thus prevent one from accessing other’s secrets. Moreover, each user in their respective namespace also can be granted access to certain NDB resources by the administrator, such as a manager (can view and manage NDB resources) or a viewer (can only view the NDB resources but doesn’t have privileges to create or destroy the same).

• NDB Operator Namespace: This namespace contains most of the NDB resources that are used to clone and provision databases. A user can be granted access to these resources by the appropriate RBAC rules set by the administrator for that particular user.

Proposed Solution

Test Plan

Secrets Management and Access Control System: To test the chosen secrets management and access control system, various users will need to be set up and configured. Those users will be given different access permissions to various fake secrets that will be created to make sure access is restricted. Below are more specific tests that will be created to test access to secrets.

Test	Test Description
Ensure the Secret Management System is Installed	This test will ensure that the chosen secret management system has been successfully installed on the system. It will call various functions of the management system to ensure that it is present.
Verify Configuration	This test’s main purpose is to ensure that the configuration of the system is correct. It will check the desired authentication methods and policies by using various users using those authentication methods.
Verify Secret Creation	Fake secrets will be created and will be tested to ensure that only verified users can access them. This test will also ensure that unauthorized users cannot create secrets.
Verify Retrieval of Secrets	The fake secrets will be retrieved both inside the system’s storage through its database as well as through an interface. This test will also ensure that users cannot retrieve secrets they are not supposed to.
Verify System Retrieval of Secrets	This test will ensure that authorized workloads within the system are able to access secrets that they are required to access. This test will also ensure that unauthorized workloads are unable to access certain secrets.
Verify encryption at all levels	Tests will be created to ensure that secrets that are being transferred from one component to another are encrypted and are inaccessible without the proper authorization.
Verify Proper Access Control	Tests will be created to ensure that unauthorized users cannot access resources they shouldn’t. Fake resources and fake users will be created with different authorization to ensure this. It will also ensure that authorized users can access what they should be able to.

Mentors

• Nandini Mundra <nmundra@ncsu.edu>
• Manav Rajvanshi <manav.rajvanshi@nutanix.com>
• Mazin Shaaeldin <mazin.shaaeldin@nutanix.com>

Team

• Rashmi Datta <rdatta2@ncsu.edu>
• Swastik Ghosh <sghosh25@ncsu.edu>
• Reuben Vandezande <rcvandez@ncsu.edu>

Github Repo

NDB Operator

References

[1] “Nutanix Database Service,” Nutanix, https://www.nutanix.com/products/database-service#features (accessed Oct. 23, 2023).

[2] Nutanix (2023). Ndb-operator (Version 0.0.7) [Source code]. https://github.com/nutanix-cloud-native/ndb-operator.

[3] “Introduction: Vault: HashiCorp developer,” Introduction | Vault | HashiCorp Developer, https://developer.hashicorp.com/vault/docs/what-is-vault (accessed Oct. 23, 2023).

[4] “API overview,” Overview - External Secrets Operator, https://external-secrets.io/latest/introduction/overview/ (accessed Oct. 23, 2023).

[5] Kubernetes documentation on Role-Based Access Control. https://kubernetes.io/docs/reference/access-authn-authz/rbac/