CSC/ECE 517 Fall 2023 - NTX-3 Usability and Security

Kubernetes

Kuberneters, or K8s, is an open-source container orchestration platform that automates many functionalities that are useful to many developers and projects, including the automation of deployment, scaling, and management of containerized applications.

Kubernetes simplifies and streamlines the process of managing containerized applications through:

- Container Orchestration: This functionality allows the user to define how the containerized applications should run, ensuring that the desired state of the application is maintained.

- Automated Scaling: This allows Kubernetes to automatically scale applications based on certain metrics, which ensures that the services are always responsive and reliable.

- Self-healing: Kubernetes monitors the users applications health and will automatically replace or reschedule failed or unresponsive containers.

- Load Balancing: This functionality automatically balances loads coming to containers, ensuring an equal amount of workloads between all containers in an efficient fashion.

- Rollouts and Rollbacks: Kubernetes allows controlled updates to applications, which allow smooth rollouts of new versions as well as easy rollbacks if necessary.

These many functionalities allow many different users to create and manage many different containers in an easy and efficient way.

Nutanix Database Service

The Nutanix Database Service is a hybrid cloud database-as-a-service that allows users to efficiently manage their databases. It supports Microsoft SQL server, Oracle Database, PostgreSQL, MongoDB, and MySQL. Through its many features, users are able to provision new databases, automate tasks like patching and backups, and choose the right operating systems, database versions, and database extensions for their needs.The Nutanix Database Service allows customers from all over the world to simplify their databases on various locations, including on-premises, colocation sites, and public clouds.

The features of the NDB Service:

1. The NDB Service allows its users to manage the entire database lifecycle through provisioning, scaling, versioning, and patch automation.

2. Users can manage many databases across various locations, including on-premises, colocation sites, and public clouds from one control plane. The supported databases are Microsoft SQL Server, Oracle, PostgreSQL, MySQL, and MongoDB.

3. Using API integration coupled with infrastructure management and development tools allows users to provision the self-service database for both dev/test and production.

4. Users are able to roll out patches over either some of their databases, or their entire database, to protect against security threats. Users are also able to ensure compliance with regulatory requirements by restricting access to databases through role-based access controls.

Incorporating the NDB Kubernetes Operator empowers your organization with a seamless approach to database management. This automation streamlines your operations, fostering secure and efficient practices while boosting developer productivity. It also liberates your DBAs to focus on strategic endeavors, all while maintaining your desired control and compliance standards for a well-rounded and agile database ecosystem.

NDB Kubernetes Operator

The NDB Kubernetes Operator is an open-source tool that simplifies database administration, provisioning, and management within Kubernetes. It allows developers to provision PostgreSQL, MySQL, and MongoDB databases directly from their Kubernetes clusters, saving significant time and effort.

Developers often choose to use the Kubernetes operator to deploy both applications and databases for various reasons, such as:

1. Simplified Deployment: Kubernetes is widely used for deploying and managing applications, offering automation and scalability. However, databases like PostgreSQL, MySQL, and MongoDB are traditionally more complex to deploy and manage. The Kubernetes operator streamlines the deployment process, making it easier for developers.

2. Reduced Dependency on DBAs: Without the Kubernetes operator, developers might need to rely on database administrators (DBAs) to deploy databases. This process can be time-consuming, taking hours to days, and typically results in a one-time, manual deployment. The Kubernetes operator empowers developers to handle database provisioning independently.

3. Automation and Reproducibility: With the Kubernetes operator, developers can automate the deployment of their chosen databases, making it easy to replicate the process for various purposes, such as testing. This automation ensures consistency and saves time.

4. Flexibility with Multiple Database Engines: Developers often work with multiple database engines, and each may require a specific Kubernetes operator. The NDB Kubernetes Operator simplifies the process of finding, installing, configuring, and validating these operators, reducing the time and effort required.

5. Streamlined Database Management: Even after provisioning, databases need ongoing management tasks like backups, cloning, scaling, and patching. Using the Kubernetes operator can make these tasks more efficient, and it allows developers to handle these responsibilities without relying on separate DBAs.

The NDB Kubernetes Operator streamlines the deployment and management of databases within Kubernetes, providing developers with the automation and flexibility they need to efficiently work with various database engines while reducing their dependency on database administrators.

Problem Statement and Possible solutions

Problem Statement: Usability and Security enhancements for NDB Kubernetes Operator The current system faces several security and operational challenges that need urgent attention:

1. Securing Secrets: Currently, sensitive information is stored in plaintext, making it vulnerable to unauthorized access and potential breaches. We need to implement robust security measures such as Vault or External Secrets to ensure the secrets are securely managed and encrypted.

2. Access Control for NDB Resources: There is a need to establish a strict access control mechanism for NDB resources within the Kubernetes environment. This involves configuring ServiceAccounts, Roles, and related policies to restrict unauthorized access and ensure that only authorized users can interact with NDB resources.

3. Cluster Identification: The system currently uses cluster IDs as inputs, which can be confusing and error-prone. To enhance clarity and ease of use, we should change the input to cluster names and internally resolve these names to their corresponding IDs.

4. Logging Failed API Responses: In the event of failed interactions with the NDB API, it's crucial to log detailed error messages alongside the response status. This enhanced logging will provide valuable insights into the nature of failures, aiding in quicker issue resolution and system improvement.

5. Refactoring NDBClient and API: The current architecture might lack flexibility and scalability. To address this, we need to refactor the NDBClient and possibly the NDB API to adhere to an interface. This abstraction will allow for easier integration of new features, improved testing, and overall system modularity.

These challenges collectively demand immediate attention to enhance the system's security, access control, user experience, error handling, and maintainability.

Enhancing the usability and security of the NDB Kubernetes Operator architecture involves implementing several key strategies and best practices. Here’s a comprehensive approach to address both aspects:

Enhancements to Secrets Management

One of the most important parts of software architecture is making sure secret information, that being passwords, data keys, credentials, and any other information that can be used to maliciously gain unauthorized access to various systems, are hidden and secure in a way that prevents malicious entities from obtaining them.

The above screenshot shows how secrets are currently stored in the NDB Kubernetes Operator. They are currently held freely in a file that can be accessed by anyone with access to it. There are various solutions to this problem, but the best proposed solutions are integrating HashiCorp Vault or integrating the Kubernetes External Secrets Operator.

HashiCorp Vault and its Tradeoffs

HashiCorp Vault is a secrets and encryption management system that uses authentication and authorization to control who has access to the secrets. It uses UI, CLI, or HTTP API to store and manage, restrict, and audit sensitive data securely.

Vault uses tokens associated with a user’s, or client’s, policy. A policy’s rules constrains the actions and accessibility for each client. Vault will ensure that anyone accessing secrets is authenticated and will prevent users from accessing secrets they should not have access to. Vault validates users against other third-party services, including Github, which will allow it to fit seamlessly with the NDB Kubernetes Operator.

HashiCorp Vault also provides the ability to control access to resources using its fine-grained access control. Users will be able to access resources that they are supposed to be able to access, while resources they aren’t can be configured to disallow their access.

An issue with HashiCorp Vault, however, is that it can be intensive on the system. Maintaining Vault clusters may require a lot of computational resources, taking away from the other K8s clusters on the system. It will also require some time put into configuring its authentication methods which may prove to be time consuming compared to other solutions.

External Secrets Operator and its Tradeoffs

Another potential solution is using the Kubernetes External Secrets Operator. This operator integrates many other secret management systems, including the HashiCorp Vault as well as AWS Secrets Manager, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, and CyberArk Conjur. The External Secrets Operator reads information from these external secret management systems and injects it into a Kubernetes Secret.

The above image shows the high level architecture of the External Secrets Operator. However, the problem with this solution is that it requires using an external secret management system for this operator to retrieve and inject into a Kubernetes secret. Currently, an external secret manager is not in use, so this solution will not be as effective as the previous solution in the NDB Kubernetes Operator in its current state.

Additionally, this operator does not have the ability to control and restrict access to Kubernetes resources. This function can be implemented through Kubernetes’ built-in mechanism, Role-Based Access Control (RBAC). This allows the ability to define access control policies that specify which users have permissions to perform actions on various Kubernetes resources, including secrets.

Proposed Solution

From the above potential solutions, the HashiCorp Vault solution seems to be the best option. It has both the ability to securely store secrets as well as built-in access control to prevent users from accessing something they shouldn’t. It provides both the functionality that is required by this project, whereas the External Secrets K8s Operator requires additional secret management systems to function. If this solution is chosen, something similar to the HashiCorp Vault will need to be integrated into the system anyway, making the External Secrets Operator less useful when integrating the first secret management system into the architecture. Additionally, another system, RBAC, will need to be implemented for access control as well, which HashiCorp Vault already provides.

Test Plan

Secrets Management and Access Control System To test the chosen secrets management and access control system, various users will need to be set up and configured. Those users will be given different access permissions to various fake secrets that will be created to make sure access is restricted. Below are more specific tests that will be created to test access to secrets.

• Ensure the Secret Management System is Installed:

This test will ensure that the chosen secret management system has been successfully installed on the system. It will call various functions of the management system to ensure that it is present.

• Verify Configuration:

This test’s main purpose is to ensure that the configuration of the system is correct. It will check the desired authentication methods and policies by using various users using those authentication methods.

• Verify Secret Creation:

Fake secrets will be created and will be tested to ensure that only verified users can access them. This test will also ensure that unauthorized users cannot create secrets.

• Verify Retrieval of Secrets:

The fake secrets will be retrieved both inside the system’s storage through its database as well as through an interface. This test will also ensure that users cannot retrieve secrets they are not supposed to.

• Verify System Retrieval of Secrets:

This test will ensure that authorized workloads within the system are able to access secrets that they are required to access. This test will also ensure that unauthorized workloads are unable to access certain secrets.

• Verify encryption at all Levels:

Tests will be created to ensure that secrets that are being transferred from one component to another are encrypted and are inaccessible without the proper authorization.

• Verify Proper Access Control:

Tests will be created to ensure that unauthorized users cannot access resources they shouldn’t. Fake resources and fake users will be created with different authorization to ensure this. It will also ensure that authorized users can access what they should be able to.

References

[1] “Nutanix Database Service,” Nutanix, https://www.nutanix.com/products/database-service#features (accessed Oct. 23, 2023).

[2] Nutanix (2023). Ndb-operator (Version 0.0.7) [Source code]. https://github.com/nutanix-cloud-native/ndb-operator.

[3] “Introduction: Vault: HashiCorp developer,” Introduction | Vault | HashiCorp Developer, https://developer.hashicorp.com/vault/docs/what-is-vault (accessed Oct. 23, 2023).

[4] “API overview,” Overview - External Secrets Operator, https://external-secrets.io/latest/introduction/overview/ (accessed Oct. 23, 2023).

[5] Kubernetes documentation on Role-Based Access Control. https://kubernetes.io/docs/reference/access-authn-authz/rbac/