Build it Live: Managing Secrets with the Application Study Tool

Key Concepts

• Application Studies Tool (AST): A tool for collecting and analyzing application performance data.
• HashiCorp Vault: A tool for managing secrets and sensitive data.
• Grafana: A data visualization and monitoring tool.
• Prometheus: A time-series database used for storing metrics.
• Docker: A platform for containerizing applications.
• Docker Compose: A tool for defining and running multi-container Docker applications.
• TLS/SSL: Protocols for encrypting communication over a network.
• App Role: A type of authentication method in HashiCorp Vault.
• Kubernetes: An open-source container orchestration system.
• YUbuntu: A Linux distribution.
• Docker Secrets: A feature of Docker for managing sensitive data within containers.

Building AST "As-Is"

• The initial goal is to build the Application Studies Tool (AST) with default settings.
• The process starts by cloning a Git repository containing the necessary configuration files.
• An .env file is created by copying an example file, and a secrets file is also copied.
• The defaults.yaml file is edited to configure the device to be scraped, including the big IP address.
• The speaker mentions that many people don't want to store their big IP password in a clear text file.
• Mike Walker, the product manager for AST, joins the stream to answer questions.
• By default, AST collects LTM (Local Traffic Manager) data. Other modules like DNS/GTM or ASM are considered add-ons.
• The speaker clarifies that the system will only be monitoring a big IP local, so no changes to the default configuration are needed.
• The big IP address is configured in the defaults.yaml file.
• The speaker and Mike discuss the default credentials for Grafana (admin/admin) and the ability to change them.
• Mike explains how to obtain tokens and IDs for sharing metrics to the XC data fabric by contacting the account team.
• The speaker then executes docker compose up to start the AST containers.
• Mike explains that docker compose up runs in connected mode, showing logs, while docker compose up -d runs in disconnected mode.
• The containers that spin up are hotel collector, Grafana, and Prometheus.
• Mike mentions that there are plans for additional containers in the future to expand the tool's features.
• The speaker verifies that the containers are running using docker ps -a.
• Prometheus is accessible on port 9090, and Grafana is accessible on port 3000.
• Mike emphasizes that users can access both Prometheus and Grafana to gather and visualize metrics.
• The default retention period for metrics in Prometheus is one year, but this can be customized.

Converting Grafana to TLS

• The next step is to convert Grafana to use TLS/SSL for secure communication.
• Jason Epstein joins the stream to guide the TLS configuration process.
• The process involves generating a self-signed certificate using OpenSSL.
• The speaker executes OpenSSL commands to generate a key and a certificate signing request (CSR).
• The CSR is then used to generate a self-signed certificate.
• The permissions of the certificate and key files are changed to 440, making them readable by root and the owner group.
• A new file, grafana.ini, is created in the /services/grafana/grafana directory to configure Grafana's TLS settings.
• The grafana.ini file is configured with the paths to the certificate and key files.
• The speaker and Jason discuss the root_url setting in grafana.ini and its relevance when domain enforcement is not enabled.
• The Docker Compose file is modified to mount the directory containing the certificate and key files into the Grafana container.
• The existing containers are stopped using docker compose down.
• The Docker Compose file is edited to remove the provisioning mount, allowing access to the certificate and key files.
• The containers are restarted using docker compose up.
• The speaker verifies that Grafana is now accessible over HTTPS on port 3000, displaying a browser warning due to the self-signed certificate.
• Jason suggests that Prometheus can be made inaccessible for general use, as it's primarily used for debugging.
• Jason explains that Grafana is the visualization tool, while Prometheus stores the data.
• Jason mentions that organizations can integrate AST metrics into their existing Grafana dashboards.
• Jason is working on porting AST to Kubernetes for easier deployment in containerized environments.

Using Vault for Secrets Management

• The final goal is to integrate HashiCorp Vault for managing secrets, specifically the big IP credentials.
• Michael Olirri's article on Dev Central is referenced as a guide.
• The article assumes a running and accessible HashiCorp Vault server with a secret called big IP password one at secrets/big IP password one config.
• The speaker uses a Vault server that was set up using MCP (Multi-Cloud Platform).
• The speaker creates the required secret in Vault using the vault kv put command.
• The speaker expresses confusion about the requirement for an "app role" with access to the secret.
• The speaker and Jason discuss the different roles within HashiCorp Vault and their access to certain values.
• The speaker attempts to follow the article's instructions for installing the Vault agent on the Docker host.
• The speaker encounters an issue when trying to create a temporary file system (tempfs) volume on macOS.
• The speaker realizes that the article is primarily based on YUbuntu, not macOS.
• The speaker decides to postpone the Vault integration to a future session due to time constraints and the need for a YUbuntu environment.
• The speaker plans to explore Docker Secrets as an alternative to HashiCorp Vault.
• The speaker acknowledges that not all sessions end in success and that learning and labbing live involves occasional setbacks.

Conclusion

The session covered building the Application Studies Tool (AST), securing Grafana with TLS, and attempting to integrate HashiCorp Vault for secrets management. While the AST build and TLS configuration were successful, the Vault integration was postponed due to environment-specific issues and time constraints. The session highlighted the importance of understanding the target environment and the value of learning and troubleshooting in a live setting. The speaker committed to revisiting the Vault integration in a future session, exploring both HashiCorp Vault and Docker Secrets on both macOS and YUbuntu.