JIT Infrastructure: Automating Secrets with OpenBao and Crossplane

The paradigm of Infrastructure as Code (IaC) has undergone a significant transformation over the last decade. We moved from manual console clicks to shell scripts, then to declarative tools like Terraform and Pulumi. However, even with these advancements, a persistent friction point remains: the management of long-lived secrets and the static nature of cloud resources.

In a traditional CI/CD pipeline, we often store high-privilege cloud credentials (like AWS Access Keys or GCP Service Account keys) as static secrets in a CI runner. This creates a massive blast radius if the CI system is compromised. Furthermore, the infrastructure we provision often sits idle, over-privileged, and under-monitored.

To solve this, senior platform engineers are moving toward Just-in-Time (JIT) Infrastructure. By combining OpenBao (the community-driven, open-source fork of HashiCorp Vault) with Crossplane (the Kubernetes-native control plane), we can create a system where infrastructure is provisioned on-demand using ephemeral, short-lived credentials.

The Architectural Shift: From Static IaC to Control Planes

Traditional IaC is usually "push-based." You run a command, the tool calculates a diff, and it pushes changes to the cloud. The state is stored in a file. The problem? The state can drift, and the credentials used for the push are usually static.

Crossplane flips this model. It turns your Kubernetes cluster into a universal control plane. Instead of running a CLI tool, you define custom resources (CRDs) in Kubernetes that represent cloud services (e.g., an S3 bucket, an RDS instance). Crossplane’s controllers continuously reconcile the desired state in Kubernetes with the actual state in the cloud.

OpenBao complements this by managing the identity and secrets layer. Instead of Crossplane using a permanent IAM user, we can configure OpenBao to generate temporary credentials for Crossplane to use during its reconciliation loop. This is the essence of JIT infrastructure: the infrastructure exists when needed, and the permissions to manage it exist only as long as the task requires.

Why OpenBao?

With the licensing changes in the HashiCorp ecosystem, OpenBao has emerged as the critical open-source alternative for secret management. It maintains the robust API compatibility of the original Vault project while ensuring it remains under a truly open license (MPL-2.0) managed by the Linux Foundation.

For a JIT workflow, OpenBao provides three essential features:

1. Dynamic Secret Engines: It can generate cloud credentials on the fly with a specific TTL (Time to Live).
2. Transit Secrets: For encrypting sensitive data at rest without storing it.
3. Identity-based Access: It allows Kubernetes pods to authenticate via Service Accounts, eliminating the need for a "master secret."

Building the JIT Pipeline: A Step-by-Step Approach

To implement a secret-injected, JIT infrastructure workflow, we need to wire Crossplane and OpenBao together so that the cloud provider configuration is dynamically updated with fresh credentials.

1. Configuring the OpenBao Cloud Secret Engine

First, we configure OpenBao to talk to our cloud provider (let's use AWS as an example). We give OpenBao a high-privilege role that allows it to create lesser privileged, short-lived IAM users or roles.

# Enable the AWS secret engine
bao secrets enable aws

# Configure the root credentials (stored securely in OpenBao)
bao write aws/config/root \
    access_key=$ROOT_KEY \
    secret_key=$ROOT_SECRET \
    region=us-east-1

# Create a role that generates temporary credentials for Crossplane
bao write aws/roles/crossplane-role \
    credential_type=iam_user \
    policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:*",
      "Resource": "*"
    }
  ]
}
EOF

2. Setting up Crossplane with Dynamic Provider Configs

Crossplane uses a ProviderConfig to know which credentials to use when talking to a cloud provider. Usually, this points to a static Kubernetes Secret. In our JIT model, we use a sidecar container or a specialized operator to fetch secrets from OpenBao and update that Kubernetes Secret every few minutes.

However, a more elegant approach involves using the OpenBao Agent inside the Crossplane provider pod. The agent authenticates via the Kubernetes Auth Method, fetches a new secret from the aws/roles/crossplane-role path, and writes it to a shared volume that Crossplane reads.

3. Defining Composite Resources (XRs)

The real power of Crossplane is abstraction. We don't want developers to worry about IAM or OpenBao. We want them to request a "Database."

apiVersion: database.example.org/v1alpha1
kind: XPostgreSQLInstance
metadata:
  name: my-db
spec:
  parameters:
    storageGB: 20
    region: us-east-1
  compositionRef:
    name: production-rds

When this resource is created, Crossplane’s composition engine kicks in. It sees that production-rds requires an AWS RDS instance. It then uses the credentials provided by OpenBao to provision that instance.

Injecting Secrets into the Provisioned Resources

Provisioning the resource is only half the battle. The application needs to use it. In a JIT world, we don't want the database password to be static.

When Crossplane finishes creating the RDS instance, it generates a connection secret. We can configure Crossplane to write this secret directly into an OpenBao path rather than a Kubernetes Secret.

1. Crossplane creates the RDS instance.
2. Crossplane outputs the host, username, and password to an OpenBao secret engine via a specialized provider (like provider-vault or provider-openbao).
3. The application pod uses the OpenBao CSI Driver or Agent to mount that secret as a file or environment variable.

This ensures that the database password never exists as a plaintext string in your CI/CD logs or even in your Kubernetes etcd (unless encrypted).

The Security Benefits of the OpenBao + Crossplane Stack

Elimination of Long-Lived Credentials

By using OpenBao to generate credentials for Crossplane, your cloud provider no longer has permanent "admin" keys floating around. If a credential is leaked, it expires automatically within minutes or hours.

Least Privilege at Scale

With Crossplane Compositions, you can enforce security policies at the infrastructure level. Developers interact with high-level abstractions, while the Platform Team defines the underlying resources with strict IAM policies generated by OpenBao.

Auditability and Compliance

OpenBao provides a detailed audit log of who requested which credential and when. Crossplane provides a continuous record of the state of your infrastructure. Together, they provide a comprehensive audit trail that is a dream for compliance teams (SOC2, HIPAA, etc.).

Practical Implementation Challenges

While this setup is powerful, it isn't without its hurdles.

1. Token Renewal and TTLs: You must ensure that the OpenBao tokens used by Crossplane have a renewal mechanism. If the token expires and Crossplane fails to renew it, the reconciliation loop will break, and you'll experience "provider lag" where infrastructure changes aren't reflected.

2. Initial Bootstrap (The Chicken and Egg Problem): You need some credentials to set up OpenBao and Crossplane in the first place. Use a "break-glass" procedure for the initial bootstrap, and then immediately move to the automated identity-based system.

3. Complexity Overhead: This architecture adds layers. For a small startup with two microservices, this might be overkill. However, for an enterprise managing hundreds of accounts across AWS, Azure, and GCP, the reduction in operational risk far outweighs the initial setup complexity.

Real-World Scenario: Multi-Cloud Database Provisioning

Imagine a scenario where a developer needs a Redis instance in Google Cloud and a Postgres instance in AWS for a distributed application.

Without this stack, the developer would need to manage two different CLIs, two sets of long-lived keys, and manually ensure the secrets are injected into the app.

With OpenBao and Crossplane:

1. The developer submits a single YAML manifest to the K8s API containing two Resource Claims.
2. Crossplane detects the claims and requests dynamic credentials from OpenBao for both AWS and GCP.
3. OpenBao generates short-lived IAM roles for AWS and Service Account keys for GCP.
4. Crossplane provisions the resources.
5. The connection details are stored in OpenBao.
6. The application starts up, authenticates to OpenBao via its K8s identity, and pulls the credentials it needs to connect to both clouds.

No static keys. No manual intervention. Fully automated and secure.

Actionable Conclusion

Implementing Just-in-Time infrastructure is no longer a luxury reserved for FAANG-scale companies. With the maturity of Crossplane and the community-backed stability of OpenBao, any platform team can build a secure, automated provisioning engine.

To get started:

1. Deploy OpenBao in your management cluster and enable the Kubernetes Auth method.
2. Install Crossplane and the providers for your primary cloud services.
3. Transition one resource (like an S3 bucket or GCS bucket) to use dynamic credentials generated by OpenBao.
4. Abstract the complexity by creating a CompositeResourceDefinition that hides the secret injection logic from your end-users (the developers).

By moving away from static secrets and toward a control-plane-driven architecture, you don't just improve security—you improve the developer experience by providing a self-service, reliable, and compliant infrastructure platform.