Secure JIT Infrastructure with OpenBao and Crossplane

Static credentials are the silent killer of modern cloud security. In the early days of Infrastructure as Code (IaC), we were content with hardcoding database passwords in variables files or, if we were feeling sophisticated, pulling them from a cloud-native secret manager. But as organizations scale across multiple clouds and hundreds of microservices, the 'static secret' model breaks down. It creates a massive attack surface, complicates rotation policies, and slows down developers who just want a database to test their code.

The solution is Just-in-Time (JIT) infrastructure—resources that are provisioned on-demand with ephemeral, short-lived credentials that expire automatically. To build this, we need two powerful tools: Crossplane, for control-plane-driven infrastructure orchestration, and OpenBao, the community-driven fork of HashiCorp Vault, for dynamic secret management.

The Shift from Push-Based to Reconciliation-Based IaC

Traditional IaC tools like Terraform operate on a push-based model. You run a plan, you apply it, and the state is updated. While powerful, this model often leads to 'configuration drift' and requires external CI/CD pipelines to trigger updates.

Crossplane flips this script by turning your Kubernetes cluster into a universal control plane. It uses the Kubernetes reconciliation loop to ensure that the actual state of your cloud resources matches the desired state defined in your YAML manifests. If someone manually deletes an S3 bucket in the AWS console, Crossplane notices and recreates it.

However, provisioning the infrastructure is only half the battle. The other half is ensuring the application using that infrastructure has the right credentials—without those credentials ever sitting in a plain-text configuration file or a long-lived environment variable.

Enter OpenBao: The Future of Open Secrets Management

OpenBao emerged as a response to the licensing changes in the HashiCorp ecosystem, preserving the robust, API-driven secrets management that engineers have relied on for years. For JIT infrastructure, OpenBao’s most critical feature is its Dynamic Secrets Engines.

Unlike a traditional key-value store, a dynamic secrets engine doesn't store a password; it generates one on the fly. When an application requests access to a PostgreSQL database, OpenBao communicates with the database, creates a new user with limited permissions, and returns the credentials to the application with a specific Time-to-Live (TTL). When the TTL expires, OpenBao automatically deletes the user from the database.

The Architecture: How They Work Together

To implement JIT infrastructure with secret injection, we follow a specific workflow:

1. The Request: A developer submits a CompositeResourceClaim (XRC) in Kubernetes, requesting a high-level resource like a 'SecureDatabase'.
2. The Provisioning: Crossplane’s Composition engine picks up the claim and provisions the actual cloud resources (e.g., an RDS instance, security groups, and IAM roles).
3. The Integration: Once the resource is ready, Crossplane writes the connection details (endpoint, port) to a Kubernetes Secret or an external store.
4. The Secret Injection: OpenBao’s Kubernetes sidecar injector or a tool like External Secrets Operator (ESO) picks up these details. OpenBao then configures a dynamic secrets engine for that specific database instance.
5. The Consumption: The application pod starts, authenticates with OpenBao via its Kubernetes Service Account, and retrieves ephemeral credentials.

Step 1: Configuring Crossplane for Multi-Cloud Provisioning

Before we can manage secrets, we need infrastructure. Crossplane uses 'Providers' to interact with cloud APIs. In a multi-cloud environment, you might have provider-aws and provider-gcp installed simultaneously.

Here is a simplified Composition that defines how a 'PostgreSQLInstance' should be provisioned on AWS:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xpostgres.aws.database.example.com
spec:
  compositeTypeRef:
    apiVersion: database.example.com/v1alpha1
    kind: XPostgreSQLInstance
  resources:
    - name: rdsinstance
      base:
        apiVersion: database.aws.upbound.io/v1beta1
        kind: Instance
        spec:
          forProvider:
            region: us-east-1
            dbInstanceClass: db.t3.micro
            engine: postgres
            engineVersion: "14.1"
            allocatedStorage: 20
            publiclyAccessible: false
          writeConnectionSecretToRef:
            namespace: crossplane-system
            name: rds-conn

This composition ensures that every time a developer asks for a database, Crossplane handles the cloud-specific heavy lifting. Note the writeConnectionSecretToRef field; this is the bridge to our secret management layer.

Step 2: Setting up OpenBao Dynamic Secrets

Once Crossplane has provisioned the database, we need to tell OpenBao how to manage it. We use the OpenBao Terraform provider (or the API) to configure the database secrets engine.

resource "openbao_database_secrets_mount" "db" {
  path = "postgres"

  postgresql {
    name       = "my-rds-db"
    plugin_name = "postgresql-database-plugin"
    allowed_roles = ["readonly", "app-role"]
    connection_url = "postgresql://{{username}}:{{password}}@rds-endpoint:5432/postgres"
    
    # This matches the admin credentials provisioned by Crossplane
    username = var.admin_user
    password = var.admin_password
  }
}

resource "openbao_database_secret_backend_role" "app_role" {
  name    = "app-role"
  backend = openbao_database_secrets_mount.db.path
  db_name = openbao_database_secrets_mount.db.postgresql[0].name
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
    "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
  ]
  default_ttl = "1h"
  max_ttl     = "24h"
}

In this setup, OpenBao acts as the gatekeeper. No human ever sees the app-role credentials. They are generated on-demand and destroyed after one hour.

Step 3: Automating the Glue with External Secrets Operator (ESO)

While Crossplane can write secrets to Kubernetes, we often want those secrets to live in OpenBao as the 'Source of Truth'. The External Secrets Operator (ESO) can synchronize secrets between Kubernetes and OpenBao.

When Crossplane creates the RDS instance and outputs the connection string, ESO can take that string and push it into an OpenBao KV store, which then triggers the configuration of the dynamic secrets engine. This creates a fully automated pipeline:

Developer Manifest -> Crossplane -> Cloud Resource -> ESO -> OpenBao -> App Pod.

Real-World Benefits of the JIT Approach

1. Zero-Trust Networking

By using OpenBao’s Kubernetes auth method, the application doesn't need a pre-shared key to talk to OpenBao. It uses its own Service Account Token. OpenBao verifies this token with the Kubernetes API server and, if valid, provides the database credentials. This is the essence of zero-trust: identity-based access rather than network-based access.

2. Simplified Compliance and Auditing

In a regulated environment (SOC2, HIPAA, PCI), auditing who accessed what data is a nightmare with shared credentials. With OpenBao, every single credential is unique to a specific pod and a specific point in time. The OpenBao audit logs provide a clear trail: "Pod A requested credentials at 10:00 AM, credentials expired at 11:00 AM."

3. Reduced Blast Radius

If an application pod is compromised, the attacker only gains access to a credential that will expire shortly. They cannot use that credential to create new users or move laterally through the infrastructure, as the permissions are strictly scoped by the OpenBao role definition.

Challenges and Considerations

Implementing this stack is not without its hurdles.

• Complexity: You are moving from a single tool (Terraform) to a distributed system (Kubernetes + Crossplane + OpenBao). This requires a higher level of operational maturity.
• Latency: Generating a dynamic secret involves a round-trip to OpenBao and then to the database. For high-scale applications that frequently restart, this can introduce minor startup latency.
• State Management: Crossplane manages the state of the infrastructure, while OpenBao manages the state of the credentials. Ensuring these two stay in sync—especially during resource deletion—requires careful lifecycle hook management.

Conclusion: The Path Forward

Automating JIT infrastructure with OpenBao and Crossplane represents the current gold standard for secure, scalable cloud operations. By treating infrastructure as a managed service within Kubernetes and secrets as dynamic, ephemeral entities, you remove the burden of manual credential management from your developers while significantly hardening your security posture.

To get started:

1. Deploy Crossplane to a management cluster and configure a provider for your primary cloud.
2. Install OpenBao and enable the Kubernetes authentication method.
3. Define a Composition that outputs connection details to a secret.
4. Configure a Dynamic Secrets Engine in OpenBao that uses those connection details to generate JIT users.

The transition from static to dynamic infrastructure is a journey, but the result—a self-service, secure, and resilient platform—is well worth the investment.