JIT Infrastructure: Securing Crossplane with OpenBao Dynamic Secrets

The paradigm of Infrastructure as Code (IaC) has evolved. We have moved from manually clicking buttons in a console to writing HCL or Pulumi scripts, but a persistent friction point remains: the management of long-lived credentials. In a traditional CI/CD pipeline, your runner often holds a static 'God-mode' API key for AWS, GCP, or Azure. If that key is compromised, the blast radius is catastrophic.

Just-in-Time (JIT) infrastructure provisioning addresses this by ensuring that the credentials used to create resources are short-lived, scoped to the specific task, and automatically revoked. By combining OpenBao (the community-driven, open-source fork of Vault) and Crossplane (the Kubernetes-native control plane), we can build a robust system that provisions cloud resources using ephemeral identities.

This article explores how to architect and implement this workflow, moving beyond static service account keys toward a zero-trust infrastructure model.

The Identity Crisis in Modern IaC

Most organizations using Crossplane today rely on a ProviderConfig that references a static Kubernetes Secret. This secret usually contains a JSON key for a GCP Service Account or an AWS IAM Access Key. While Crossplane handles the lifecycle of the infrastructure (the 'what'), the 'how'—the authentication—remains brittle.

Static credentials present three primary risks:

1. Exfiltration: Keys stored in Kubernetes secrets are often base64 encoded and easily accessible to anyone with get secret permissions.
2. Rotation Overhead: Manually rotating keys across multiple clusters and cloud providers is an operational nightmare.
3. Lack of Granularity: Teams often default to over-privileged keys to avoid the friction of creating specific roles for every resource type.

By integrating OpenBao, we shift the responsibility of identity to a dedicated secrets engine that generates credentials on the fly.

The Architecture: OpenBao + Crossplane

The goal is to have Crossplane request credentials from OpenBao only when it needs to reconcile a resource. The architecture looks like this:

1. OpenBao is configured with a Cloud Secrets Engine (e.g., AWS or GCP).
2. Crossplane is installed on a Kubernetes cluster.
3. A Secret Store mechanism (like the Crossplane StoreConfig or a sidecar) bridges the gap between OpenBao's dynamic output and Crossplane's requirements.
4. When a developer requests a resource (e.g., an RDS instance), OpenBao generates a temporary IAM role, passes it to Crossplane, and Crossplane uses it to provision the resource.

Step 1: Configuring OpenBao for Dynamic Cloud Secrets

OpenBao excels at generating dynamic credentials. For AWS, this involves the aws secrets engine. Instead of storing a key, OpenBao uses a root credential to generate child tokens with specific TTLs (Time to Live).

First, enable the engine and configure the root credentials:

# Enable the AWS engine
bao secrets enable aws

# Configure the root connection
bao write aws/config/root \
    access_key=AKIA... \
    secret_key=wJalr... \
    region=us-east-1

Next, define a role. This is the 'template' for the JIT credentials. If a developer needs to provision S3 buckets via Crossplane, we create a role that allows only S3 actions:

bao write aws/roles/crossplane-s3-role \
    credential_type=iam_user \
    policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}
EOF

Now, whenever someone reads from aws/creds/crossplane-s3-role, OpenBao generates a brand-new IAM user with those permissions and a default TTL (e.g., 1 hour).

Step 2: Integrating OpenBao with the Kubernetes Control Plane

Crossplane needs to consume these dynamic secrets. There are two primary ways to do this: the Vault Sidecar Injector approach or the External Secrets Operator (ESO) approach. For a senior-level architecture, I recommend the External Secrets Operator because it decouples the secret lifecycle from the Crossplane pod lifecycle.

Using External Secrets Operator (ESO)

ESO can watch OpenBao and sync the dynamic secret into a standard Kubernetes Secret that Crossplane can read.

Define a SecretStore that points to OpenBao:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: openbao-backend
spec:
  provider:
    vault:
      server: "https://openbao.internal.svc:8200"
      path: "aws"
      version: "v1"
      auth:
        kubernetes:
          mountPath: "auth/kubernetes"
          role: "crossplane-controller"

Then, create an ExternalSecret that fetches the dynamic credentials. Note the refreshInterval; this determines how often ESO asks OpenBao for a new set of credentials.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dynamic-aws-creds
spec:
  refreshInterval: "55m" # Slightly less than the 1h TTL
  secretStoreRef:
    name: openbao-backend
    kind: SecretStore
  target:
    name: crossplane-aws-creds # This is the K8s secret Crossplane will use
  data:
    - secretKey: credentials
      remoteRef:
        key: aws/creds/crossplane-s3-role
        property: access_key
    # ... repeat for secret_key and session_token

Step 3: Configuring the Crossplane Provider

With the secret being continuously rotated and injected into the cluster by ESO, we now point Crossplane's ProviderConfig to this managed secret.

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: crossplane-aws-creds
      key: credentials

Because Crossplane reconciles resources periodically, it will pick up the new credentials whenever the secret is updated. If a reconciliation loop fails due to an expired token, the next loop (after ESO refreshes the secret) will succeed. This creates a self-healing, JIT authentication loop.

Deep Dive: Composition and Multi-Cloud Abstraction

The true power of this setup is realized when using Crossplane Compositions. As a platform engineer, you can expose a simplified API to your developers (e.g., CompositeDatabase) while the underlying machinery handles the complex JIT logic across different clouds.

Imagine a scenario where a developer needs a PostgreSQL instance. They don't care if it's on AWS RDS or GCP CloudSQL. You can define a Composition that selects the correct ProviderConfig based on the environment.

By mapping different OpenBao roles to different Crossplane ProviderConfigs, you ensure that the 'Database' team’s credentials can only create databases, and the 'Networking' team’s credentials can only modify VPCs. This is the principle of least privilege applied to infrastructure automation.

Handling the TTL and Race Conditions

One technical challenge in JIT infrastructure is the "TTL Gap." If OpenBao issues a credential with a 60-minute TTL, but Crossplane starts a long-running resource operation (like provisioning a large RDS cluster) at minute 59, the operation might fail mid-way.

To mitigate this:

1. Buffer the TTL: Ensure the refreshInterval in your secret sync mechanism is significantly shorter than the OpenBao TTL (e.g., refresh every 30 minutes for a 60-minute token).
2. Graceful Retries: Crossplane is designed for eventual consistency. A transient authentication failure during a credential swap is expected and will be resolved in the next sync period.
3. OpenBao Leases: Use OpenBao's lease ID to monitor when credentials are about to expire and trigger proactive rotation.

Security Gains and Auditability

By moving to this model, your security posture improves dramatically:

• No Static Keys: There are no long-lived AWS IAM keys to be stolen from a developer's .aws/credentials or a CI system.
• Audit Trails: Every single resource provisioned can be traced back to a specific OpenBao lease. You can see exactly which Kubernetes ServiceAccount requested which dynamic credential at what time.
• Instant Revocation: If you suspect a cluster is compromised, you can revoke all active leases in OpenBao with a single command, immediately invalidating every credential used by Crossplane across your entire multi-cloud footprint.

Operational Considerations for Senior Engineers

Implementing this at scale requires thinking about the "Who watches the watchmen?" problem. OpenBao itself becomes a Tier-0 service. If OpenBao is down, your infrastructure control plane is paralyzed.

• High Availability: OpenBao must be deployed in a highly available configuration (typically using Raft storage) across multiple availability zones.
• Seal Management: Use Auto-Unseal features (via AWS KMS or GCP KMS) to ensure OpenBao can recover from node failures without manual intervention.
• Monitoring: Monitor the secret_expiry metrics in OpenBao. A spike in expired but un-rotated leases is a leading indicator of a failure in your JIT pipeline.

Actionable Conclusion

Static credentials are a legacy burden that modern platform engineering can no longer afford. Implementing JIT infrastructure with OpenBao and Crossplane transforms your security model from reactive to proactive.

To get started:

1. Audit your current Crossplane providers: Identify which ones use static keys and map them to required IAM permissions.
2. Deploy OpenBao: Set up the cloud secrets engines for your primary providers.
3. Bridge the gap: Use the External Secrets Operator to sync dynamic leases into Kubernetes secrets.
4. Refactor ProviderConfigs: Shift Crossplane to consume these managed secrets.

By decoupling identity from the control plane, you create a system that is not only more secure but also more resilient and easier to audit in complex, multi-cloud environments.