A build pipeline turned red at 3:47 PM on a Friday. A senior developer at a fintech startup had pushed what he thought was a routine patch to a microservice that handled user session tokens. The change was small—a bump to a logging library that fixed a memory leak. But the CI system didn't care about the patch's intent. It rejected the merge because one transitive dependency's Software Bill of Materials (SBOM) wasn't cryptographically signed.

The signed SBOM mandate had been in place for six months, enacted after a board-level security review. Most of the time, automated signing tools handled it silently. But this particular dependency—a Redis client wrapper maintained by a solo developer—had been updated two hours earlier, and the new version's SBOM was uploaded without a valid GPG signature. The build system's verifier caught it, blocked the merge, and the developer, eager to head home, overrode the gate with a force push that bypassed signing checks. That override propagated the unsigned artifact to production, where a separate runtime verifier flagged the mismatch and triggered an automatic rollback that took four hours to complete across twelve microservices. The on-call engineer spent the next four hours coordinating rollbacks, verifying that no data was corrupted, and manually re-deploying the previous signed versions. Postmortems later revealed that the Redis wrapper maintainer had updated the package without re-signing because his local signing key had expired. He hadn't noticed because his own CI didn't enforce signing on every commit. The incident was classified as a process gap, not a people failure, but the team's Friday evening was gone.

This incident, described to me by an engineer who asked not to be named, is not unusual. As signed SBOMs move from compliance artifacts to operational gates, teams are discovering that the cryptographic chain of custody has real teeth—and real costs. The technology works as designed. The question is whether organizations are ready for what that means on a Friday afternoon.

The Commit That Broke Production

The merge request looked clean. It passed unit tests, integration tests, and a static analysis scan. The developer had followed the team's usual Friday routine: push a small fix, let the pipeline run, and head out early if everything went green. But the pipeline's final stage—a verification step that checked every dependency's SBOM signature against a public key trust root—failed on a transitive dependency three levels deep. The SBOM for that Redis wrapper was present, but the detached signature file was missing. The cosign verifier returned a 403 from the package registry, and the pipeline halted. The developer, seeing a red X on a Friday afternoon, made a judgment call: the signing requirement was a recent addition, the dependency was widely used, and the risk seemed low. He bypassed the gate and merged anyway. That bypass worked locally, but the production environment had a separate enforcement layer. When the deployment controller fetched the package, it checked the signature against a certificate authority that had been rotated earlier that week. The mismatch triggered an automatic rollback that cascaded through dependent services.

SBOMs: From Compliance Artifact to Runtime Gate

Software Bills of Materials started as a response to supply-chain attacks like SolarWinds. The US Executive Order 14028, issued in 2021, mandated SBOMs for software sold to the federal government. Many teams treated them as PDFs to be generated once and filed with auditors. The content was often incomplete, listing only direct dependencies and omitting transitive ones. Signatures were an afterthought.

That changed as the industry realized that an unsigned SBOM is about as useful as a grocery list without a receipt. Anyone could modify it. The open source maintainer burnout crisis made things worse: unmaintained packages with unsigned metadata became attack vectors. Attackers could inject malicious code and alter the SBOM to hide the change.

Signed SBOMs, enforced by frameworks like in-toto and SLSA, turn the document into a cryptographic chain of custody. Each step—from source commit to build to package upload—carries an attestation signed by the entity that performed it. The verifier checks that every link in the chain is intact and that no step was tampered with. When a build system rejects an unsigned SBOM, it's not being pedantic. It's enforcing a trust model that assumes compromise until proven otherwise.

Today, major cloud providers and package registries push signed SBOMs by default. GitHub, GitLab, and Artifactory all have built-in signing workflows. But the shift from optional to mandatory has created operational friction. Teams that once generated SBOMs as a post-build step now must integrate signing into every merge pipeline. And when a signature is missing, the gate slams shut.

The Cryptographic Tax on Developer Velocity

Key management is the part of supply-chain security that nobody wants to talk about. Signing every SBOM requires keys that are stored securely, rotated regularly, and accessible to automated build systems. In practice, this means key ceremonies that slow down CI/CD pipelines. A single key rotation can take hours if it involves multiple teams and manual approval gates.

Sigstore's cosign tool, which is now the de facto standard for container and SBOM signing, adds roughly 10–30 seconds to each package push. That doesn't sound like much, but multiplied across hundreds of daily builds, it adds up. More importantly, cosign requires that the signing key be available to the build agent, which means either storing it in a secrets manager or using a short-lived key issued by an OIDC provider. Both approaches add complexity.

Some developers respond by hoarding long-lived signing keys, storing them in environment variables or even in code comments. This defeats the purpose of signing. Security teams then enforce gating only after incidents, creating a cycle of panic and remediation. The Friday afternoon patch is a cultural habit born from this friction: developers know that bypassing the gate is possible, and they do it when the cost of compliance feels higher than the risk of failure.

The trade-off is real. Every signature check adds latency. Every key rotation risks breaking a pipeline. The industry is still learning how to make signing boring—something that happens automatically without developer intervention. Tools like short-lived keys with automatic renewal and pre-commit hooks that verify signatures before push are steps in that direction, but adoption is uneven.

Why the Industry Shifted to Signed Attestations

The SolarWinds attack in 2020 was a watershed moment. Attackers compromised the build pipeline and inserted malicious code into signed software updates. The SBOM for those updates listed only the intended dependencies, not the backdoor. The industry realized that an unsigned SBOM is just a list; a signed one is a promise that can be verified.

The SLSA framework, developed by Google and adopted by the CNCF, defines levels of supply-chain security. Level 2 requires that the build process be signed. Level 3 adds verification that the source and dependencies are also signed. Level 4 requires a two-person review of every change. Most organizations today operate at SLSA Level 1 or 2. The push to Level 3 is driving the adoption of signed SBOMs and in-toto attestations.

In-toto, a CNCF project, provides a framework for defining and verifying the steps in a software supply chain. Each step produces a signed attestation—a statement that a particular action was performed by a particular entity at a particular time. The final verification checks that all attestations are present, signed, and consistent. This is what caught the unsigned Redis wrapper in the Friday afternoon incident.

Large cloud providers are now pushing signed SBOMs by default. AWS requires signed SBOMs for certain services in its FedRAMP environments. Google Cloud's Artifact Registry automatically signs every container image and SBOM. GitHub's Dependabot now checks for signed attestations when suggesting dependency updates. The infrastructure is in place. The friction is in the human layer.

What a Signed SBOM Actually Changes at the Wire

At the protocol level, a signed SBOM is a JSON or SPDX document with a detached cryptographic signature. The signature is typically a GPG or Sigstore signature stored as a separate file or embedded in the document header. When a package registry receives a push, it verifies the signature against a public key that is either stored in the registry or retrieved from a key server.

The verification process is straightforward: the verifier computes a hash of the SBOM content, decrypts the signature using the signer's public key, and compares the result. If they match, the signature is valid. If not, the registry returns an HTTP 403 Forbidden or a 422 Unprocessable Entity. The package is not stored, and the pipeline fails.

But the real change is in the metadata. Each signed SBOM includes a timestamp and the signing identity. This allows the verifier to check not just that the signature is valid, but that it was created within a valid time window and by an authorized entity. Revocation lists, distributed via CRL or OCSP, can block compromised keys within minutes. This is a significant improvement over the old model, where a stolen key could be used indefinitely.

The wire-level impact is that every dependency fetch now includes a signature verification step. This adds latency, but it also adds traceability. When a build fails because of a missing signature, the developer knows exactly which dependency is at fault and who last signed it. The trade-off is that the system becomes brittle: a single unsigned transitive dependency can block an entire deployment.

The Human Cost: On-Call Fatigue and Blameless Postmortems

The Friday afternoon patch is a symptom of a deeper cultural problem. Developers want to ship code and go home. Security teams want to prevent breaches. The tension between velocity and security is not new, but signed SBOMs have made it visible in a way that abstract policies never did.

On-call engineers bear the brunt of this tension. When a signed SBOM gate fails, they are the ones who get paged. They have to diagnose whether the failure is a genuine security issue or a false positive caused by an expired key or a misconfigured verifier. In the incident described earlier, the on-call engineer spent two hours just identifying the root cause: the Redis wrapper maintainer's expired key.

Blameless postmortems are the standard response, and they help. The incident review for that Friday afternoon focused on process gaps: the lack of automated key renewal, the absence of pre-commit signature checks, the cultural norm of Friday patches. But the human cost remains. On-call rotations now include training on key management, which is a skill that few developers signed up for.

Some teams have responded by automating the signing process entirely. CI pipelines generate short-lived keys for each build, sign the SBOM, and discard the key. Others have adopted pre-commit hooks that verify signatures before the code ever reaches the build server. These measures reduce cognitive load, but they require investment in tooling and training. The teams that treat signing as part of the build, not an afterthought, are the ones that avoid the Friday afternoon panic.

Making Signed SBOMs Boring Again

The goal of supply-chain security is not to make developers' lives harder. It's to make attacks harder. But the current state of signed SBOMs is anything but boring. Every key rotation, every expired certificate, every missing signature is a potential incident. The industry needs to make signing invisible.

Short-lived keys with automatic renewal are a step in the right direction. Tools like cosign's keyless mode, which uses OIDC to issue ephemeral keys, eliminate the need for long-lived secrets. Pin base images with digest and signature, so that the build system knows exactly what it's getting. Include SBOM generation and signing in every merge pipeline, not just release builds. Audit logs should verify signatures, not just existence.

The hardware supply chain has dealt with similar challenges for decades. Software is catching up, but the transition is painful. The Friday afternoon incident was a learning experience. The team now has automated key renewal, pre-commit signature checks, and a policy that bans production patches after 3 PM. The signed SBOM gate stays in place. It's just less likely to catch anyone off guard.

To truly make signed SBOMs boring, the industry must standardize on ephemeral signing and integrate verification into the developer's workflow, not as a separate gate. The next frontier is automated remediation: when a signature is missing, the pipeline should not only fail but also suggest the exact command to fix it. Tools like Dependabot already propose dependency updates; similar automation for signing could reduce the cognitive load. Until then, every Friday afternoon is a reminder that security is a process, not a checkbox. The goal is not to eliminate failures but to make them predictable and fast to recover from. That's the path to making signed SBOMs as unremarkable as a checksum.