Use cases
Software Products E-commerce MSPs Schools Development & Marketing DevOps Agencies Help Desk
Company
Internet Status Blog Pricing Log in Get started free

Outage in Harness

Pipeline and secret resolution failures

Resolved Minor
May 08, 2026 - Started about 1 month ago - Lasted 10 days
Official incident page

Incident Report

On May 8, 2026, a CI performance optimization feature flag was inadvertantly impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions

Trusted by 1,000+ teams

The Status Page Aggregator with Early Outage Detection

Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.

IsDown status aggregator dashboard
Latest Updates ( sorted recent to last )
POSTMORTEM 21 days ago - at 05/19/2026 12:33AM

On May 8, 2026, a CI performance optimization feature flag was inadvertently impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions.

Customers running delegate versions that did not support the new runtime secret resolution capability experienced pipeline execution failures where secrets referenced in Run step commands resolved as empty strings.

The issue primarily impacted customers running older delegate versions outside the currently supported delegate release window. Customers on supported delegate versions were not impacted.

The issue was mitigated by disabling the feature flag, which immediately restored the prior secret resolution behavior. No customer configuration changes were required.

**Impact**

* Pipeline execution failures for affected Kubernetes CI builds
* Secrets referenced in Run step commands resolved as empty strings
* Customers running supported delegate versions were not affected

**Root Cause**

A feature flag intended for a limited internal rollout was unintentionally enabled for a broader set of accounts during a routine rollout process. The feature relied on delegate-side functionality available only in newer delegate versions. Older delegate versions did not support the required runtime secret resolution behavior, resulting in pipeline failures.



**Resolution**

The feature flag was disabled globally, restoring the previous behavior where secrets are resolved during pipeline initialization. Impacted pipelines recovered immediately after rollback.

**Preventive Actions**

Harness is implementing the following improvements:

* Enhancements to the feature flag approval workflow to more clearly display rollout scope.
* Improved process and documentation for feature flags that depend on minimum delegate versions
* Expanded monitoring and alerting for secret resolution failures in CI pipelines

RESOLVED 21 days ago - at 05/19/2026 12:33AM

On May 8, 2026, a CI performance optimization feature flag was inadvertantly impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions

The Status Page Aggregator with Early Outage Detection

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 6320 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook