Outage in Python

PyPI Dictionary Attack

Resolved Maintenance
November 11, 2022 - Started almost 3 years ago - Lasted 6 days
Official incident page

Need to monitor Python outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Python, and never miss an outage again.
Start Free Trial

Outage Details

Beginning on Nov 6th, 2022, PyPI began to become inundated with failed authentication attempts due to a dictionary attack (https://en.wikipedia.org/wiki/Dictionary_attack). At its peak, the attack was attempting more than 260K passwords per hour, and at the time of writing has resulted in more than 6 million invalid authentication attempts. The attacker used a large pool of IP addresses to cycle through multiple IPs as each one reached our ratelimit for authentication failures. The attacker specifically used PyPI’s ‘basic’ HTTP authentication methods associated with our upload endpoint. PyPI integrates with the Have I Been Pwned (HIBP) API (https://haveibeenpwned.com/API/v2) to prevent dictionary and credential stuffing attacks. Four times during the period of this attack, PyPI used the HIBP API to determine that a compromised password was being used to log into a user’s account, making it likely that the attacker was using a set of leaked passwords that HIBP includes in its dataset. For comparison, in the two weeks prior to the start of the attack, PyPI identified the usage of 16 compromised passwords via HIBP. As a result, in each of these occurrences, PyPI prevented the authentication attempt from succeeding, forcing the user’s password to be reset, and notifying the user. Due to insufficient metrics at the time, we are unable to determine what proportion of these login attempts with compromised passwords were likely legitimate, and what percentage were from IPs participating in the dictionary attack. At this time, we have determined that no successful logins to legitimate user accounts have been performed from requests originating from IP addresses participating in the dictionary attack. As a result of this attack, PyPI has made a number of changes to improve our ability to respond to such an attack: - Added a new user event type for successful ‘basic’ HTTP authentication logins - Added a new user event type for detection of compromised passwords via HIBP - Added new moderation tools to track the occurrence of abuse attempts grouped by IP address At the time of writing, the attack has temporarily abated, as it has a few times over the past week. The PyPI volunteer team will continue to monitor for changes, as well as for potentially compromised accounts.
Latest Updates ( sorted recent to last )
RESOLVED almost 3 years ago - at 11/17/2022 05:47PM

This incident has been resolved.

MONITORING almost 3 years ago - at 11/11/2022 10:24PM

Beginning on Nov 6th, 2022, PyPI began to become inundated with failed authentication attempts due to a dictionary attack (https://en.wikipedia.org/wiki/Dictionary_attack). At its peak, the attack was attempting more than 260K passwords per hour, and at the time of writing has resulted in more than 6 million invalid authentication attempts.

The attacker used a large pool of IP addresses to cycle through multiple IPs as each one reached our ratelimit for authentication failures. The attacker specifically used PyPI’s ‘basic’ HTTP authentication methods associated with our upload endpoint.

PyPI integrates with the Have I Been Pwned (HIBP) API (https://haveibeenpwned.com/API/v2) to prevent dictionary and credential stuffing attacks. Four times during the period of this attack, PyPI used the HIBP API to determine that a compromised password was being used to log into a user’s account, making it likely that the attacker was using a set of leaked passwords that HIBP includes in its dataset. For comparison, in the two weeks prior to the start of the attack, PyPI identified the usage of 16 compromised passwords via HIBP.

As a result, in each of these occurrences, PyPI prevented the authentication attempt from succeeding, forcing the user’s password to be reset, and notifying the user. Due to insufficient metrics at the time, we are unable to determine what proportion of these login attempts with compromised passwords were likely legitimate, and what percentage were from IPs participating in the dictionary attack.

At this time, we have determined that no successful logins to legitimate user accounts have been performed from requests originating from IP addresses participating in the dictionary attack.

As a result of this attack, PyPI has made a number of changes to improve our ability to respond to such an attack:

- Added a new user event type for successful ‘basic’ HTTP authentication logins
- Added a new user event type for detection of compromised passwords via HIBP
- Added new moderation tools to track the occurrence of abuse attempts grouped by IP address

At the time of writing, the attack has temporarily abated, as it has a few times over the past week. The PyPI volunteer team will continue to monitor for changes, as well as for potentially compromised accounts.

Latest Python outages

PyPI Partial Outage - 4 months ago
PyPI Elevated Error Rates - 5 months ago
Download file corruption - 7 months ago

Be the First to Know When Vendors Go Down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4484 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook