Outage

hex.pm and the API have been stable since 19:11 UTC, and package file preview pages have been serving normally since 19:36 UTC.

The cause was a sustained flood of automated traffic against package file preview pages. Because each of those requests unpacks a package archive, the volume exhausted memory on the web servers and took them out of rotation, which is why the site returned errors. We have put rate limiting in place at the edge and raised capacity limits.

The traffic is still arriving and is being rate limited. We are continuing to monitor.

We are continuing to monitor for any further issues.

hex.pm and the API are responding normally again as of 19:11 UTC. The cause was a surge of automated traffic against package file preview pages, which exhausted memory on the web servers. We have blocked that traffic at the edge and raised capacity limits. Package file preview pages are temporarily returning errors while the block is in place.

The hex.pm website and API are unavailable. The cause is a surge of automated traffic against package file preview pages, which exhausted memory on the web servers and took them out of rotation. We are applying mitigations now.

Public package downloads and hexdocs.pm are not affected. Private package downloads and private documentation that triggers reauthentication will currently fail due to the website and API outage.

We are currently investigating this issue.