Node out of sync / partial network outage

Lotus patches are out now (https://github.com/filecoin-project/lotus/releases/tag/v1.28.3; https://github.com/filecoin-project/lotus/releases/tag/v1.29.1) and the impacted nodes are recovering. Implementers are considering this incident as resolved now, and we will keep monitoring the network. Thank you to the Filecoin node operators for all your support for a speedy recovery!

We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.

We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.

Many Filecoin nodes are not syncing due to "failed to verify beacon". Lotus implementers are investigating the issue with the Drand implementers, and we will provide updates in #fil-network-announcement and https://github.com/filecoin-project/lotus/issues/12467 as soon as we have more information.