Sequencer Handover Malfunction Causes 33-Minute System Interruption
The Base Layer 2 blockchain, a prominent player in the DeFi space with $4.1 billion in total value locked, experienced a 33-minute network halt on September 5, 2023. This outage affected deposits, withdrawals, and block production on the mainnet.
The incident was due to a failure in the sequencer handoff process, a critical single point of failure in the current architecture. The sequencer, which orders and processes transactions before batching them on Ethereum's mainnet, is managed centrally by a system called Conductor. During a surge in network activity, the new sequencer was not ready to take over, causing block production to halt completely.
This centralized sequencer model has drawn criticism due to its potential to weaken blockchain reliability and resilience, especially under high transaction volumes. Past outages, including one shortly after launch, have highlighted recurring risks tied to reliance on a single sequencer entity, undermining user trust and raising questions about Base’s ability to scale securely.
To address these centralization concerns, the Base team is updating its infrastructure to ensure all sequencers in the system will be fully configured and ready to take over block-building duties immediately upon election. This aims to remove the single point of failure by allowing seamless sequencer handoffs and improve the network’s operational robustness going forward.
Meanwhile, Solana, the second-largest DeFi chain with $9.6 billion in total value locked, also faced similar problems. During the outage, the active sequencer on Base started to fall behind due to rising on-chain activity, much like what occurred on Solana.
In comparison, Solana boasts 2.83 million active addresses, while Base has 1.09 million active addresses, according to DefiLlama data. Despite these differences, both blockchains rely on a single system to manage their sequencer clusters, potentially exposing them to similar risks.
Mert Mumtaz, CEO of Helius Labs, compared the incident to similar problems seen on Solana, while former Coinbase engineer and Save Finance Founder, '0xrooter', stated that downtime for chains with actual users is a sign of success.
In an official postmortem report, the Base team explained that the issue came from a failed sequencer handoff. Currently, Base depends on a single system, Conductor, to manage its sequencer cluster. The team's infrastructure updates aim to rectify this issue and improve the network’s resilience.
It's worth noting that Coinbase, the company behind Base, is planning to raise $2 billion through convertible notes, although this funding is not directly related to the Base blockchain issue.
The outage underscores the inherent risks of Base’s current sequencer centralization and motivates the team to implement infrastructure changes to improve failover capabilities and reduce dependence on a single sequencer controller.
- The Base team is updating their infrastructure to remove the single point of failure, allowing seamless sequencer handoffs and improving the network’s operational robustness, addressing concerns regarding the centralization of sequencers.
- Despite differences in activity, both Base and Solana rely on a single system to manage their sequencer clusters, potentially exposing them to similar risks, as evidenced by their recent network halts.
- As the Base team works on improving the network’s resilience, other players in the DeFi space, such as Solana, are also grappling with similar issues, highlighting the inherent risks associated with centralization in blockchain technology.
