Node operators publicly questioned, and the Hyperliquid centralized API has sparked controversy againA Letter To The Hyperliquid Core Team

A Letter To The Hyperliquid Core Team

Author: Kam Benbrik, Researcher @Chorus One
Compiled by: Ashley, BlockBeats

Editor's Note: Node operator Chorus One detailed multiple issues with the Hyperliquid testnet in an open letter on the X platform, including frequent node jailing, operational difficulties due to closed-source code, and single point of failure risks from centralized APIs. They proposed several improvement suggestions aimed at enhancing the chain's transparency and decentralization. In response, Hyperliquid founder Jeff emphasized that the validator selection criteria have been outlined in announcements; the official Hyperliquid account also posted separately on the X platform to clarify the issues mentioned in the letter and stated that node code will be open-sourced when safe.

Below is the original content (reorganized for readability):

This letter is written by Chorus One to the @HyperliquidX engineering team, hoping the team can take some time to review this feedback regarding Hyperliquid chain management.

TL;DR

• Due to closed-source code, lack of documentation, and reliance on centralized APIs, validators face significant challenges, leading to frequent node jailing and unstable performance.

• The testnet incentive mechanism has led to black market trading of HYPE tokens, favoring trades with large holders rather than fairly selecting validators.

• Low returns for mainnet validators cannot cover high self-staking requirements, limiting decentralization as 81% of the stake is controlled by foundation nodes.

• To compete with major layer 1s, Hyperliquid must improve transparency, decentralize staking, implement a fair validator selection mechanism, and engage more with external validators.

I started engaging with Hyperliquid in December 2023 and found this application to be outstanding. It is user-friendly, offers an excellent user experience, and provides unique features not found elsewhere, such as vaults and the well-known HLP. Currently, HLP manages over $350 million in funds and allows anyone to participate in Hyperliquid passively.

After witnessing the platform's excellent performance and learning that Hyperliquid operates independently as its layer 1, I hope Chorus One can participate as an operator on the Hyperliquid chain. I am an employee of Chorus One, one of the largest node operators in the industry. Since 2018, Chorus One has been active in the Proof of Stake sector. We have collaborated with many excellent teams, participated in the development of various blockchain designs and consensus algorithms, and played a key role in some of the earliest Proof of Stake chains like Tezos and Cosmos Hub. Currently, Chorus One manages over 50 blockchains with a total staked asset value exceeding $3 billion and has collaborated with all major Proof of Stake blockchains since the early stages.

Chorus One joined the Hyperliquid testnet after being whitelisted on October 17. I would like to share our overall experience on the testnet with the Hyperliquid engineering team, as we have not had the opportunity to interact with the team even after nearly three months of testnet operation. During this time, we witnessed one of the most successful token launches of 2024—the launch of the HYPE token. At the same time, we also experienced a testnet environment that was both interesting and challenging. I hope to mention some observed points that I hope will be considered in the coming days, weeks, or months.

Testnet Experience

The experience on the testnet has been very challenging so far. Node operators have almost no information on how to run nodes, with only limited resources available for reference.

Frequent Node Jailing Issues with Unclear Reasons

Initially, we were jailed multiple times without understanding the reasons. Due to the closed-source code, we could not accurately assess the causes of the jailing. The only way was to communicate with other validators on Discord and speculate on possible reasons together. After communicating with several validators, we learned that they also faced repeated jailing and were not entirely clear on the reasons.

Node Location Issues

Later, we discovered that the jailing issues might be due to our nodes not being deployed in Tokyo. Migrating the nodes to Tokyo might help. Unfortunately, the team never explicitly informed us of this, and we only found out after encountering issues multiple times.

After migrating the nodes to Tokyo, the situation improved. This may be because many testnet nodes with significant stakes are also deployed in Tokyo, allowing our nodes to catch up slightly and reduce missed blocks. However, even after the migration, we still faced jailing issues, with specific reasons remaining unclear. This lack of understanding is primarily due to the closed-source code.

Reliance on Automatic Unjailing Scripts

We realized that maintaining good uptime on the Hyperliquid testnet depends on the speed of the scripts that automatically unjail nodes. The only way to improve uptime is to rely on fast automatic unjailing scripts. Validators cannot fully understand or resolve potential issues and can only unjail nodes automatically without understanding the situation.

Centralized Hyperliquid API as a Single Point of Failure

There were several instances where our attempts to unjail failed due to the Hyperliquid API being down. Since validators must send requests to the Hyperliquid server to unjail, they cannot unjail themselves when the API is down.

The team may have already recognized this, but this design needs to be reconsidered, as it makes the API a critical single point of failure for the network. If the goal is to build a Byzantine Fault Tolerant system, then no node should have special privileges, such as relying on a centralized API.

Validator Selection on the Mainnet

Hyperliquid recently selected about 16 validators in the process of decentralizing its validator set. Previously, the 4 validators managed by the core team received a lot of criticism. Hyperliquid recently took an important step by expanding the validator set from 4 to 16.

Regarding the selection of validators, the 4 validators were announced through the following Discord post:

These validators are Validao, Bharvest, Hypurrstake, and Prrposefulnode. They were selected based on maintaining over 90% uptime in the past 7 or 30 days.

This is a significant achievement in multiple ways, primarily because validator performance is also affected by external factors such as Hyperliquid API downtime, jailing issues, and ongoing binary crashes, all of which have a non-negligible impact on performance.

In addition to the 4 validators selected based on testnet performance, 5 validators from the Hyperliquid Foundation are also operating on the mainnet. Additionally, another 7 validators have been selected to participate in the mainnet, but the reasons for their selection have not been made public.

Subsequently, a black market for HYPE testnet tokens began to emerge.

The Hyperliquid testnet initially had a set of 50 validators. Initially, specific entities were whitelisted to join the testnet, but since December 12, the validator set has been completely open.

The conditions are simple: 10,000 HYPE testnet tokens are required to register as a validator. However, to become an active validator, one must rank in the top 50; otherwise, the validator will remain inactive.

This decision led to a surge in the price of HYPE testnet tokens. The price initially rose to over 3,000 simulated USDC, and days later, it even exceeded 28,000 simulated USDC. As of the writing of this article, the price is approximately 700 simulated USDC per token.

Unfortunately, the faucet only distributes 100 simulated USDC every 4 hours. To become one of the top 50 validators on the testnet currently requires over 528,747 HYPE testnet tokens. Assuming a price of 700 simulated USDC per token and relying solely on the faucet, the calculation is as follows:

Days = (528,747 × 700) ÷ (100 × 6) = 616,871.5 days

This means that relying solely on the faucet would take approximately 616,871.5 days, or 1,690 years, to obtain enough HYPE testnet tokens to become an active validator on Hyperliquid.

However, those who received HYPE airdrops on the mainnet also received the same amount of tokens on the testnet. This provided validators with the opportunity to collaborate with these community members by allowing them to stake testnet HYPE tokens, enabling validators to secure entry into the active set.

Meanwhile, this situation also provided another perspective for holders of testnet HYPE tokens. Given the fierce competition to join the testnet validator set, many validators are eager to acquire as many HYPE testnet tokens as possible. As a result, a black market emerged where whales holding large amounts of testnet HYPE tokens began selling their testnet tokens to validators in exchange for USDC on the mainnet.

I have never seen such a chaotic situation before. Although the Hyperliquid team clearly disapproves of these practices, they are fully capable of addressing this issue. One potential solution is to implement a proper testnet validator selection process.

In most other PoS networks, the core team typically shares a form that any validator can fill out to express their willingness to run the chain. The team then reviews these applications based on various criteria and makes preliminary selections, such as the validator's node operation experience, past contributions, community involvement, or other factors.

This group of pre-selected validators can then participate in the testnet, working closely with the engineering team to provide feedback and ensure everything runs smoothly. We have made multiple attempts to provide feedback, but so far, we have not been successful.

Mainnet and Decentralization

As mentioned earlier, the current validator set on the Hyperliquid mainnet consists of 16 validators, which can be viewed at: https://app.hyperliquid.xyz/staking

• 5 validators are from the Hyperliquid Foundation.

• 4 validators were selected based on their performance on the testnet, maintaining over 90% uptime in the past 7 days.

• 7 validators were chosen by the Hyperliquid team.

Out of the staked 404,495,250 HYPE tokens, approximately 329,578,724 HYPE tokens are staked on foundation nodes, accounting for about 81.4% of the total staked amount. We know very little about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, the core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controls 1/3 of the stake, they can halt the chain. If they control 2/3 of the stake, they have complete control over the network.

The Hyperliquid Foundation initially staked 60 million HYPE tokens on each foundation node. However, many HYPE holders also chose to stake on foundation nodes, which is not ideal for decentralization. The team should engage more with the community to encourage a more decentralized staking distribution.

There are three potential solutions:

• Educate the community about the importance of staking with external validators to enhance the chain's security and decentralization.

• Implement a 100% commission rate on foundation nodes to incentivize users to stake with external validators, promoting decentralization.

• Redistribute foundation stakes to external validators, which is a practice adopted by most chains.

Decentralizing staking to external validators will also help them become more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and its infrastructure costs can be high, especially when nodes are deployed in Tokyo. Currently, validators at the bottom of the set earn between $3,000 and $5,000 per year, which is insufficient to cover costs. This is particularly challenging as they must self-stake the initial 10,000 HYPE tokens (approximately $250,000 at current prices) to validate on the mainnet.

Currently, users interact with Hyperliquid by bridging USDC from Arbitrum to the Hyperliquid chain. By reviewing the bridge's contract, it appears that the bridge is still managed by 4 validators. These validators do not seem to be associated with the chain's consensus or the 16 validators on the mainnet.

Hyperliquid has a great product, but the team still needs to improve several aspects of the infrastructure to truly compete with major layer 1s. Some improvements are quite simple, such as:

• Listening to the opinions of experienced validators managing multiple networks. While the team's current independent working style is very effective in building its perpetual product, validators are a pillar of a layer 1. Listening to their feedback is equally important to ensure everything runs smoothly.

• Open-sourcing the code. This will help validators better understand the issues they face when running nodes on Hyperliquid L1, while also helping users trust the product. Open-sourcing the code will also allow validators to gain more insight into the architecture and consensus algorithms. Currently, information about HyperBFT is very limited; open-sourcing can provide the much-needed transparency and understanding. Chorus One has an online handbook on the importance of open source. Operators should be able to build all the software they operate from the source code: https://handbook.chorus.one/node-software/open-source.html

• Creating a proper validator selection process to stop the black market trading of HYPE testnet tokens. Selecting validators based on uptime is a fair method, but obtaining good uptime should also be fair. It should not depend on relationships to acquire testnet tokens, purchasing testnet tokens, or external factors (such as reliance on Hyperliquid API uptime).

Overall, Hyperliquid does not need to make too many changes to compete with major layer 1s. The main focus should be on engaging more with external parties and adopting their feedback. I look forward to seeing changes in the coming weeks and months, and our team is always ready to provide assistance and feedback.

Hyperliquid Founder Jeff and Official Account Respond

In response to this letter, Hyperliquid founder Jeff made a statement on the X platform.

He emphasized that successfully running a validator is not difficu< the key lies in the validator's own setup and specialization. Additionally, he pointed out that the validator selection criteria have been outlined in announcements, based on high uptime performance in the early stages of the testnet. This suggests that Jeff is more inclined to believe that the current issues stem more from the validators' own configurations rather than flaws in the system design.

Furthermore, Hyperliquid's official account also released further clarifications, stating that node code will be open-sourced when safe.

• All validators qualified based on their testnet performance and cannot obtain seats through purchase; related false statements undermine the efforts of those who invested time and energy to understand the system; as the blockchain matures, the validator set will gradually expand.

• As previously announced, a foundation delegation program will be launched to support high-performing validators and further decentralize the network.

• Anyone can run an API server pointing to any node; example client code sends requests to specific API servers, but this is not a fundamental requirement of the network.

• Users attempting to create a black market for testnet HYPE tokens is unacceptable; this has been stated multiple times; we will continue to work on improving the onboarding process for the testnet.

• Node code is currently closed-source; open-sourcing is important, and the project will open-source once development stabilizes; Hyperliquid's development speed is several orders of magnitude faster than most projects, and its scope is also several orders of magnitude larger than most projects; the code will be open-sourced when safe.

• Currently, there is only one binary file. Even for very mature networks like Solana, the vast majority of validators run a single client.