The Importance of Blockchain Data and the Rise of Indexers

The core of blockchain technology lies in data. It is the cornerstone for developing decentralized applications (dApp). While the industry is currently focused primarily on data availability (DA), data accessibility is equally important yet often overlooked.

In the era of modular Blockchain, DA solutions have become an indispensable part. They ensure that all participants can access transaction data, thus enabling real-time verification and maintaining network integrity. However, the DA layer functions more like a billboard than a database. This means that data is not permanently stored, but rather replaced by new data over time.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for developing dApps and conducting blockchain analysis. It is particularly important for tasks that require access to past data to ensure accurate representation and execution. Although less discussed, data accessibility is equally important as data availability. Both play different but complementary roles in the blockchain ecosystem, and a comprehensive data management approach must address both issues to support robust and efficient blockchain applications.

Traditional Blockchain Data Retrieval Methods

Since its inception, blockchain has fundamentally transformed infrastructure, driving the creation of dApps in areas such as gaming, finance, and social networks. However, building these dApps requires access to a large amount of blockchain data, which is both challenging and costly.

For dApp developers, one option is to host and run their own archive RPC nodes. These nodes store all historical Blockchain data, allowing for full access. However, maintaining archive nodes is costly, and their query capabilities are limited, making it impossible to query data in the format required by developers. While running cheaper nodes is an option, these nodes have limited data retrieval capabilities, which may impact the operation of the dApp.

Another method is to use commercial RPC node providers. These providers are responsible for the costs and management of the nodes, providing data through RPC endpoints. Public RPC endpoints are free but have rate limits, which can affect the user experience of dApps. Private RPC endpoints offer better performance by reducing congestion, but even simple data retrieval requires a lot of back-and-forth communication. This makes them request-heavy and inefficient for complex data queries. Additionally, private RPC endpoints are often difficult to scale and lack cross-network compatibility.

Blockchain Indexer: A Better Solution

Blockchain indexers play a key role in organizing chain data and sending it to databases for easier querying, often referred to as the "Google of Blockchain." They index blockchain data and make it readily available through SQL-like query languages such as GraphQL API (. Indexers provide a unified data query interface, allowing developers to quickly and accurately retrieve the information they need using standardized query languages, greatly simplifying the process.

Different types of indexers optimize data retrieval in various ways:

1. Full Node Indexer: Extracts data directly from full blockchain nodes, ensuring data completeness and accuracy, but requires substantial storage and processing power.

2. Lightweight Indexer: Relies on full nodes to fetch specific data on demand, reducing storage requirements but potentially increasing query time.

3. Specialized indexer: Optimized retrieval for specific types of data or blockchain, such as NFT data or DeFi transactions.

4. Aggregated Indexer: Extracts data from multiple Blockchains and sources, including off-chain information, providing a unified query interface, particularly useful for multi-chain dApps.

Ethereum alone requires 3TB of storage space. As the Blockchain grows, the amount of archive node data continues to increase. The indexer protocol deploys multiple indexers, which can efficiently index and quickly query a large amount of data, something that RPC cannot achieve.

The indexer also allows for complex queries, easy data filtering, and post-analysis. Some indexers can aggregate multi-source data, avoiding the need to deploy multiple APIs in multi-chain dApps. By being distributed across multiple nodes, indexers provide enhanced security and performance, while RPC providers may face interruptions and downtime due to centralized characteristics.

Overall, compared to RPC node providers, indexers improve the efficiency and reliability of data retrieval while reducing the cost of deploying a single node. This makes the Blockchain indexer protocol the preferred choice for dApp developers.

![Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-16396b955382c2c74010c264affdca46.webp(

Application Scenarios of Indexers

Building a dApp requires retrieving and reading Blockchain data to operate services. This encompasses various types of dApps, including DeFi, NFT platforms, games, and even social networks, as these platforms need to read data first before executing other transactions.

DeFi

DeFi protocols require different information to provide users with specific prices, rates, and fees. The automated market maker )AMM( needs price and liquidity information from liquidity pools to calculate swap rates, while lending protocols need utilization rates to determine borrowing rates and the debt ratio for liquidation. It is essential to input information into the dApp before calculating the rates executed by users.

Game

GameFi requires fast indexing and access to data to ensure a smooth gaming experience for users. Only through rapid data retrieval and execution can Web3 games match the performance of Web2 games, thereby attracting more users. These games need data such as land ownership, in-game token balances, and in-game operations. Using an indexer can better ensure a stable data flow and reliable uptime, ensuring a perfect gaming experience.

NFT

NFT markets and lending platforms need to index data to access various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing such data can avoid browsing each NFT individually to find ownership or attribute data.

Analysis

The indexer provides a method to extract specific data from the raw Blockchain data ), including smart contract events in each Block (. This offers opportunities for more specific data analysis, thereby providing comprehensive insights.

For example, perpetual trading protocols can identify which tokens have high trading volumes and generate fees, thereby deciding whether to list them as perpetual contracts on the platform. DEX developers can create dashboards for their products to gain insights into which liquidity pools offer the highest returns or strongest liquidity. They can also create public dashboards that allow developers to flexibly query any type of data to display on the charts.

Overview of Main Blockchain Indexers

The Graph

The Graph is one of the first indexing protocols launched on Ethereum, allowing easy access to previously hard-to-reach transaction data. It uses subgraph definitions and filters to collect subsets of data from the Blockchain, such as all transactions related to a specific transaction pool.

Using index proof, indexers stake the native token GRT for indexing and query services, and delegators can choose to stake their tokens here. Curators can access high-quality subgraphs to help indexers determine which subgraphs to curate data for in order to earn the best query fees. In the process of transitioning to a greater degree of decentralization, The Graph will eventually stop hosting services, requiring subgraphs to upgrade to its network while providing upgraded indexers.

Its infrastructure brings the average cost per million queries to $40, much lower than self-hosted nodes. Using file data sources, it also supports parallel indexing of both on-chain and off-chain data, enabling efficient data retrieval.

The rewards for The Graph's indexers have steadily increased over the past few quarters, partly due to the increase in query volume and also attributed to the rise in token prices. They plan to integrate AI-assisted queries in the future.

Subsquid

Subsquid is a peer-to-peer, horizontally scalable decentralized data lake that efficiently aggregates large amounts of on-chain and off-chain data, protected by zero-knowledge proofs. As a decentralized worker network, each node is responsible for storing data of a specific subset of blocks, speeding up the retrieval process by quickly identifying the nodes that hold the required data.

Subsquid supports real-time indexing, allowing indexing before the block is finalized. It also supports storing data in formats chosen by developers, making it easy to analyze using tools like BigQuery, Parquet, or CSV. In addition, subgraphs can be deployed on the Subsquid network without code, without the need to migrate to the Squid SDK.

Although still in the testnet phase, Subsquid has achieved significant results, with over 80,000 testnet users and more than 60,000 Squid indexers deployed, along with over 20,000 verified developers on the network. Recently, Subsquid launched its data lake mainnet.

In addition to indexing, the Subsquid Network data lake can also replace RPC in scenarios such as analytics, ZK/TEE co-processors, AI agents, and oracles.

SubQuery

SubQuery is a decentralized middleware infrastructure network that provides RPC and indexing data services. Initially supporting Polkadot and Substrate networks, it has now expanded to over 200 chains. Its working principle is similar to The Graph, which uses indexing proofs; indexers index data and provide query requests, while delegators stake their shares to the indexers. It introduces consumers to submit purchase orders, indicating that indexer income is guaranteed, rather than managed.

It will introduce SubQuery data nodes that support sharding, preventing continuous synchronization of new data between nodes, optimizing query efficiency, and moving towards greater decentralization. Users can choose to pay approximately 1 SQT token for computation costs per thousand requests, or set custom fees for indexers through the protocol.

Although SubQuery launched its token only this year, the issuance rewards for nodes and delegators have increased in USD value on a month-on-month basis, reflecting the growing number of query services offered on its platform. Since the TGE, the total amount of staked SQT has risen from 6 million to 125 million, highlighting the increase in network participation.

Covalent

Covalent is a decentralized indexing network, where block sample producers )BSP( network nodes create copies of blockchain data through bulk export and publish proofs on the Covalent L1 blockchain. This data is then refined by block result producers )BRP( nodes according to rules, filtering out the data that meets the requirements.

Through a unified API, developers can easily extract relevant Blockchain data in a consistent format without the need to write complex queries to access data. Preconfigured datasets can be extracted from network operators using CQT tokens settled on Moonbeam.

The rewards of Covalent showed an overall upward trend from the first quarter of 2023 to the first quarter of 2024, partly due to the increase in CQT price.

![The Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-53dbb4fd659cf6a7184990c886901658.webp(

Factors to Consider When Choosing an Indexer

Data Customizability

Some indexers ), such as Covalent (, are general-purpose indexers that provide standard pre-configured datasets solely via API. While fast, they lack flexibility and cannot meet the needs of developers requiring custom datasets. The use of an indexer framework allows for more custom data processing to meet specific application requirements.

Security

Index data must be secure; otherwise, dApps built on these indexers are also vulnerable to attacks. If transactions and wallet balances can be manipulated, the dApp may lose liquidity, affecting users. Although all indexers adopt some form of security through staked tokens, other solutions may use additional proofs to enhance security.

Subsquid offers optimistic and zero-knowledge proof options, while Covalent releases proofs that include block hash values. The Graph provides an optimistic challenge window for indexer queries, and SubQuery generates Merkle Mountain proofs for each block, calculating the block hash value for every block in its database.

Speed and Scalability

As the Blockchain grows and transaction volume increases, indexing large amounts of data becomes more cumbersome, requiring more processing power and storage space. Maintaining efficiency becomes more difficult, but the indexer protocol introduces solutions to meet these growing demands.

Subsquid achieves horizontal scaling by adding more nodes to store data, allowing for expansion with hardware improvements. Graph provides parallel streaming data to speed up synchronization, while SubQuery introduces node sharding to accelerate the synchronization process.

Supported Networks

Although most blockchain activities are still taking place on Ethereum, different blockchains are becoming increasingly popular over time. Layer 2, Solana, Move Blockchain, and the Bitcoin ecosystem chain all have their own growing developers and activities, and also require indexing services.

Supporting certain chains that are not supported by other indexer protocols can gain more market share fees. Indexing data-intensive networks ) such as Solana ( is not an easy task, and currently only Subsquid has successfully provided indexing support for it.

![Development of Web3 Data Access: Indexer