The Importance of Blockchain Data and the Application of Indexers

The core of blockchain technology lies in data. Data is not only the foundation for developing decentralized applications ( dApp ), but also the key to maintaining network integrity. Although the industry currently focuses mainly on data availability ( DA ), which ensures that network participants can access the latest transaction data for verification, data accessibility, while equally important, is often overlooked.

In the era of modular Blockchain, DA solutions have become an indispensable part. These solutions ensure that participants can access transaction data, enabling real-time verification and maintaining network integrity. However, the DA layer is more like a temporary bulletin board rather than a permanent database. Data will be deleted over time, just like posters on a billboard will eventually be replaced by new content.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for developing dApps and conducting Blockchain analysis. Although less discussed, data accessibility is equally important as data availability. Both play different but complementary roles in the Blockchain ecosystem, and a comprehensive data management approach must address both issues simultaneously to support robust and efficient Blockchain applications.

Traditional Methods of Blockchain Data Retrieval

Since its inception, Blockchain has fundamentally changed infrastructure, driving innovation in dApps across areas such as gaming, finance, and social networks. However, building these dApps requires access to massive amounts of Blockchain data, a process that is both difficult and costly.

For dApp developers, one option is to self-host and run archive RPC nodes. These nodes store all historical Blockchain data, allowing for full access. However, the maintenance costs are high, and the query capabilities are limited, making it difficult to meet the specific needs of developers. While running lightweight nodes is less expensive, the data retrieval capabilities are limited, which may impact the normal operation of the dApp.

Another method is to use commercial RPC node services. These providers are responsible for the cost and management of the nodes, providing data through RPC endpoints. Public RPC endpoints are free to use but have rate limits that may affect the dApp user experience. Private RPC endpoints perform better, but even simple data retrieval requires a lot of communication, making it inefficient and difficult to scale. Additionally, they often lack cross-network compatibility.

Better Solution: Blockchain Indexer

Blockchain indexers play a key role in organizing chain data and storing it in databases for query purposes, hence they are referred to as "the search engine of the blockchain." They index blockchain data and use SQL-like query languages such as GraphQL API( to make data readily available. Indexers provide developers with a unified data query interface, significantly simplifying the data retrieval process.

Different types of indexers optimize data retrieval in various ways:

1. Full Node Indexer: Runs a complete Blockchain node and extracts data directly, ensuring data is complete and accurate, but requires a large amount of storage and processing power.

2. Lightweight Indexer: Relies on full nodes to fetch specific data on demand, reducing storage requirements but may increase query time.

3. Specialized Indexer: Optimized for specific types of data or specific Blockchains, such as NFT data or DeFi transactions.

4. Aggregated Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface suitable for multi-chain dApps.

Ethereum alone requires 3TB of storage space, which continues to expand as the Blockchain grows. The indexer protocol deploys multiple indexers for efficient indexing and fast querying of large amounts of data, something that RPC cannot achieve.

Indexers support complex queries, flexible data filtering, and post-extraction analysis. Some indexers can also aggregate multi-source data, avoiding the need for multiple APIs for multi-chain dApps. Through distributed nodes, indexers provide enhanced security and performance, while centralized RPC providers may experience interruptions and downtime.

Overall, compared to RPC node services, indexers improve the efficiency and reliability of data retrieval while reducing the cost of deploying a single node. This makes the Blockchain indexer protocol the preferred solution for dApp developers.

![Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-16396b955382c2c74010c264affdca46.webp(

Indexer Application Scenarios

As mentioned earlier, building a dApp requires retrieving and reading Blockchain data to operate services. This includes various types of dApps, such as DeFi, NFT platforms, games, and even social networks, as these platforms need to read data first to execute other transactions.

DeFi

DeFi protocols require different information to provide users with specific prices, rates, fees, etc. The automated market maker )AMM( needs certain liquidity pool prices and liquidity information to calculate swap rates, while lending protocols need utilization rates to determine lending rates and the debt ratio for liquidation. Inputting this information into the dApp is essential before calculating the rates executed by users.

Game

GameFi requires quick indexing and access to data to ensure a smooth gaming experience for users. Only with lightning-fast data retrieval and execution can Web3 games compete with Web2 games in terms of performance, thus attracting more users. These games need data such as land ownership, in-game token balances, and in-game actions. By using indexers, they can better ensure stable data flow and uptime, guaranteeing a perfect gaming experience.

NFT

NFT markets and lending platforms need to index data to access various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing such data can avoid browsing each NFT individually to find ownership or NFT attribute data.

Whether it is the DeFi automated market maker )AMM( that requires price and liquidity information, or the SocialFi applications that need to update new user posts, the ability to quickly retrieve data is crucial for the normal operation of dApps. With the help of indexers, they can efficiently and accurately retrieve data, thus providing a smooth user experience.

Analysis

The indexer provides a method to extract specific data from the raw Blockchain data ), including smart contract events in each Block (. This offers opportunities for more specific data analysis, thus providing comprehensive insights.

For example, perpetual trading protocols can identify which tokens have high trading volumes and which tokens will incur fees, thereby deciding whether to list these tokens as perpetual contracts on their platform. DEX developers can create dashboards for their products to gain insights into which liquidity pools offer the highest returns or the most liquidity. They can also create public dashboards that allow developers to freely and flexibly query any type of data to display on the charts.

With multiple blockchain indexers available, identifying the differences between indexing protocols is crucial to ensure that developers choose the indexer that best fits their needs.

Blockchain Indexer Overview

The Graph

The Graph is the first indexing protocol launched on Ethereum, simplifying the previously difficult access to transaction data queries. It uses subgraph definitions and filters to collect subsets of data from the blockchain, such as all transactions related to a specific transaction pool.

Using index proof, indexers stake the native token GRT for indexing and query services, and delegators can choose to stake their tokens here. Curators can access high-quality subgraphs to help indexers determine which subgraphs to curate data for to earn the best query fees. As The Graph transitions to a greater degree of decentralization, it will ultimately cease its hosting services and require subgraphs to upgrade to its network, while providing upgraded indexers.

Its infrastructure brings the average cost per million queries to $40, significantly lower than the cost of self-hosted nodes. Using file data sources, it also supports parallel indexing of both on-chain and off-chain data for efficient data retrieval.

The rewards for The Graph's indexers have steadily increased over the past few quarters. This is partly attributed to the rise in query volume, and is also related to the increase in token prices, as they plan to integrate AI-assisted queries in the future.

![The development of Web3 data access: Introduction to indexers and related projects])https://img-cdn.gateio.im/webp-social/moments-53dbb4fd659cf6a7184990c886901658.webp(

Subsquid

Subsquid is a peer-to-peer, horizontally scalable decentralized data lake that efficiently aggregates a large amount of on-chain and off-chain data, protected by zero-knowledge proofs. As a decentralized worker network, each node is responsible for storing data from a specific block subset, accelerating the data retrieval process by quickly identifying nodes that have the required data.

Subsquid supports real-time indexing, allowing indexing before the block is finalized. It also supports storing data in formats chosen by developers, making it easier to analyze using tools like BigQuery, Parquet, or CSV. Additionally, subgraphs can be deployed on the Subsquid network without code, eliminating the need to migrate to the Squid SDK.

Although still in the testnet phase, Subsquid has achieved significant results, with over 80,000 testnet users, more than 60,000 Squid indexers deployed, and over 20,000 verified developers on the network. Recently, Subsquid launched the mainnet of its data lake.

In addition to indexing, the Subsquid Network data lake can also replace RPC in use cases such as analytics, ZK/TEE co-processors, AI agents, and Oracles.

SubQuery

SubQuery is a decentralized middleware infrastructure network that provides RPC and indexing data services. It originally supported the Polkadot and Substrate networks and has now expanded to include over 200 chains. Its operation is similar to The Graph, which uses indexing proofs; indexers index data and provide query requests, while delegators stake their shares to the indexers. However, it introduces consumers to submit purchase orders, indicating that the indexers' revenue is guaranteed, rather than the administrators.

It will introduce SubQuery data nodes that support sharding to prevent constant synchronization of new data between each node, thereby optimizing query efficiency while moving towards greater decentralization. Users can choose to pay a computing fee of approximately 1 SQT token for every 1000 requests, or set custom fees for the indexer through the protocol.

Although SubQuery launched its token earlier this year, the issuance rewards for nodes and delegators are also on the rise, which represents an increasing number of query services provided on its platform. Since the TGE, the total amount of staked SQT has increased from 6 million to 125 million, highlighting the growth of network participation.

Covalent

Covalent is a decentralized indexing network, created by block sample producers )BSP( network nodes through batch exporting to create copies of blockchain data, and publish proofs on the Covalent L1 blockchain. This data is then refined by block result producers )BRP( nodes according to established rules to filter out the data that meets the requirements.

Through a unified API, developers can easily extract relevant blockchain data in a consistent request and response format, without the need to write custom complex queries to access the data. The CQT token settled on Moonbeam can be used as a means of payment to retrieve these pre-configured data sets from network operators.

The rewards of Covalent seem to show an overall upward trend from the first quarter of 2023 to the first quarter of 2024, partly due to the increase in the price of Covalent token CQT.

![Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-52ee29205aa307720198994a5f3de61f.webp(

Factors to Consider When Choosing an Indexer

Customizability of Data

Some indexers ) like Covalent ( are general-purpose indexers that provide standard preconfigured datasets through APIs. While they may be fast, they do not offer the flexibility needed for developers requiring custom datasets. Using an indexer framework allows for more customized data processing to meet application-specific needs.

Security

Index data must be secure, otherwise dApps built on these indexers are also vulnerable to attacks. For example, if transactions and wallet balances can be manipulated, the dApp may lose liquidity, which in turn affects its users. Although all indexers adopt some form of security through staking tokens, other indexer solutions may use proofs to further enhance security.

Subsquid offers options using optimistic and zero-knowledge proofs, while Covalent also released proofs that include block hash values. The Graph provides a dispute challenge period for indexer queries in an optimistic challenge window manner, while SubQuery provides for each