Skip to main content

Custom Indexers

Sui generates rich, complex data, including transactions, events, object changes, and more. While Sui provides standard APIs to access this data, many applications need customized data processing for workflows like tracking specific events, aggregating analytics, building dashboards, or creating specialized databases. Custom indexers let you extract, transform, and store the specific blockchain data you need. Instead of querying Sui APIs repeatedly or building complex filtering logic, you process the raw blockchain data once and store it in your preferred format.

Custom indexers enable use cases such as tracking DEX trading volumes, monitoring NFT collection activity, building analytics dashboards, or aggregating cross-chain data. You build custom indexers using the sui-indexer-alt-framework, a Rust framework that provides production-ready components for data ingestion, processing, and storage while giving you full control over your indexing logic.

When to use​

Use custom indexers when your application's data needs cannot be fulfilled with managed services from RPC providers, or if the General-Purpose Indexer is too complex for your use case. Custom indexers are a good choice for app-specific or protocol-specific logic and data layout.

Consider custom indexers when:

  • You need more control over the types, granularity, and retention period of data for your application.
  • You have specific query patterns that are not served with gRPC or GraphQL RPC.
  • You want to optimize storage costs through custom pruning strategies based on your retention needs.

If you set up your own indexer, you are responsible for its ongoing maintenance and the related infrastructure and operational costs.

How custom indexers fit into the application stack​

At a high level, the indexing framework is a streaming pipeline that continuously polls for the latest available checkpoints from a predefined data source and streams that checkpoint data to your pipeline's processing logic.

Checkpoints on Sui are batches of transactions that represent consistent blockchain state snapshots. Each checkpoint contains complete transaction details, events, object changes, and execution results with guaranteed ordering. See Life of a Transaction for more information on checkpoints.

Overview

Architecture overview​

The following diagram details the custom indexer architecture.

Custom indexer architecture

Your indexer runs as one executable program with multiple coordinated background tasks (threads). The ingestion layer polls for new checkpoints while your processing pipelines transform and store data, all within the same process.

Data sources​

The indexing framework supports multiple data sources to give you flexibility in how you ingest checkpoint data. These sources fall into two categories:

Polling-based sources that periodically check for new checkpoints:

  • Remote stores connect to public checkpoint repositories such as https://checkpoints.mainnet.sui.io. This option offers the simplest setup and does not require you to operate any infrastructure.

  • Local files process checkpoint files produced by a local full node. This method offers the lowest latency but is recommended only for testing because the node does not automatically clean up these files.

  • RPC endpoints connect directly to a full node RPC endpoint. This approach lets you pull checkpoints from a trusted source you operate or access networks (like Devnet) that lack a remote store.

Push-based sources provide real-time checkpoint data as it becomes available:

  • gRPC streaming pushes new checkpoints from full nodes as soon as they are available, providing lower latency than polling. Because streaming only emits the latest checkpoints, you must configure a polling-based fallback source to fetch historical data and ensure reliability.

For configuration examples and guidance on when to use each source, see Checkpoint Data Sources.

Ingestion layer​

The indexing framework manages the ingestion layer, which handles the complex task of reliably fetching and distributing checkpoint data. The Broadcaster receives checkpoints from data sources and efficiently distributes them to multiple processing pipelines running in parallel. This is essential for indexers that run multiple different data processing workflows simultaneously.

Broadcaster applies backpressure based on the highest watermark reported by its subscribers, ensuring it does not push data faster than the slowest pipeline can process it.

Processing layer​

Both the indexing framework and your code manage the processing layer, which is where your custom logic integrates with the framework. The pipeline framework components are the heart of the system, as they orchestrate checkpoint processing, manage concurrency to maximize throughput, maintain watermarks to track progress and ensure data consistency, and coordinate the entire data flow from ingestion to storage. The framework components adapt based on your chosen pipeline type:

  • Sequential pipelines use different components optimized for in-order processing with batching capabilities.

  • Concurrent pipelines use components like Collector, Committer, and Pruner for high-throughput out-of-order processing.

Storage layer​

The indexing framework abstracts database operations through a flexible storage layer. PostgreSQL comes with built-in support using Diesel ORM, providing production-ready database operations, connection pooling, migrations, and watermark management out of the box. For custom database implementations, you can implement the framework's storage interfaces to use any database, for example, MongoDB for document storage or ClickHouse for analytics.

Building custom indexers​

To build your own pipelines for application-specific data, see Building Your First Custom Indexer. You can also reach out to a data indexer operator that might already have set one up for your needs.

•
Build a Custom Indexer

Build a custom indexer using the sui-indexer-alt-framework module. The example indexer demonstrates a sequential pipeline that extracts transaction digests from Sui checkpoints and stores them in a local PostgreSQL.

•
Indexer Pipeline Architecture

The sui-indexer-alt-framework provides two distinct pipeline architectures. Understand the differences between the sequential and concurrent pipelines that the sui-indexer-alt-framework provides to decide which best suits your project needs.

•
Life of a Transaction

The life of a transaction on the Sui network has some differences compared to those from other blockchains.