Indexer Runtime and Performance

Proper configuration and resource monitoring delivers the most performant custom indexer possible. For example:

• Runtime configuration options for ingestion, database connections, and pipeline selection, as well as purposeful use of debugging tools like tokio_console help dial in your indexer performance.

• A sensible strategy targeting efficient data pruning for your tables keeps them performant over time.

• Following best practices for exposing and extending Prometheus metrics helps you keep track of indexer performance.

Together, these techniques help you run indexers that are fast, resource-efficient, and easier to monitor in both development and production.

Fine-tuning configurations

The indexing framework provides multiple levels of configuration to optimize performance for different use cases. This section covers basic configuration options, while complex pipeline-specific tuning is covered in Indexer Pipeline Architecture.

Ingestion layer configuration

Control how checkpoint data is fetched and distributed:

let ingestion_config = IngestionConfig {
    // Buffer size across all downstream workers (default: 5000)
    checkpoint_buffer_size: 10000,
    
    // Concurrent checkpoint fetches (default: 200)
    ingest_concurrency: 500,
    
    // Retry interval for missing checkpoints in ms (default: 200)
    retry_interval_ms: 100,
};

Tuning guidelines:

• checkpoint_buffer_size: Increase for high-throughput scenarios, decrease to reduce memory usage.
• ingest_concurrency: Higher values improve ingestion speed but increase network/storage load.
• retry_interval_ms: Lower values reduce latency for live data, higher values reduce unnecessary retries.

Database connection configuration

let db_args = DbArgs {
    // Connection pool size (default: 100)
    db_connection_pool_size: 200,
    
    // Connection timeout in ms (default: 60,000)
    db_connection_timeout_ms: 30000,
    
    // Statement timeout in ms (default: None)
    db_statement_timeout_ms: Some(120000),
};

Tuning guidelines:

• db_connection_pool_size: Size based on write_concurrency across all pipelines.
• db_connection_timeout_ms: Reduce for faster failure detection in high-load scenarios.
• db_statement_timeout_ms: Set based on expected query complexity and database performance.

Command-line arguments

Include the following command-line arguments to help focus processing. These values are for demonstration. Use values that make sense to your environment and goals.

# Checkpoint range control
--first-checkpoint 1000000     # Start from specific checkpoint
--last-checkpoint 2000000      # Stop at specific checkpoint

# Pipeline selection
--pipeline "tx_counts"          # Run specific pipeline only
--pipeline "events"             # Can specify multiple pipelines

# Watermark behavior
--skip-watermark  

Use cases:

• Checkpoint range: Essential for backfills and historical data processing.
• Pipeline selection: Useful for selective reprocessing or testing.
• Skip watermark: Enables faster backfills when watermark consistency isn't required.

Pipeline-specific advanced tuning

For complex configuration scenarios requiring deep understanding of pipeline internals:

• Sequential pipeline architecture
• Concurrent pipeline architecture

Tokio runtime debugging

For performance-sensitive pipelines or when troubleshooting async runtime issues, the sui-indexer-alt-framework integrates with tokio-console, a powerful debugger for async Rust applications. This tool provides real-time insights into task execution, helping identify performance bottlenecks, stuck tasks, and memory issues.

GitHub repo for tokio-console

When to use Tokio console

The Tokio console is particularly useful for:

• Performance debugging: Identifying slow or blocking tasks.
• Memory analysis: Finding tasks consuming excessive memory.
• Concurrency issues: Detecting tasks that never yield or wake themselves excessively.
• Runtime behavior: Understanding task scheduling and execution patterns.

Setup instructions

Consult the README in the Tokio GitHub repo for additional information.

1. Add dependencies

   Add the telemetry_subscribers dependency to your Cargo.toml:

   [dependencies]
   telemetry_subscribers = { git = "https://github.com/MystenLabs/sui.git", branch = "main" }

2. Initialize telemetry

   Add telemetry initialization at the beginning of your main function:

   #[tokio::main]
   async fn main() -> Result<()> {
       // Enable tracing, configured by environment variables
       let _guard = telemetry_subscribers::TelemetryConfig::new()
           .with_env()
           .init();
   
       // Your indexer code here...
   }

3. Run with console enabled

   Start your indexer with the required flags:

   RUSTFLAGS="--cfg tokio_unstable" TOKIO_CONSOLE=1 cargo run

   Flag explanations:

   • TOKIO_CONSOLE=1: Enables tokio-console integration in telemetry_subscribers.
   • RUSTFLAGS="--cfg tokio_unstable": Required by tokio-console to collect task instrumentation data.

4. Launch the console dashboard

   # Install tokio-console if not already installed
   cargo install tokio-console
   
   # Connect to your running indexer (default: localhost:6669)
   tokio-console

   If successful, the dashboard appears with information about every running Tokio task:

   

Console features

For detailed information about the console dashboard, available views, warnings, and diagnostic capabilities, refer to the official tokio-console documentation

Production considerations

tokio-console introduces runtime overhead and should be used carefully in production. While it's safe to use regularly during development and staging, production usage requires careful evaluation. Before enabling in production, you should run performance benchmarks with and without Tokio console enabled to measure the impact on your specific workload. Consider enabling only during maintenance windows or targeted troubleshooting sessions while monitoring system resources.

Metrics

The sui-indexer-alt-framework provides built-in Prometheus metrics for monitoring indexer performance and health. All metrics are automatically exposed via HTTP and can be extended with custom metrics.

Built-in metrics

The framework tracks extensive metrics across ingestion, pipeline processing, database operations, and watermark management. For a complete list of available metrics with descriptions, refer to the IndexerMetrics struct in sui-indexer-alt-framework/src/metrics.rs.

crates/sui-indexer-alt-framework/src/metrics.rs

#[derive(Clone)]
pub struct IndexerMetrics {
    // Statistics related to fetching data from the remote store.
    pub total_ingested_checkpoints: IntCounter,
    pub total_ingested_transactions: IntCounter,
    pub total_ingested_events: IntCounter,
    pub total_ingested_inputs: IntCounter,
    pub total_ingested_outputs: IntCounter,
    pub total_ingested_bytes: IntCounter,
    pub total_ingested_transient_retries: IntCounterVec,
    pub total_ingested_not_found_retries: IntCounter,

    // Checkpoint lag metrics for the ingestion pipeline.
    pub latest_ingested_checkpoint: IntGauge,
    pub latest_ingested_checkpoint_timestamp_lag_ms: IntGauge,
    pub ingested_checkpoint_timestamp_lag: Histogram,

    pub ingested_checkpoint_latency: Histogram,

    // Statistics related to individual ingestion pipelines' handlers.
    pub total_handler_checkpoints_received: IntCounterVec,
    pub total_handler_checkpoints_processed: IntCounterVec,
    pub total_handler_rows_created: IntCounterVec,

    pub latest_processed_checkpoint: IntGaugeVec,
    pub latest_processed_checkpoint_timestamp_lag_ms: IntGaugeVec,
    pub processed_checkpoint_timestamp_lag: HistogramVec,

    pub handler_checkpoint_latency: HistogramVec,

    // Statistics related to individual ingestion pipelines.
    pub total_collector_checkpoints_received: IntCounterVec,
    pub total_collector_rows_received: IntCounterVec,
    pub total_collector_batches_created: IntCounterVec,
    pub total_committer_batches_attempted: IntCounterVec,
    pub total_committer_batches_succeeded: IntCounterVec,
    pub total_committer_batches_failed: IntCounterVec,
    pub total_committer_rows_committed: IntCounterVec,
    pub total_committer_rows_affected: IntCounterVec,
    pub total_watermarks_out_of_order: IntCounterVec,
    pub total_pruner_chunks_attempted: IntCounterVec,
    pub total_pruner_chunks_deleted: IntCounterVec,
    pub total_pruner_rows_deleted: IntCounterVec,

    // Checkpoint lag metrics for the collector.
    pub latest_collected_checkpoint: IntGaugeVec,
    pub latest_collected_checkpoint_timestamp_lag_ms: IntGaugeVec,
    pub collected_checkpoint_timestamp_lag: HistogramVec,

    // Checkpoint lag metrics for the committer.
    // We can only report partially committed checkpoints, since the concurrent committer isn't aware of
    // when a checkpoint is fully committed. So we report whenever we see a checkpoint. Since data from
    // the same checkpoint is batched continuously, this is a good proxy for the last committed checkpoint.
    pub latest_partially_committed_checkpoint: IntGaugeVec,
    pub latest_partially_committed_checkpoint_timestamp_lag_ms: IntGaugeVec,
    pub partially_committed_checkpoint_timestamp_lag: HistogramVec,

    // Checkpoint lag metrics for the watermarker.
    // The latest watermarked checkpoint metric is already covered by watermark_checkpoint_in_db.
    // While we already have watermark_timestamp_in_db_ms metric, reporting the lag explicitly
    // for consistency.
    pub latest_watermarked_checkpoint_timestamp_lag_ms: IntGaugeVec,
    pub watermarked_checkpoint_timestamp_lag: HistogramVec,

    pub collector_gather_latency: HistogramVec,
    pub collector_batch_size: HistogramVec,
    pub committer_commit_latency: HistogramVec,
    pub committer_tx_rows: HistogramVec,
    pub watermark_gather_latency: HistogramVec,
    pub watermark_commit_latency: HistogramVec,
    pub watermark_pruner_read_latency: HistogramVec,
    pub watermark_pruner_write_latency: HistogramVec,
    pub pruner_delete_latency: HistogramVec,

    pub watermark_epoch: IntGaugeVec,
    pub watermark_checkpoint: IntGaugeVec,
    pub watermark_transaction: IntGaugeVec,
    pub watermark_timestamp_ms: IntGaugeVec,
    pub watermark_reader_lo: IntGaugeVec,
    pub watermark_pruner_hi: IntGaugeVec,

    pub watermark_epoch_in_db: IntGaugeVec,
    pub watermark_checkpoint_in_db: IntGaugeVec,
    pub watermark_transaction_in_db: IntGaugeVec,
    pub watermark_timestamp_in_db_ms: IntGaugeVec,
    pub watermark_reader_lo_in_db: IntGaugeVec,
    pub watermark_pruner_hi_in_db: IntGaugeVec,
}

Key metric categories include:

• Ingestion metrics: Global checkpoint and transaction processing stats.
• Pipeline metrics: Per-pipeline processing performance (labeled by pipeline name).
• Database metrics: Batch processing, commit latency, and failure rates.
• Watermark metrics: Progress tracking and lag measurements.

Accessing metrics

By default, metrics are served on 0.0.0.0:9184/metrics in Prometheus format. You can override this address:

• Via the command line:

  $ cargo run --metrics-address "0.0.0.0:8080"

• Or, via the builder configuration:

  let cluster = IndexerCluster::builder()
      .with_metrics_args(MetricsArgs {
          metrics_address: "0.0.0.0:8080".parse().unwrap(),
      })

Adding metric labels

It's highly recommended to add a metric label prefix to differentiate between different indexer instances:

let cluster = IndexerCluster::builder()
    .with_metric_label("my_custom_indexer")

This prefixes all default metrics (for example, my_custom_indexer_indexer_total_ingested_checkpoints).

Adding custom metrics

You can register custom metrics using the shared Registry:

use prometheus::{IntCounter, register_int_counter_with_registry}

// Get the metrics registry before running the cluster
let cluster = IndexerCluster::builder()
    .with_database_url(database_url)
    .with_args(args)
    .build()
    .await?;

// Register custom metrics
let custom_counter = register_int_counter_with_registry!(
    "my_custom_metric_total",
    "Description of my custom metric",
    cluster.metrics().registry(),
)?;

Metrics collection and visualization

The framework only exposes metrics via HTTP, and you're responsible for collection and storage. For example, some indexers use the Alloy agent to scrape the /metrics endpoint and write to a metrics database for visualization and alerting.

See Collect Prometheus metrics on the Grafana Labs website.

Pruning best practices

If your table prunes data in checkpoint order, you can improve efficiency by partitioning the table, so that you can prune by just dropping whole partitions. This usually requires only a single pruner task, avoiding the need for concurrent batched DELETE queries.

Partition size is the key tradeoff: you need N full partitions plus one trailing active partition that cannot be dropped until it falls outside the read window. Larger partitions mean fewer total partitions to scan, reducing read amplification. Smaller partitions reduce retention overhead but increase the number of partitions queried. For example, keeping 10M checkpoints could use a 10M partition plus 1 active partition (querying at most 2 partitions) or 10 1M partitions plus 1 active partition (querying 11 partitions). In practice, per-epoch partitioning tends to produce too many small partitions, and not partitioning at all leads to performance degradation from repeated DELETEs creating dead tuples and fragmentation.

Pipelines with more complex pruning rules can still benefit. For example, in consistent pipelines, a record can only be pruned if there is a newer record for the same key or if it is a sentinel row. A separate, leaner table can track deletable records and be partitioned by checkpoint to make pruning efficient.

Implementation

You can use pg_partman to simplify partition management. Configure a partitioned table with create_parent, followed by a cron job to periodically run_maintenance. You might need to iterate to determine the correct frequency of run_maintenance.

info

Tables with primary key or unique constraints must additionally include all columns used in the partitioning scheme.

CREATE EXTENSION pg_partman;

SELECT create_parent(
    p_parent_table := 'public.tx_calls',
    p_control := 'cp_sequence_number',
    p_type := 'range',
    p_interval := '10000000',
    p_start_partition := '70000000',
    p_premake := 5,
);

-- Create a maintenance job to run every hour
SELECT cron.schedule('0 * * * *', $$SELECT run_maintenance(p_analyze := false)$$);

This can be included as a migration:

CREATE EXTENSION IF NOT EXISTS pg_partman;
CREATE EXTENSION IF NOT EXISTS pg_cron;

-- Function to safely set up partitioning for a table
CREATE OR REPLACE FUNCTION safe_create_parent(
    p_schema text,
    p_table text,
    p_control text,
    p_type text,
    p_interval text,
    p_start_partition text,
    p_premake int
) RETURNS text AS $$
DECLARE
    full_table_name text := p_schema || '.' || p_table;
    result_message text;
BEGIN
    -- Check if the table is already managed by pg_partman
    IF NOT EXISTS (
        SELECT 1 FROM part_config
        WHERE parent_table = full_table_name
    ) THEN
        -- If not managed, set up partitioning
        PERFORM create_parent(
            p_parent_table := full_table_name,
            p_control := p_control,
            p_type := p_type,
            p_interval := p_interval,
            p_start_partition := p_start_partition,
            p_premake := p_premake
        );
        result_message := 'CREATED: Partitioning set up for ' || full_table_name;
    ELSE
        -- Table is already managed
        result_message := 'EXISTS: Table ' || full_table_name || ' is already managed by pg_partman';
    END IF;
    
    RETURN result_message;
END;
$$ LANGUAGE plpgsql;

SELECT safe_create_parent('public', 'tx_affected_objects', 'cp_sequence_number', 'range', '10000000', '70000000', 1);

SELECT cron.schedule('0 * * * *', 'SELECT run_maintenance()');

The accompanying down.sql looks like the following:

-- Remove the scheduled job
DELETE FROM cron.job WHERE command = 'SELECT run_maintenance()';

-- Undo partitioning for all tables
SELECT undo_partition('public.tx_affected_objects', p_keep_table := true);

You can confirm that the partitioned table and cron job are set up correctly with the following SQL queries:

SELECT parent_table, partition_type, control, interval_val
FROM part_config
ORDER BY parent_table;

-- Check maintenance job is scheduled
SELECT jobid, schedule, command, nodename, database, username 
FROM cron.job
WHERE command = 'SELECT run_maintenance()';

Related links

•

Custom Indexing Framework

The sui-indexer-alt-framework is a powerful Rust framework for building high-performance, custom blockchain indexers on Sui. It provides customizable, production-ready components for data ingestion, processing, and storage.

•

Custom Indexer

Establishing a custom indexer helps improve latency, allows pruning the data of your Sui full node, and provides efficient assemblage of checkpoint data.