Running the Archival Store and Service

The Archival Store and Service are part of the Sui data access infrastructure. The stack provides long-term storage and low-latency point lookups of historical onchain data through a gRPC service backed by Google Cloud Bigtable. This stack is optimized for operators and data providers who need to serve historical transactions, checkpoints, objects, and epoch data beyond the retention horizon of full nodes or indexer databases.

The Archival Service exposes the same gRPC LedgerService API as a Sui full node, so existing gRPC clients can query it by changing the endpoint. The service is powered by an indexer (sui-kvstore-alt) that reads checkpoints from the remote checkpoint store and writes processed data to Bigtable, and a gRPC server (sui-kv-rpc) that reads from Bigtable to serve client requests.

See Archival Store and Service for more information on the stack.

Architecture overview

1. Google Cloud Bigtable: The backing store that holds all historical chain data across 14 tables.
2. sui-kvstore-alt (Indexer): Reads checkpoints from the remote checkpoint store and writes processed data to Bigtable.
3. sui-kv-rpc (Archival Service): A gRPC server that reads from Bigtable and exposes the LedgerService API to clients.

Prerequisites

• A Google Cloud Platform (GCP) project with the Bigtable API enabled.
• Two GCP service accounts:
  • Read/write account for the indexer; requires roles/bigtable.user.
  • Read-only account for the gRPC service; requires roles/bigtable.reader.
• A checkpoint bucket for your network, for history:
  • Mainnet: --remote-store-gcs mysten-mainnet-checkpoints-use4 (Requester Pays).
  • Testnet: --remote-store-gcs mysten-testnet-checkpoints-use4 (Requester Pays).
• A Sui full node with gRPC enabled, for the tip.

Authentication

Both binaries authenticate to GCP using Application Default Credentials (ADC), for both Bigtable and GCS.

On GKE: Use Workload Identity. No keys or environment variables needed.

Outside GKE: Point GOOGLE_APPLICATION_CREDENTIALS at a service account JSON key file:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

The public HTTPS endpoints (https://checkpoints.mainnet.sui.io, https://checkpoints.testnet.sui.io) keep only the latest 30 days. Use the GCS buckets above for backfill.

Bigtable setup

Create a Bigtable instance

Create a Bigtable instance in your GCP project. See the Bigtable documentation.

Setting	Recommendation
Storage type	SSD. HDD is untested and not recommended.
Node count	Enable autoscaling. See Scaling.

Do not use Bigtable replication for this deployment. Run a separate Bigtable instance, sui-kvstore-alt indexer, and sui-kv-rpc service in each region.

Create tables

Create these 14 tables, each with a single column family sui and maxversions=1:

Table	Description
checkpoints	Checkpoint summaries, signatures, and contents
checkpoints_by_digest	Checkpoint lookup by digest
transactions	Transaction data, effects, events, and balance changes
objects	Object data keyed by object ID and version
epochs	Epoch start and end data including system state
watermark_alt	Internal indexer watermark tracking
protocol_configs	Protocol configuration per epoch
packages	Package metadata keyed by original ID and version
packages_by_id	Package lookup by ID
packages_by_checkpoint	Package lookup by checkpoint
system_packages	System package data
tx_seq_digest	Transaction sequence number to digest. Backs the List APIs
transaction_bitmap_index	Bitmap index over transaction filters
event_bitmap_index	Bitmap index over event filters

for table in checkpoints checkpoints_by_digest transactions objects epochs \
    watermark_alt protocol_configs packages packages_by_id \
    packages_by_checkpoint system_packages tx_seq_digest \
    transaction_bitmap_index event_bitmap_index; do
  cbt -project <GCP_PROJECT> -instance <INSTANCE_ID> createtable "$table"
  cbt -project <GCP_PROJECT> -instance <INSTANCE_ID> createfamily "$table" sui
  cbt -project <GCP_PROJECT> -instance <INSTANCE_ID> setgcpolicy "$table" sui maxversions=1
done

Storage requirements

Measured 2026-07-27. Grows with network activity.

Table	Mainnet	Testnet
transactions	10.8 TB	3.2 TB
objects	5 TB	1.3 TB
checkpoints	947.7 GB	572.9 GB
transaction_bitmap_index	606.7 GB	311.6 GB
tx_seq_digest	502.5 GB	335.5 GB
event_bitmap_index	250.9 GB	21.3 GB
checkpoints_by_digest	18.2 GB	21.8 GB
epochs	169.2 MB	156.1 MB
packages	30.9 MB	71 MB
watermark_alt	23.5 MB	41.5 MB
packages_by_id	19.1 MB	43.2 MB
packages_by_checkpoint	16.5 MB	38.1 MB
protocol_configs	1.2 MB	610.2 KB
system_packages	1.6 KB	2.9 KB
Total	~18.1 TB	~5.8 TB

Scaling

Use autoscaling. The indexer paces its own writes, so the cluster grows during backfill and settles back at the tip on its own.

Autoscaling setting	Mainnet	Testnet
Minimum nodes	7	3
Maximum nodes	30	10
CPU utilization target	60%	60%

Each SSD node holds up to 5 TB; stay under 70% (3.5 TB per node).

Backup policy

Configure automated backups. Daily with 7-day retention is a reasonable default. See the Bigtable backup documentation.

Indexer setup

sui-kvstore-alt reads checkpoints and writes to Bigtable through 14 pipelines.

	CPU	Memory
At the tip	1 core	1 GB
Backfilling	16 cores	32 GB

Run sui-kvstore-alt

sui-kvstore-alt \
    --chain mainnet \
    my-bigtable-instance \
    --remote-store-gcs mysten-mainnet-checkpoints-use4 \
    --streaming-url http://my-fullnode:9000

For Testnet, use --chain testnet with --remote-store-gcs mysten-testnet-checkpoints-use4.

That one command backfills from genesis and then indexes at the tip. There is no separate backfill config and no cutover. Do not set --rpc-api-url.

CLI parameter	Required	Description
<INSTANCE_ID>	Yes	Bigtable instance ID. Positional.
--chain	Yes	mainnet, testnet, or unknown.
--remote-store-gcs	Yes	GCS bucket to read checkpoint history from.
--streaming-url	No	Fullnode gRPC URL for live checkpoints. Use it.
--config	No	TOML config path. Omit it. See Indexer configuration.
--rpc-api-url	No	Read history from a fullnode instead of a bucket. Alternative to --remote-store-gcs, not an addition.
--remote-store-url	No	HTTPS checkpoint store. Alternative to --remote-store-gcs.
--remote-store-header	No	Repeatable Name:Value header for remote-store requests.
--rpc-username / --rpc-password	No	Credentials for --rpc-api-url.
--checkpoint-timeout-ms	No	Per-checkpoint download timeout. Default 120000.
--checkpoint-connection-timeout-ms	No	Checkpoint-store connection timeout. Default 120000.
--bigtable-project	No	GCP project ID. Defaults to the credentials' project.
--app-profile-id	No	Bigtable app profile. Must be single-cluster routing.
--bigtable-max-decoding-message-size	No	Max decoded gRPC response, bytes. Default 33554432.
--pipeline	No	Repeatable pipeline name. Omit to run all 14.
--first-checkpoint	No	Start checkpoint for pipelines with no watermark. Default 0.
--last-checkpoint	No	Inclusive stop checkpoint.
--metrics-address	No	Prometheus bind address. Default 0.0.0.0:9184.
--write-legacy-data	No	Do not set this.

Exactly one checkpoint source is allowed (--remote-store-gcs, --remote-store-url, --rpc-api-url, and so on); passing two is a startup error. --streaming-url is not one of them and runs alongside whichever you pick.

info

Mysten Labs plans to enable requester-pays on the public checkpoint buckets in the future. Streaming from your own full node avoids these costs.

Indexer configuration

--config takes an optional TOML file. You do not need one. The tables below are reference only.

Adaptive write rate limiting

The indexer paces its own Bigtable writes from the RateLimitInfo directives Bigtable returns on every MutateRows response, combined with observed write latency. This is on by default and replaces manual rate-limit sizing.

• The unit is MutateRows requests per second, not rows. Each request carries up to max-rows rows (default 100).
• It starts at 10 requests per second.
• It grows only when the directive is positive, latency is healthy, and the indexer is driving at least 80% of the current limit.
• It cuts to 0.7x on a severe latency observation or a DeadlineExceeded, Unavailable, or ResourceExhausted response.

It requires a single-cluster-routing app profile. Disable it only if you cannot provide one:

batch-write-flow-control = false

Top-level options

Option	Default	Description
batch-write-flow-control	true	Adaptive write rate limiting.
total-max-rows-per-second	unlimited	Static global rate limit, rows/sec. Leave unset; it starves the adaptive controller.
max-rows-per-second	unlimited	Static per-pipeline rate limit, rows/sec. Leave unset.
bigtable-channel-timeout-ms	60000	Channel-level Bigtable gRPC timeout.
bigtable-connection-pool-size	N/A	Deprecated. Use [bigtable-pool].

Connection pool ([bigtable-pool])

Option	Default	Description
initial-pool-size	10	Channels created at startup.
min-pool-size	1	Minimum channels.
max-pool-size	200	Maximum channels.
min-rpcs-per-channel	5	Load below which the pool scales down.
max-rpcs-per-channel	50	Load above which the pool scales up.
max-resize-delta	2	Max channels removed per scale-down.
downscale-threshold	3	Low-load observations before scaling down.
maintenance-interval-ms	60000	Time between maintenance cycles.
refresh-age-ms	2700000	Channel age before refresh (45 min).
refresh-jitter-ms	300000	Jitter on refresh age (5 min).

Committer ([committer])

Option	Default	Description
write-concurrency	256	Concurrent write tasks per pipeline.
collect-interval-ms	500	How often to flush buffered rows.
watermark-interval-ms	500	How often to update the watermark.
watermark-interval-jitter-ms	0	Jitter on the watermark interval.

Per-pipeline overrides ([pipeline.<name>])

Section names drop the kvstore_ prefix and use kebab-case, except epoch-start and epoch-end, and the bitmap pipelines, which are transaction-bitmap-index and event-bitmap-index.

Option	Default	Description
max-rows	100	Max rows per Bigtable write request.
max-rows-per-second	unlimited	Per-pipeline rate limit. Leave unset.
committer.*	N/A	Any committer field, per pipeline.

See Pipeline architecture: Performance tuning for the remaining framework options.

Pipelines

All 14 are required for full archival service functionality.

Pipeline	Target table	Type
kvstore_checkpoints	checkpoints	Concurrent
kvstore_checkpoints_by_digest	checkpoints_by_digest	Concurrent
kvstore_transactions	transactions	Concurrent
kvstore_objects	objects	Concurrent
kvstore_epochs_start	epochs	Concurrent
kvstore_epochs_end	epochs	Concurrent
kvstore_protocol_configs	protocol_configs	Concurrent
kvstore_packages	packages	Concurrent
kvstore_packages_by_id	packages_by_id	Concurrent
kvstore_packages_by_checkpoint	packages_by_checkpoint	Concurrent
kvstore_system_packages	system_packages	Concurrent
kvstore_tx_seq_digest	tx_seq_digest	Concurrent
kvstore_transaction_dimensions	transaction_bitmap_index	Sequential
kvstore_event_dimensions	event_bitmap_index	Sequential

Backfill

Start the indexer without --first-checkpoint. A full Mainnet backfill takes approximately 3-4 days on a 30-node SSD cluster with a 16 CPU indexer.

Use one indexer instance. One instance can drive a 30-node cluster, the largest scale tested.

Bigtable writes are idempotent, so you can run several instances at the tip for rolling deployments.

Archival service setup

sui-kv-rpc is a gRPC server that reads from Bigtable and exposes the LedgerService API.

Run sui-kv-rpc

sui-kv-rpc --config /etc/sui-kv-rpc/config.toml

instance-id is the only required setting:

config.toml

instance-id = "my-bigtable-instance"
address = "[::]:8000"
metrics-host = "0.0.0.0"
credentials = "/etc/sui-kv-rpc/bigtable-ro-sa.json"
tls-cert = "/secrets/cert.pem"
tls-key = "/secrets/key.pem"
enable-list-apis = true

Run sui-kv-rpc --config-schema to print the full JSON Schema with per-field documentation.

Configuration reference

Setting	Default	Description
instance-id	none	Required. Bigtable instance ID.
address	[::1]:8000	gRPC listen address.
metrics-host	127.0.0.1	Prometheus bind host. Set 0.0.0.0 to be scrapable.
metrics-port	9184	Prometheus bind port.
credentials	ADC	Service account JSON key path.
bigtable-project	credentials' project	GCP project ID.
app-profile-id	none	Bigtable app profile ID.
tls-cert / tls-key	none	PEM cert and key. TLS activates only when both are set.
bigtable-channel-timeout-ms	60000	Channel-level Bigtable gRPC timeout.
bigtable-initial-pool-size	10	Channels created at startup.
bigtable-min-pool-size	1	Minimum channels.
bigtable-max-pool-size	200	Maximum channels.
enable-list-apis	false	Serve the three List APIs. Requires the tx_seq_digest and both bitmap pipelines.
request-bigtable-concurrency	50	Per-request cap on concurrent Bigtable operations.
ledger-history	see below	List API tuning.
stages	see below	Read concurrency tuning.

The old positional and flag form (<INSTANCE_ID>, <ADDRESS>, --credentials, --tls-cert, and so on) still works, takes precedence over the config file, and logs a deprecation warning. Use the config file.

List API tuning ([ledger-history])

Only used when enable-list-apis = true. The [ledger-history.list-transactions], [ledger-history.list-events], and [ledger-history.list-checkpoints] tables each take timeout-ms (default 30000), default-limit-items, max-limit-items, and render-ahead (default 4).

Method	default-limit-items	max-limit-items
list-transactions	50	500
list-events	50	1000
list-checkpoints	10	50

The [ledger-history] table takes bitmap-bucket-budget-tx (4000), bitmap-bucket-budget-event (4000), max-bitmap-filter-literals (10), and bitmap-drain-probe-rows (50).

Read stages ([stages.<name>])

The [stages.tx-seq-digest], [stages.transactions], [stages.objects], and [stages.checkpoints] tables each take chunk-size (default 100) and concurrency.

Stage	Default concurrency
tx-seq-digest	10
transactions	25
objects	50
checkpoints	10

gRPC API

Same LedgerService API as a Sui full node, so existing gRPC clients work by changing the endpoint URL. Server reflection is enabled.

Method	Description	Limit
GetServiceInfo	Chain ID, current epoch, latest checkpoint, server version.	N/A
GetObject	Object by ID, optionally at a version.	N/A
BatchGetObjects	Batch object lookup. Requires exact versions.	1000
GetTransaction	Transaction by digest.	N/A
BatchGetTransactions	Batch transaction lookup.	200
GetCheckpoint	Checkpoint by sequence number or digest.	N/A
GetEpoch	Epoch data by epoch number.	N/A
ListTransactions	Streaming transaction listing, optionally filtered.	50 default, 500 max
ListEvents	Streaming event listing, optionally filtered.	50 default, 1000 max
ListCheckpoints	Streaming checkpoint listing, optionally filtered.	10 default, 50 max

The three List methods return Unimplemented unless enable-list-apis = true. There are no subscription RPCs. Every method except GetServiceInfo supports read_mask.

Health check

GET /health on port 8081.

tip

Production hardening checklist:

• Least-privilege service accounts. roles/bigtable.user for the indexer, roles/bigtable.reader for the service. Do not share one account.
• Credentials in a secret manager. Prefer Workload Identity on GKE over key files.
• Test backup and restore. Configure Bigtable backups and verify you can restore, not just that backups exist.
• TLS. Set tls-cert and tls-key, and restrict the gRPC and health endpoints to trusted networks.
• Single-cluster-routing app profile for the indexer. Nothing validates this at runtime.

See Security Best Practices.

Monitoring

Both binaries export Prometheus metrics on port 9184 at /metrics.

The indexer exports the standard indexer framework metrics with a kvstore_alt_ prefix, so existing Postgres indexer dashboards apply.

scrape_configs:
  - job_name: sui-kvstore
    static_configs:
      - targets: ['<INDEXER_HOST>:9184']
  - job_name: sui-kv-rpc
    static_configs:
      - targets: ['<RPC_HOST>:9184']

Write flow-control metrics

Metric	Type	Description
kv_bt_flow_control_effective_qps	gauge	Current admitted MutateRows requests per second.
kv_bt_flow_control_last_observation_start_qps	gauge	Actual request rate last window. Well below effective_qps means the indexer is not demand-limited.
kv_bt_flow_control_write_latency_window_avg_ms	gauge	Ten-second average write latency.
kv_bt_flow_control_write_latency_baseline_ms	gauge	Learned healthy-latency baseline.
kv_bt_flow_control_throttle_ms	histogram	Time each write waited for admission.
kv_bt_flow_control_events_total	counter	Lifecycle events, labeled event.

The bitmap pipelines export bitmap_watermark_lag_ms_*, bitmap_write_chunk_latency_seconds_*, bitmap_write_retry_rows_total_*, bitmap_row_key_size_bytes_*, and bitmap_serialized_bitmap_size_bytes_*, suffixed with the pipeline name rather than labeled.

Archival service metrics

Metric	Type	Description
rpc_request_latency	histogram	End-to-end request latency.
rpc_request_handler_latency	histogram	Handler latency, by path.
rpc_first_chunk_latency	histogram	Time to first streamed chunk, by path.
rpc_requests	counter	Request count, by status.
rpc_inflight_requests	gauge	Concurrent requests in flight, by path.
kv_get_latency_ms	histogram	Bigtable read latency per batch request.
kv_get_latency_ms_per_key	histogram	Bigtable read latency divided by batch size.
kv_scan_latency_ms	histogram	Bigtable scan latency.
kv_bt_chunk_latency_ms	histogram	Bigtable processing time per response chunk.
kv_get_success	counter	Successful Bigtable reads.
kv_get_not_found	counter	Bigtable reads that found no row.
kv_get_errors	counter	Failed Bigtable reads.
kv_scan_success	counter	Successful Bigtable scans.
kv_rpc_response_render_latency_ms	histogram	Response render latency.
kv_rpc_response_page_bytes	histogram	Rendered page size.
kv_rpc_stream_first_frame_latency_ms	histogram	Time to first frame of a List response.
kv_rpc_bitmap_buckets_evaluated	histogram	Bitmap buckets evaluated per query.
kv_rpc_bitmap_buckets_discarded	histogram	Bitmap buckets discarded per query.
kv_rpc_bigtable_permit_wait_ms	histogram	Wait for a per-request Bigtable permit.
bt_pool_pool_size	gauge	Channels in the connection pool.
bt_pool_channels_replaced	counter	Channels replaced due to age refresh.
bt_pool_rpcs_completed	counter	RPCs completed through the pool.
thread_stall_duration_sec	histogram	Tokio thread stall duration.

The kv_* read metrics are labeled client and table.

Recommended log levels

Environment	RUST_LOG
Production	info
Debugging	info,sui_kvstore=debug,sui_indexer_alt_framework=debug