Ingest-Tier Scaling

Ingest-Tier Scaling decouples ingestion and indexing by routing Splunk pipeline data through SmartBus and a durable remote queue. This approach provides a scalable ingestion plane, improves data durability, and reduces backpressure in the forwarding path. Depending on how the deployment is configured, the separation can be logical or physical.

Logical separation

In logical separation, ingestion and indexing use separate processing pipelines with distinct responsibilities, but they can run on the same indexer instances and share their compute resources. This provides functional separation without requiring a separate ingestion tier.

• The ingestion pipeline receives incoming data and performs ingestion-stage processing, such as parsing and typing.

• The indexing pipeline processes the prepared data, writes it to indexes, and makes it available for search.

• Because the pipelines share instances and system resources, ingestion and indexing can still contend for CPU, memory, storage, and other available capacity.

In Splunk Enterprise 10.2 and higher, existing Splunk indexer clusters can leverage the benefits of the logical separation in Ingest-Tier Scaling.

Physical separation

Physical separation extends the logical-separation model from separate processing pipelines to separate workload tiers. In this model, ingestion runs in an IngestorCluster managed by Splunk Operator for Kubernetes (SOK). Indexing runs in a separate indexer cluster that is not managed by SOK. The tiers can be scaled independently.

Important: Physical separation is supported only for new physical-separation deployments using Splunk Enterprise v. 10.6 or higher and the Bring Your Own License (BYOL) model on a customer-managed platform (CMP) for Kubernetes. Physical separation is currently supported only on Amazon Web Services (AWS). Support for configuring physical separation on an existing SOK-managed deployment has not been established. Existing configuration and data are not automatically migrated or made available through the new configuration.

For more information about physical separation, see Physical separation of indexing and ingestion.

System / Hardware Requirements

To use Ingest-Tier Scaling, provide the following resources:

• A cloud-hosted message queue like Amazon SQS or Azure Storage Queue
• A SmartStore bucket in Amazon S3 or Azure Blob Storage
• Configure a Key Management System (KMS) key for Amazon or an equivalent for Azure
• For multisite deployments, create the SmartBus queue and SmartStore per site