MySQL receiver

This receiver queries MySQL and MariaDB global status and InnoDB tables.

The MySQL receiver is a component of the OpenTelemetry Collector. It connects to a MySQL or MariaDB instance and supports metrics and logs pipelines.

MySQL and MariaDB support starts from these minimum collector versions:

  • Splunk Distribution of the OpenTelemetry Collector (splunk-otel-collector) v0.154.0 or later
  • Community (OSS) version of the OpenTelemetry Collector (opentelemetry-collector-contrib) v0.154.0 or later

Supported versions and platforms

Splunk Database Monitoring supports these MySQL and MariaDB versions and platforms:

  • MySQL versions: 5.7+
  • MariaDB versions: 10.5+
  • Platforms: AWS RDS, standalone
Note: On Amazon RDS for MySQL, Amazon Aurora MySQL, and Amazon RDS for MariaDB, you can authenticate with AWS Identity and Access Management (IAM) database authentication instead of a static password. This requires Splunk Distribution of the OpenTelemetry Collector (splunk-otel-collector) version 0.161.0 or higher, or Community (OSS) version of the OpenTelemetry Collector (opentelemetry-collector-contrib) version 0.161.0 or higher. See Authenticate to Amazon RDS or Aurora with AWS IAM. IAM database authentication isn't available for standalone MySQL or standalone MariaDB.

Prerequisites

  1. Enable performance schema for MySQL and MariaDB:

    AWS RDS

    1. On the AWS console, navigate to Aurora and RDS > Parameter groups > Edit parameter group. In your MySQL or MariaDB parameter group, change these values:

      • performance_schema: 1

      • max_digest_length: 4096

      • performance_schema_max_digest_length: 4096

      • performance_schema_max_sql_text_length: 4096

      • (Optional) require_secure_transport: 0 only if non-TLS connections are allowed

    2. Save the parameter group.

    3. Attach the parameter group to the database instance if it isn't already attached.

    4. Restart the database instance.

    5. (Optional) Verify:

      SQL 
      SHOW VARIABLES LIKE 'performance_schema';
SHOW VARIABLES LIKE 'max_digest_length';
SHOW VARIABLES LIKE 'performance_schema_max_digest_length';
SHOW VARIABLES LIKE 'performance_schema_max_sql_text_length';
    Note:

    • performance_schema = 1 is mandatory.

    • Splunk strongly recommends the 4096 values for some settings above to reduce query text truncation.

    • require_secure_transport is optional and depends on your TLS policy.

    • RDS parameter group values must be changed from the AWS Console, AWS CLI, Terraform, or API; not from a normal MySQL or MariaDB SQL session.

    • User creation and grants can still be done from any database client connected to the RDS instance.

    Note:

    These parameter group settings apply to both AWS RDS for MySQL and AWS RDS for MariaDB. MariaDB supports performance_schema_max_sql_text_length and performance_schema_max_digest_length from version 10.5.2 onward. All MariaDB versions available on AWS RDS support these settings.

    Standalone

    1. Update my.cnf or mysqld.cnf. MariaDB also uses my.cnf with the same [mysqld] section:

      SQL 
      [mysqld]
performance_schema=ON
max_digest_length=4096
performance_schema_max_digest_length=4096
performance_schema_max_sql_text_length=4096

      In this section, add require_secure_transport=OFF only if TLS isn't enabled.

    2. Restart the database instance.

    3. (Optional) Verify:

      SQL 
      SHOW VARIABLES LIKE 'performance_schema';
SHOW VARIABLES LIKE 'max_digest_length';
SHOW VARIABLES LIKE 'performance_schema_max_digest_length';
SHOW VARIABLES LIKE 'performance_schema_max_sql_text_length';
SHOW GRANTS FOR 'username'@'%';
    Note:

    • performance_schema=ON is mandatory.

    • The 4096 settings are strongly recommended.

    • require_secure_transport=OFF is optional. Keep it enabled if TLS is required and configure the receiver accordingly.

    Note:

    MariaDB uses the same configuration file format and [mysqld] section as MySQL. On MariaDB installations, the configuration file is typically located at /etc/mysql/my.cnf, /etc/my.cnf, or under /etc/mysql/mariadb.conf.d/. The same parameter names and values apply.

  2. Create a database user for the receiver to use.

    These commands apply to both AWS RDS and standalone platforms.

    Use these privileges to configure the receiver user:

    Privilege Required for Applies to
    REPLICATION CLIENT Replica status monitoring with SHOW SLAVE STATUS or SHOW REPLICA STATUS MySQL
    SLAVE MONITOR Replica status monitoring with SHOW SLAVE STATUS MariaDB 10.5.9+ (use instead of REPLICATION CLIENT)
    PROCESS InnoDB buffer pool metrics collection MySQL and MariaDB
    SELECT ON performance_schema.* Query sample and top query collection MySQL and MariaDB
    SELECT ON schema-name.* EXPLAIN plan collection for queries in that schema MySQL and MariaDB
    BACKUP_ADMIN Optional InnoDB redo-log metrics (mysql.innodb.redo_log.lsn.current, mysql.innodb.redo_log.lsn.checkpoint, and mysql.innodb.redo_log.checkpoint.age) on MySQL 8.0.11 through 8.0.29 only. Not required on MySQL 8.0.30 and higher. MySQL 8.0.11–8.0.29
  3. Run these commands:
    MySQL
    SQL 
    CREATE USER 'otel-user'@'%' IDENTIFIED BY 'otel-user-password';

-- Required for replica status monitoring
GRANT REPLICATION CLIENT ON *.* TO 'otel-user'@'%';

-- Required for InnoDB metrics collection
GRANT PROCESS ON *.* TO 'otel-user'@'%';

-- Required for performance-related metadata and query collection
GRANT SELECT ON performance_schema.* TO 'otel-user'@'%';

-- MySQL 8.0.11 through 8.0.29 only: required for the optional InnoDB
-- redo-log metrics. Skip this on MySQL 8.0.30 and higher.
GRANT BACKUP_ADMIN ON *.* TO 'otel-user'@'%';

FLUSH PRIVILEGES;
    Note:

    Grant BACKUP_ADMIN only if your MySQL version is between 8.0.11 and 8.0.29 and you plan to activate the optional InnoDB redo-log metrics. On those versions, the receiver reads redo-log data from performance_schema.log_status, which requires this privilege in addition to SELECT on performance_schema. MySQL 8.0.30 and higher read the same data from SHOW GLOBAL STATUS and don't need BACKUP_ADMIN. To check your version, run SELECT VERSION();.

    MariaDB
    SQL 
    CREATE USER 'otel-user'@'%' IDENTIFIED BY 'otel-user-password';

-- Required for replica status monitoring
GRANT SLAVE MONITOR ON *.* TO 'otel-user'@'%';

-- Required for InnoDB metrics collection
GRANT PROCESS ON *.* TO 'otel-user'@'%';

-- Required for performance-related metadata and query collection
GRANT SELECT ON performance_schema.* TO 'otel-user'@'%';

FLUSH PRIVILEGES;
    Note:

    Starting with MariaDB 10.5.2, the REPLICATION CLIENT privilege was renamed to BINLOG MONITOR and the privilege required for SHOW SLAVE STATUS changed. From MariaDB 10.5.9 onward, use SLAVE MONITOR, also known as REPLICA MONITOR, for replica status monitoring. If you upgrade from an older MariaDB version, existing REPLICATION CLIENT grants are automatically mapped to the new privilege names.

    Note:

    The optional InnoDB redo-log metrics aren't available on MariaDB, and no additional privilege is needed. BACKUP_ADMIN doesn't exist in MariaDB.

    Note:

    On AWS RDS, you can grant EXECUTE on mysql.rds_version to suppress collector log warnings.

    SQL 
    GRANT EXECUTE ON PROCEDURE mysql.rds_version TO 'otel-user'@'%';
    Note:

    Amazon RDS for MySQL doesn't allow the BACKUP_ADMIN privilege to be granted on MySQL 8.0.36 and higher minor versions, or on MySQL 8.4.3 and higher. Those versions don't require it because the receiver reads redo-log data from SHOW GLOBAL STATUS instead. RDS instances running MySQL 8.0.11 through 8.0.29 can still be granted BACKUP_ADMIN.

  4. For Amazon RDS for MySQL, Amazon Aurora MySQL, and Amazon RDS for MariaDB, create the monitoring user for AWS IAM database authentication instead of using a static password:

    MySQL
    SQL 
    CREATE USER 'otel-user'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';

GRANT REPLICATION CLIENT ON *.* TO 'otel-user'@'%';
GRANT PROCESS ON *.* TO 'otel-user'@'%';
GRANT SELECT ON performance_schema.* TO 'otel-user'@'%';

ALTER USER 'otel-user'@'%' REQUIRE SSL;
    MariaDB
    SQL 
    CREATE USER 'otel-user'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';

GRANT SLAVE MONITOR ON *.* TO 'otel-user'@'%';
GRANT PROCESS ON *.* TO 'otel-user'@'%';
GRANT SELECT ON performance_schema.* TO 'otel-user'@'%';

ALTER USER 'otel-user'@'%' REQUIRE SSL;

    Grant SELECT on all schemas or on specific schemas using the existing commands.

    Complete the AWS-side setup. For full instructions, see IAM database authentication for MariaDB, MySQL, and PostgreSQL in the Amazon RDS User Guide.

    1. Activate IAM database authentication on the RDS DB instance or Aurora DB cluster.
    2. Attach an IAM policy that grants rds-db:connect to the IAM role that runs the collector:
      JSON 
      {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["rds-db:connect"],
      "Resource": ["arn:aws:rds-db:us-east-1:111122223333:dbuser:db-ABCDEFGHIJKL01234/otel-user"]
    }
  ]
}

      In the resource ARN, use the DbiResourceId of the DB instance for Amazon RDS, or the DbClusterResourceId of the cluster for Amazon Aurora. The final path segment is the database user name and is case-sensitive.

    3. Make the IAM role available to the collector host through the standard AWS credential chain, such as an Amazon EC2 instance profile, an Amazon ECS task role, or IAM roles for service accounts (IRSA) on Amazon EKS. Don't put AWS access keys in the collector configuration file.

  5. Grant the user you created SELECT access to some or all schemas:

    Note: The MySQL receiver depends on schema-level SELECT privileges to retrieve EXPLAIN plans. If you don't grant access to a schema, queries from that schema may still appear in the UI, but their corresponding EXPLAIN plans will not be visible.
    Grant access to all schemas

    This option grants SELECT access across all schemas. It is the simplest configuration and ensures that EXPLAIN plans for top queries are consistently available in the UI without additional configuration.

    SQL 
    GRANT SELECT ON *.* TO 'otel-user'@'%';
    Grant access to specific schemas (least privilege)

    This option limits SELECT access to only the specified schemas. It is recommended for environments with strict access controls.

    SQL 
    GRANT SELECT ON `schema-name`.* TO 'otel-user'@'%';

Configure the receiver

Modify your collector configuration file as follows. All examples are for the Splunk Distribution of the OpenTelemetry Collector.

  1. In the receivers: section, add mysql:

    YAML 
    mysql:
  collection_interval: 10s
  endpoint: host:port
  username: otel-user
  password: otel-user-password
  events:
    db.server.query_sample:
      enabled: true
    db.server.top_query:
      enabled: true
  resource_attributes:
    mysql.instance.endpoint:
      enabled: true
    Note: On Amazon RDS for MySQL, Amazon Aurora MySQL, and Amazon RDS for MariaDB, you can replace password with AWS IAM database authentication. IAM authentication doesn't store a database password in the collector configuration and requires tls.insecure: false. See Authenticate to Amazon RDS or Aurora with AWS IAM.
    Important:

    If you're using the Splunk Distribution of OpenTelemetry Collector, leave the following receiver settings at their default values:

    • collection_interval (Default: 10s)

    • query_sample_collection.max_rows_per_query (Default: 100)

    • top_query_collection.collection_interval (Default: 60s)

    • top_query_collection.max_query_sample_count (Default: 1000)

    These values support Database Monitoring without affecting the performance of the database or the collector. If you increase these values you might adversely affect the performance of your database or collector, and this could result in ingest throttling.

  2. In the processors: section, add this:

    YAML 
    resource/mysql_service_instance_id:
  attributes:
    - action: insert
      from_attribute: mysql.instance.endpoint
      key: service.instance.id

  3. In the exporters: section, add otlp_http/dbmon:

    YAML 
    otlp_http/dbmon:
  headers:
    X-SF-Token: your-splunk-access-token
    X-splunk-instrumentation-library: dbmon
  logs_endpoint: https://ingest.your-splunk-realm.observability.splunkcloud.com/v3/event
  sending_queue:
    batch:
      flush_timeout: 15s
      max_size: 10485760
      sizer: bytes

  4. In the service.pipelines: section, create a metrics pipeline named metrics/dbmon and a logs pipeline named logs/dbmon:

    YAML 
    metrics/dbmon:
  receivers:
    - mysql
  processors:
    - memory_limiter
    - batch
    - resourcedetection
    - resource/mysql_service_instance_id
  exporters:
    - signalfx
logs/dbmon:
  receivers:
    - mysql
  processors:
    - memory_limiter
    - batch
    - resource/mysql_service_instance_id
  exporters:
    - otlp_http/dbmon

  5. Restart the collector to apply your configuration changes.

    The restart command varies depending on what platform you deployed the collector on and what tool you used to deploy it. Here are general examples of the restart command:

    Linux

    Linux with installer script:

    BASH 
    sudo systemctl restart splunk-otel-collector
    Windows

    Windows with installer script:

    BASH 
    stop-service splunk-otel-collector
start-service splunk-otel-collector
    Kubernetes

    Kubernetes with Helm:

    BASH 
    helm upgrade your-splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector -f your-override-values.yaml

    where splunk-otel-collector-chart is the name you gave to the Helm chart in the helm repo add command.

Your database instance should now be visible on APM > Database monitoring as well as on Infrastructure > Infrastructure monitoring if you have a Database Monitoring license. For troubleshooting, see Troubleshoot data collection .

Advanced configurations

Collect data from multiple MySQL or MariaDB instances

Omit the database name parameter, receivers.mysql.database. If omitted, the receiver will monitor all databases in the configured instance.

Authenticate to Amazon RDS or Aurora with AWS IAM

By default, the receiver authenticates with the password setting. On Amazon RDS for MySQL, Amazon Aurora MySQL, and Amazon RDS for MariaDB, you can use AWS IAM database authentication instead. With IAM authentication, the collector creates a short-lived authentication token for each database connection instead of storing a database password in the collector configuration.

Note: AWS IAM database authentication requires Splunk Distribution of the OpenTelemetry Collector (splunk-otel-collector) version 0.161.0 or higher, or Community (OSS) version of the OpenTelemetry Collector (opentelemetry-collector-contrib) version 0.161.0 or higher.

Before you configure the receiver, complete the AWS and database prerequisites. For AWS setup instructions, see IAM database authentication for MariaDB, MySQL, and PostgreSQL in the Amazon RDS User Guide. At minimum, activate IAM database authentication, create a database user with AWSAuthenticationPlugin, require SSL, and attach an IAM policy that grants rds-db:connect to the IAM role that runs the collector.

IAM authentication has two configuration parts:

  • The aws_iam_db_auth extension, which stores the AWS Region and creates tokens.
  • The db_auth receiver setting, which points to the extension by its component ID.

Configure IAM authentication in the collector:

  1. In the extensions: section, add aws_iam_db_auth and set region to the AWS Region of your database. The region setting is required.
  2. In the receivers: section, set db_auth to aws_iam_db_auth and remove the password setting. Keep username. The receiver passes the user name to the extension so the extension can create the token.
  3. Set tls.insecure to false. IAM database authentication requires an encrypted connection.
  4. In the service.extensions: section, add aws_iam_db_auth to the existing list of extensions.

For example:

YAML 
extensions:
  aws_iam_db_auth:
    region: us-east-1

receivers:
  mysql:
    collection_interval: 10s
    endpoint: your-service-endpoint:3306
    username: otel-user
    db_auth: aws_iam_db_auth
    events:
      db.server.query_sample:
        enabled: true
      db.server.top_query:
        enabled: true
    resource_attributes:
      mysql.instance.endpoint:
        enabled: true
    tls:
      insecure: false
      insecure_skip_verify: false
      ca_file: /etc/ssl/certs/rds-global-bundle.pem

service:
  extensions:
    - aws_iam_db_auth
  pipelines:
    metrics/dbmon:
      receivers:
        - mysql
      processors:
        - memory_limiter
        - batch
        - resourcedetection
        - resource/mysql_service_instance_id
      exporters:
        - signalfx
Important: db_auth and password are mutually exclusive. If you set both, the collector fails to start with invalid config: set either 'password' or 'db_auth', not both. Remove password when you use db_auth.
Note: The service.extensions list in the example is abbreviated. Add aws_iam_db_auth to the extensions that your configuration already declares. An extension that is configured but missing from service.extensions doesn't start, and the receiver fails with db_auth: requested credential provider is not present.

Configure TLS for IAM authentication. AWS requires an SSL or TLS connection because the MySQL driver sends the IAM token by using the MySQL cleartext password plugin.

tls setting TLS behavior IAM authentication support
insecure: true, or tls omitted No TLS Not supported
insecure: false and insecure_skip_verify: true TLS without server certificate verification Supported
insecure: false and insecure_skip_verify: false TLS with server certificate verification Supported and recommended

AWS recommends verifying the server certificate with the Amazon RDS certificate bundle. Set insecure_skip_verify: false and point ca_file at the certificate bundle. To download the bundle, see Using SSL/TLS to encrypt a connection to a DB instance or cluster in the Amazon RDS User Guide.

The sample in Configure the receiver omits tls. Omitting tls is equivalent to insecure: true and isn't compatible with IAM authentication.

To monitor databases in more than one AWS Region, declare one named instance of the extension for each Region and point each receiver to the extension instance it needs:

YAML 
extensions:
  aws_iam_db_auth:
    region: us-east-1
  aws_iam_db_auth/west:
    region: us-west-2

receivers:
  mysql/east:
    endpoint: east-db.example.com:3306
    username: otel-user
    db_auth: aws_iam_db_auth
    tls:
      insecure: false
  mysql/west:
    endpoint: west-db.example.com:3306
    username: otel-user
    db_auth: aws_iam_db_auth/west
    tls:
      insecure: false

service:
  extensions:
    - aws_iam_db_auth
    - aws_iam_db_auth/west

Databases in the same AWS Region can share one extension instance. To add another database in the same Region, add another receiver with the same db_auth value.

Tip:

Authentication token lifecycle

AWS IAM database authentication tokens have a 15-minute validity window. The collector requests a new token when it opens a database connection, so you don't need to restart the collector to renew tokens. IAM authenticates when a connection opens, not for each query, so established connections remain valid for their lifetime.

Use the following table to troubleshoot AWS IAM authentication errors:

Message in the collector log Cause and resolution
invalid config: set either 'password' or 'db_auth', not both Both credential settings are present. Remove password.
invalid config: 'db_auth' requires TLS; set 'tls.insecure' to false tls is omitted or tls.insecure is true. Set tls.insecure: false.
db_auth: requested credential provider is not present: "aws_iam_db_auth" The extension is configured but isn't listed in service.extensions. Add it.
db_auth: requested extension is not a credential provider db_auth names an extension that doesn't provide database credentials. Set it to an aws_iam_db_auth instance.
aws_iam_db_auth: region must be set on the extension Add the required region setting to the extension.
aws_iam_db_auth: mint RDS token for ... The collector can't resolve AWS credentials. Confirm that the instance profile, task role, or IAM role for service accounts (IRSA) is attached and reachable.
Access denied for user or an AWSAuthenticationPlugin or mysql_clear_password plugin error The AWS or database-user setup is incomplete. Confirm that IAM database authentication is active, the database user uses AWSAuthenticationPlugin, the IAM policy allows rds-db:connect for the exact user name, and the extension Region matches the database Region.
An SSL or TLS connection error, or an error that says insecure transport isn't allowed TLS isn't established. Set tls.insecure to false and, if the server requires a verified CA, set ca_file to the Amazon RDS certificate bundle.
Enable optional metrics

Set metrics.metric-name.enabled to true. For example, for the optional metrics mysql.commands, mysql.connection.count, and mysql.connection.errors:

YAML 
mysql/extra_metrics:
  endpoint: host:port
  username: otel-user
  password: ${env:MYSQL_PASSWORD}
  database: mysql-database-name
  collection_interval: 10s
  mysql.commands:
    enabled: true
  mysql.connection.count:
    enabled: true
  mysql.connection.errors:
    enabled: true
TLS

Set these parameters at a minimum:

YAML 
mysql/default_tls:
  endpoint: host:port
  username: otel-user
  password: ${env:MYSQL_PASSWORD}
  database: mysql-database-name
  collection_interval: 10s
  tls:
    server_name_override: localhost

Set up APM correlation

See Correlate database queries with Splunk APM traces.

Settings reference

Configuration options for this receiver:

included

https://raw.githubusercontent.com/splunk/collector-config-tools/main/cfg-metadata/receiver/mysql.yaml

Metrics reference

Metrics, attributes, and resource attributes reported by this receiver:

included

https://raw.githubusercontent.com/splunk/collector-config-tools/main/metric-metadata/mysqlreceiver.yaml