Configuring a Cluster Diagnostic Instance

The cluster diagnostic instance is an optional, but strongly recommended, aid to monitoring the health and performance of the Seq cluster. Seq nodes in the cluster send OpenTelemetry logs, metrics, and traces to the diagnostic instance for:

• Query tracing, including full visualizations of how a Seq query handled by the cluster was distributed over the participating nodes,
• Error monitoring and problem troubleshooting, and
• Visualization of key performance and health metrics.

These instructions show how to configure an additional Seq server for use as the diagnostic instance, however, any standard OTLP receiver can be used. Contact the Seq support team for advice if you are configuring an alternative monitoring tool for use as a cluster diagnostic instance.

Provisioning the Diagnostic Instance

First, provision an additional Seq instance using the instructions in Getting Started on Windows or Getting Started with Docker.

👍

Cluster Worksheet Update

Assuming your diagnostic instance listens on port 80, the Diagnostic Instance Ingestion URI will be an HTTP URL pointing at the diagnostic instance, with /ingest/otlp appended. For example, https://diagnostics.seq.local/ingest/otlp. Don't forget to add that path to the URL.

The diagnostic instance must be accessible to the cluster nodes at the Diagnostic Instance Ingestion URI.

Enabling OTLP Metrics

The diagnostic instance uses a feature flag to enable ingestion of OTLP metric samples as wide events. This approach is not efficient for regular production metrics duties, and works in this context only because the cluster sends a modest number of metrics, using a limited subset of OTLP features, at a wide sampling interval.

seq config set -k features.enabled -v IngestOtlpMetrics
seq service restart

Retention Policy Configuration

The diagnostic instance does not need a long retention window. Setting a limited retention window will reduce the likelihood of disk space exhaustion and similar issues.

Because only low-volume information is sent from the cluster, a good starting point is to set a 30 day "all events" retention policy on the diagnostic instance.

In Data > Storage, choose Add Retention Policy:

API Key Creation

Cluster nodes should authenticate with the diagnostic instance when sending data.

First, in Settings > API Keys, choose "Require an API key for HTTP/S ingestion":

Then, choose Add API Key and create a new key with Ingestion permissions:

👍

Cluster Worksheet Update

Add the generated Diagnostic Instance API Key token to the cluster worksheet.

Diagnostic Instance Templates

A set of pre-built signals and dashboards are available for the cluster diagnostic instance. Seq Cluster Monitoring and Diagnostics for further information.