Configuring a Clustered Seq Node

The nodes in a Seq cluster each store a replica of the ingested log stream and participate in distributed queries.

These steps should be repeated for each node in the cluster, except where noted.

📘

To convert an existing Seq instance into a cluster node, see Converting a Standalone Seq Instance into a Cluster.

To add a node to a Seq cluster that is already receiving ingestion traffic from applications, see Adding, Updating, and Removing Nodes.

Install the Seq Binaries

http://node01.seq.local,http://node01.seq.local:5341

Generate a Secret Key

If this is the first node to be deployed, generate a secret key. This can be performed on any workstation with Seq available:

seq show-key --generate

The secret key is a Base64-encoded, cryptographically-secure random byte string. Every node in a Seq cluster must use the same secret key.

👍

Cluster Worksheet Update

Add the Secret Key to the cluster worksheet for use when configuring this and additional nodes.

Inbound Ports

The following Seq ports should be open on the Seq node. As in Configuring a Cluster Load Balancer , these instructions assume TLS will be terminated at the load balancer. If TLS will be terminated at the Seq nodes instead, adjust accordingly.

Configure the Node

The goal of this step is to completely configure the new cluster node before starting it. You will need the values previously recorded in the Cluster Worksheet.

seq service stop
seq config set -k storage.secretKey -v "{Secret Key}"
seq config set -k api.ingestionPorts -v 5341
seq config set -k api.canonicalUri -v "{Load Balancer URI}"
seq config set -k cluster.clusterListenUri -v "{Node Cluster URI}"
seq config set -k cluster.internalApiUri -v "{Node Internal API URI}"
seq config set -k cluster.isEnabled -v True
seq config set -k diagnostics.otlp.endpoint -v "{Diagnostic Instance Ingestion URI}"
seq config set -k diagnostics.otlp.protocol -v "http/protobuf"
seq config set -k diagnostics.otlp.metricsSamplingIntervalSeconds -v 10
seq config set -k firstRun.adminUsername -v "{Admin Username}"
seq config set -k firstRun.adminPassword -v "{Initial Admin Password}"
seq config set -k firstRun.requireAuthenticationForHttpIngestion -v True

seq secret set -k diagnostics.otlp.headers -v "X-Seq-ApiKey={Diagnostic Instance API Key}"
seq secret set -k metastore.msSql.connectionString -v "{Cluster State Database Connection String}"

seq secret set -k metastore.postgres.connectionString -v "{Cluster State Database Connection String}"

seq service start

Troubleshooting

If something isn't right, here are some places to look for clues. The steps above can be repeated if settings/variables need to be adjusted.

The node won't start...

• Open up the cluster diagnostic instance. If the node got far enough to record some diagnostics, the Errors and Warnings signals on the diagnostic instance will have some details.
• On Windows, check the latest logs in the C:\ProgramData\Seq\Logs directory on the node. Note that this exact path is used, even if Seq's storage location is elsewhere. If this is unsuccessful, stop the service using seq service stop, if necessary, then run it interactively from the command-line using seq run.
• Under Docker, check the Seq container's STDOUT, and the latest entries in the Logs/ directory under the mounted storage volume.

The node starts, but can't be reached by the load balancer...

• Follow the instructions in The node won't start... to check the diagnostic instance and container logs to look for related errors.
• Check that the Node Listen URI is reachable from the network that the load balancer is running in.
• If there's no response when attempting to GET the Node Listen URI, check that the appropriate ports are open in the Seq node's firewall.
• If the node can be reached at the expected URI, retrieve the /health endpoint and check that the returned status code is 200.
• Under Docker, make sure that whatever port is used for the Node Listen URI maps to port 80 (or 443 for Seq-terminated TLS) in the container.

The node starts, but doesn't appear in the Data > Cluster screen...

• Look in the cluster diagnostic instance. This problem is generally due to the cluster network failing to connect nodes.
• Are all nodes using the same storage.secretKey/SEQ_STORAGE_SECRETKEY value?
• Did you accidentally include the braces {...} in your secret key value?
• Check that the Node Cluster URI port is open on all nodes (port 5344 in the instructions here).
• Was there already data in the Stream/ folders of one or more nodes when joining the cluster? A node must have a completely empty Stream/ folder in order to connect and synchronize with the cluster.

The node starts, but applications can't send logs or traces...

• The default instructions here set the "Require an API key for HTTP/S ingestion" option. Create API Keys and use them to configure your apps, or deselect this option in the Settings > API keys page.
• Are you looking in the diagnostic instance, instead of the cluster?

The node starts, but authentication in fails...

• On Windows, the username/password entered into the post-install setup dialog (Seq Service Administration) will be overridden by the firstRun.* settings set when configuring the first node.
• The firstRun.* settings or SEQ_FIRSTRUN_* variables are only applied when the first node to start initializes the cluster state database. These are subsequently ignored: it's not possible to change or reset authentication details after the first Seq node starts.
• Did you accidentally include the braces {...} when configuring your admin password?

The node starts, but displays a "permission denied" message...

• On Windows, make sure that the Node Listen URI is not configured as an ingestion port (api.ingestionPorts). These ports allow ingestion only, and return HTTP 403 otherwise.
• Under Docker, make sure that the Node Listen URI is mapped to either port 80 or 443 on the datalust/seq container. The other ports exposed by the container don't allow API requests.

The node starts, but requests randomly fail with status code 503...

• Check that the Node Internal API URI is accessible by other nodes in the cluster.

🫠

Stuck?

Contact [email protected] and let us know.