Migrating Legacy DR Clusters to HA

Seq versions up to and including 2024.3 include a two-node clustering capability focused on disaster recovery (DR). This clustering technology is no longer supported by new Seq versions, which replace it with more capable HA/scale-out clustering.

To migrate a Seq DR cluster to HA clustering, manual coordination is required. This document outlines the steps required for migration; for assistance, please reach out to the Seq support team.

Preparing to Migrate

Before commencing migration, both nodes in the Seq DR cluster must be updated to the latest Seq 2024.3 patch release. Once this has been verified, check that both nodes are healthy in the Settings > Cluster screen:

If both nodes are online and in sync, take note of which is leader and which is follower before proceeding. The migration sequence depends on the identities of the leader and follower nodes, and data loss or extended downtime may result if the nodes are not migrated in the correct order.

🚧

If only one node appears, or if the synchronization delay exceeds 5 seconds, do not proceed.

Shutting Down the DR Cluster

The migration process begins with stopping the Seq service on each node (Windows) or stopping the containers running for each node (Docker/Kubernetes).

On Windows:

seq service stop

On Docker, use the commands appropriate to your container host/runtime.

Both nodes must be fully and completely stopped before proceeding.

Upgrading the Follower

The follower node is upgraded first; this provides a recovery path if the upgrade fails: the leader can be re-started and availability restored if the follower upgrade fails.

Upgrading the Follower on Windows

On Windows, run the MSI corresponding to the new Seq version. When the install wizard completes, de-select the box "Install or start the Seq service".

📘

If the box is not unchecked, ignore any errors displayed in the restart dialog, and ensure the service is stopped using seq service stop before continuing.

Next, at an administrative PowerShell prompt, run:

seq node migrate

If the command succeeds, proceed to start the Seq node:

seq service start

Once the service starts, the node will become leader of the new HA cluster. Ensure the node can be reached through the load balancer, bringing it back into service manually if necessary.

In the new Seq UI, go to Data > Cluster and verify that the node is shown.

🚧

If the node does not restart and successfully serving traffic, or appears not to be in a clustered configuration, go to Recovering if the Follower Upgrade Fails below.

Upgrading the Follower under Docker

Under Docker, the follower node must be migrated using an Init Script.

The init script contents must be:

seqsvr node migrate

Note that this will cause the node's cluster configuration to be written locally to its Seq.json file, and some existing environment variables may be ignored. To add additional cluster nodes, ensure the Adding, Updating, and Removing Nodes instructions are used.

Once the container has been restarted and the init script is run, the node will become leader of the new HA cluster. Ensure the node can be reached through the load balancer, bringing it back into service manually if necessary.

In the new Seq UI, go to Data > Cluster and verify that the node is shown.

🚧

If the node does not restart and successfully serving traffic, or appears not to be in a clustered configuration, go to Recovering if the Follower Upgrade Fails below.

Recovering if the Follower Upgrade Fails

If the follower node cannot be successfully upgraded:

1. Do not attempt to start the follower; ensure it is shut down, using seq service stop or the appropriate Docker commands.
2. Start the old leader node. If the steps above have been followed correctly, no modifications will have been made to it, and it should start normally in the old DR configuration.
3. Gather error messages/observations and contact [email protected] before proceeding any further.

Upgrading the Leader

Once the follower has been upgraded and starts serving traffic as the new HA cluster leader, the upgrade process is essentially irreversible, and the old leader must be successfully updated before being brought back online.

To upgrade the leader, use the same commands as for the follower:

• seq node migrate and seq service start on Windows, or
• seqsvr node migrate in an init script under Docker.

The old leader, once upgraded, should appear in the Data > Cluster page and begin synchronizing with the new cluster.

Once the old leader node displays a Data age value under 5 seconds, the upgrade has completed successfully.

If this does not occur within a few minutes, contact [email protected] for advice.

Finalizing Configuration

Various settings have been added to Seq HA clusters that improve manageability. These should now be configured:

• In Settings > General, set the Target replica count field to 2.
• Also in Settings > General, select an app instance in the Notification app instance field to which system notifications should be sent. It's a good idea for this to be via a robust system on separate infrastructure, for example, a shared Slack or Teams channel.
• Follow the instructions in Cluster Monitoring and Diagnostics to configure a second Seq instance that receives diagnostic information from the running cluster.

Finally, review Clustering Recommendations and Best Practices.