Troubleshooting unexpected FortiGate HA failover and cluster role changes

1. Confirm that HA override is disabled before interpreting the election results. This procedure is based on the default election behavior with override disabled.

2. In a VDOM environment, enter the global configuration context:

config system global

3. Review the current cluster state:

get system ha status

Check the warnings near the top of the output, the last four HA historical events and their timestamps, whether the cluster is in sync, and the time of the last synchronization. Inspect heartbeat-interface counters for errors or interfaces changing to down. Near the bottom, verify the Primary and Secondary roles against each hostname.

4. Read the persistent election history on each cluster member:

diagnose sys ha history read

The history is limited to 512 entries and persists across reboots. Each unit maintains its own event history. It can be cleared manually, and older entries are overwritten as the history fills. Look for monitored-interface link status changes, where 0 means down and 1 means up. These changes can invoke the election algorithm. The same events may also appear in system event logs.

5. If a monitored interface is a LAG, remember that it is treated as down only when all LAG members are down. If appropriate for the design, configure the LAG minimum-link threshold:

config system interface
    edit LAG_Interface_Name
        set min-links 1
end

min-links 1 is the default. It keeps the LAG up while at least one LACP member remains up.

6. Examine link monitoring, interface state, role, and cluster age:

diagnose sys ha dump-by group

Interpret the output as follows:

• linkfails=35 represents the total number of down interfaces for that serial number.
• 2(work) identifies Primary state 2; Secondary state is 3.
• mondev summarizes each monitored device, where 1 means up and 0 means down.
• uptime cnt=35, reset 3 shows cluster uptime and the number of resets.
• Cluster uptime resets whenever a mondev status changes.

The link_failure count can also be reviewed with:

diagnose sys ha dump-by vcluster

7. Account for boot-time failure scoring. When a unit boots with monitored interfaces down, link_failure increases by 50 for each down interface. This value cannot be configured. For example, three down monitored interfaces produce link_failure=150. If both members boot simultaneously, the unit with the lowest link_failure count is preferred as Primary.

8. Consider the uptime grace period. Although the cluster can initially prefer the member with fewer failed monitored interfaces, age is considered only after ha-uptime-diff-margin. The default is 300 seconds, or 5 minutes. Age and link_failure can trigger transitions only after the cluster has been running longer than this margin.

9. If the monitored-interface list was changed while the cluster was operating, recalculate expectations for link_failure. The count is reset to reflect the current monitored-interface states. For example, if three down interfaces produced link_failure=150 and two are removed from monitoring, the result becomes link_failure=50 because one down interface remains monitored.

10. Inspect interface state and virtual MAC information on both members:

diagnose sys ha mac

In this output, a down interface is shown as 1 and an up interface as 0. In an active-passive cluster, a subordinate unit does not have vMAC information until it becomes Primary.

11. If testing requires an intentional uptime reset, run:

diagnose sys ha reset-uptime

This action can cause a cluster transition. The HA history records entries similar to:

<timestamp> FG800D3916801158 is elected as the cluster primary of 2 member
<timestamp> user="admin" ui=ssh(10.10.10.1) msg="Reset HA uptime"

Afterward, the reset unit shows uptime zero, and diagnose sys ha dump-by group or diagnose sys ha dump-by vcluster shows an incremented reset_cnt, for example:

'FG800D3916-----7': ha_prio/o=1/1, link_failure=50, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/4
'FG800D3916801158': ha_prio/o=0/0, link_failure=50, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=349084/1

12. To clear error status associated with cluster members that were removed or re-added, reset HA health status:

diagnose sys ha reset-health-status

13. Physically verify that the heartbeat cable is connected and that the heartbeat ports match between members.

14. Correlate HA history with system event logs. If the WAN is the only HA-monitored interface, check for WAN-flapping events. If redundant WAN links are unavailable, internet connectivity can be interrupted during the failover impact window.

15. After any HA or connected-network change, rerun the applicable test booklet to confirm that the intended topology and failover behavior still operate as designed.

Run get system ha status and confirm there are no unexpected warnings, the cluster is in sync, heartbeat interfaces are healthy, and the hostnames have the intended Primary and Secondary roles. Run diagnose sys ha history read to confirm that no new unexplained elections or monitored-link changes occur. Use diagnose sys ha dump-by group or diagnose sys ha dump-by vcluster to verify expected mondev, link_failure, uptime, and reset_cnt values. Use diagnose sys ha mac to confirm the expected interface states. Finally, rerun the test booklet after changes.