Collecting and troubleshooting FortiGate HA synchronization data for TAC

1. On the primary FortiGate, collect general system and HA information:

get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by vcluster
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster
diagnose sys ha checksum show global
diagnose sys ha checksum show root
show full-configuration sys ha
diagnose debug crashlog read

2. Recalculate the configuration hashes on the primary unit. Run the VDOM command separately for every VDOM, replacing <VDOM_name> with its name:

diagnose sys ha checksum recalculate global
diagnose sys ha checksum recalculate root
diagnose sys ha checksum recalculate <VDOM_name>

3. Record the recalculated hashes from the primary unit. Again, run the VDOM command for each VDOM:

diagnose sys ha checksum show global
diagnose sys ha checksum show root
diagnose sys ha checksum show <VDOM_name>

4. Open a CLI session to the secondary member by replacing <ID> and <User_Name> with the applicable values:

exec ha manage <ID> <User_Name>

Example:

exec ha manage 0 admin

5. From the secondary FortiGate, collect its general system and HA diagnostics:

get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster
diagnose debug crashlog read

6. Recalculate all relevant hashes on the secondary unit:

diagnose sys ha checksum recalculate global
diagnose sys ha checksum recalculate root
diagnose sys ha checksum recalculate <VDOM_name>

7. Record the secondary unit's hashes:

diagnose sys ha checksum show global
diagnose sys ha checksum show root
diagnose sys ha checksum show <VDOM_name>

8. Compare the output gathered in steps 3 and 7 with a text-comparison utility. For example, install the Compare plug-in in Notepad++, place the primary and secondary output in separate files, and run a comparison. Record whether each mismatch belongs to the global configuration, root VDOM, or another VDOM.

Example global mismatch:

• Primary system.admin: 951a796c3d5cce457563476f9aa9b98e
• Secondary system.admin: 91fb884502aa610692bac371d6490456

Example root VDOM mismatch:

• Primary firewall.policy: a7f03dc8d82aca5281510c989723e120
• Secondary firewall.policy: 50318191a743e5d5884c57f407e6b66e

9. Identify the exact configuration difference on each FortiGate. Set <value_1> to global, root, or a VDOM name. Set <value_2> to the mismatched table or string, such as system.admin:

diagnose sys ha checksum show <value_1> <value_2>

From FortiOS 7.0 onward, the exact difference can also be viewed in the GUI by hovering over the member that is not synchronized. At this point, open a Fortinet TAC ticket and attach all collected output and identified differences.

10. If continuing the investigation locally, correct the differing configuration through the CLI on both FortiGate members. In the source example, the differences involved administrator john.connor and Firewall Policy ID 66. A special character can sometimes cause a mismatch.

11. If the mismatch concerns data that cannot be edited manually through the CLI—such as UUIDs, replacement messages, default objects, or ISDB objects—or if there are too many differences, rebuild the HA cluster. The first method is to rebuild it using only the HA parameters:

• Ensure physical access to the HA cluster.
• Have a serial console cable available.
• Identify the physical HA port.
• Connect to the secondary FortiGate's serial console port.
• Run and confirm the factory reset:

execute factory reset

• Access the secondary FortiGate GUI with the default credentials: user admin, password blank.
• Configure the HA parameters. If override is enabled, make certain that the secondary priority is lower than the primary priority; otherwise, all configurations could be erased.
• Wait approximately 5–15 minutes for synchronization. The secondary unit's serial console indicates when synchronization has completed.

12. The second rebuild method is to reconstruct the HA cluster from a primary-unit backup. The detailed procedure is in a separate KB article and is not included in the source material.

13. If troubleshooting a FortiGate-120G running FortiOS v7.2.9 or v7.2.10, run the following HA debug commands and check for the known behavior associated with bug 1056138:

diagnose debug application hasync -1
diagnose debug application hatalk -1

Also collect:

get system ha status
diagnose sys ha status

Indicators include a secondary checksum consisting entirely of zeros:

chksum dump: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The cluster checksum entry for the secondary may also be empty after its serial number:

================== FG120GXXXXXXXXXXXXXX==================

The HA synchronization debug can repeatedly show:

hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)

14. For bug 1056138, use one of these workarounds:

15. Use a port other than the dedicated HA port.

16. Roll back to the previous FortiOS version.

The issue is scheduled to be fixed in FortiOS v7.2.11 and v7.6.1.

Recalculate and display the global, root, and per-VDOM hashes on both members. Confirm that the corresponding primary and secondary hash values match and that the cluster no longer reports the member as unsynchronized. After a cluster rebuild, allow 5–15 minutes for synchronization and confirm completion from the secondary unit's serial console.

For FortiGate-120G systems affected by bug 1056138 on FortiOS v7.2.9 or v7.2.10, roll back to the previous version as one of the documented workarounds.