highHA & Clustering

Troubleshooting an HA firmware upgrade stuck in the SENT-IMAGE state

Confirmed 8/8/2026

Problem

During a firmware upgrade of a FortiGate High Availability (HA) cluster, the Secondary unit may successfully install the target firmware while the original Primary never starts or completes its own upgrade. The members can subsequently re-form the cluster while running different firmware versions, causing them to remain out of synchronization.

Symptoms

The Secondary FortiGate is running the new firmware, but the original Primary remains on the previous version. The cluster may form but stays out-of-sync because of the firmware mismatch. The hatalk daemon on the Primary repeatedly reports `uprade_state=3(SENT-IMAGE)` when debugging is enabled: ``` diagnose debug application hatalk -1 diagnose debug enable ``` Example output: ``` <hatalk> entering hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000 ```

Environment

FortiGate HA clusters undergoing a firmware upgrade. In the expected upgrade sequence, the current Primary transfers the target firmware image to the Secondary. The Secondary installs the image and reboots, then assumes the Primary role so that the original Primary can upgrade.

Root Cause

The usual cause is an excessively long HA peer-failure detection time configured under `config system ha`. The relevant settings are: ``` config system ha set hb-interval <1-20, default = 2> set hb-interval-in-milliseconds [100ms** | 10ms] set hb-lost-threshold <1-60, default = 6> end ``` `hb-interval` controls the interval between heartbeats, `hb-interval-in-milliseconds` supplies the interval unit, and `hb-lost-threshold` specifies how many missed heartbeats indicate failure. For example: ``` config system ha set hb-interval 20 set hb-interval-in-milliseconds 100ms set hb-lost-threshold 60 end ``` This produces one heartbeat every 2 seconds (`20 * 100ms`) and requires 60 missed heartbeats before declaring the peer down, resulting in a 120-second detection period. Depending on the FortiGate hardware, the Secondary may finish rebooting in less than 120 seconds. The Primary therefore never detects that the Secondary went offline to install the image and remains in `SENT-IMAGE` instead of continuing its own upgrade.

Solution

Before proceeding, review the standard FortiGate HA firmware upgrade process: https://community.fortinet.com/t5/FortiGate/Technical-Tip-FortiGate-HA-upgrade-procedure-and-the-status/ta-p/246692.

Preferred recovery procedure

Temporarily prevent heartbeat communication between the Primary and Secondary units. Choose an appropriate method:
- If physical access is available, isolate the Primary from the network by disconnecting data links first and heartbeat links afterward. This allows the already-upgraded Secondary to carry network traffic.
- Alternatively, shut down the Secondary temporarily so it cannot communicate with the Primary or create a split-brain condition.
Reboot the Primary. This clears the current hatalk daemon state and allows the Primary to recognize itself as the only member of its HA cluster.
Upgrade the Primary to the same firmware version already installed on the Secondary.
After the Primary finishes upgrading, power on or reconnect the Secondary. Confirm that the HA cluster forms again and that both members eventually synchronize.
After synchronization, reduce the heartbeat and peer-detection timings under config system ha. The following example sends a heartbeat every 0.5 seconds (5 * 100ms) and declares the peer down after 20 missed heartbeats, or 10 seconds:

config system ha
    set hb-interval 5
    set hb-interval-in-milliseconds 100ms
    set hb-lost-threshold 20
end

Alternative when the Secondary cannot be disconnected or shut down

Apply the shorter heartbeat settings before retrying the upgrade:

config system ha
    set hb-interval 5
    set hb-interval-in-milliseconds 100ms
    set hb-lost-threshold 20
end

Reboot the Primary to clear the current hatalk process state.
Start the firmware upgrade on the Primary. It will attempt to transfer the firmware image to the Secondary. The Secondary already runs that version, which is acceptable. The Primary will use the revised heartbeat thresholds to detect the Secondary reboot.
Provided that the Secondary reboots, the Primary should detect the outage and trigger its own installation of the new firmware.
After the process completes, confirm that both members run the same firmware and synchronize their configurations. Also verify that the new heartbeat threshold settings remain configured on both units.

Verification

Verify all of the following after recovery:

Both HA members are running the same firmware version.
The HA cluster has re-formed.
Both units eventually report an in-sync state.
The Primary no longer remains in the repeating SENT-IMAGE hatalk state.
The revised heartbeat interval and lost-heartbeat threshold are present on both units.

Community rating

— / 5 (0)

Discussion (0)

No comments yet.