FortiExpert logoFortiExpert
Sign in
← Back to knowledge base
highSD-WAN

Troubleshooting SD-WAN health checks marked down despite receiving probe replies

Confirmed 8/8/2026

Problem

An SD-WAN health check can report a link as down when probe replies arrive with high latency or when probe sequence-number handling causes otherwise valid replies to be rejected. This may occur even when packet captures show successful probe transmission and reception with no packet loss.

Symptoms

- The SD-WAN health check intermittently reports a down or dead state. - Latency is unusually high, for example approximately 500 ms or higher. - Probe packet loss is not observed. - The fail count increases unexpectedly. - Debug output shows delayed replies or messages such as `can not find probe for monitor`. - Policy routes or internet service routes associated with the monitor may be removed. Example debug output: ```text 2024-11-21 10:35:51 lnkmtd::monitor_proto_peer_send_request(638): ---> HUB-13-VIRTUAL_WAN_LINK-13(10.117.141.1:ping) send probe packet, fail count(4) 2024-11-21 10:35:51 lnkmtd::ping_send_msg(435): ---> ping 10.117.141.1 seq_no=55331, icmp id=5348, send 20 bytes 2024-11-21 10:35:51 lnkmtd::ping_do_addr_up(136): ---> HUB-13-VIRTUAL_WAN_LINK-13->10.117.141.1(10.117.141.1), rcvd 2024-11-21 10:35:51 lnkmtd::monitor_peer_recv(2152): ---> can not find probe for monitor HUB-13-VIRTUAL_WAN_LINK-13, seq_num 686. 2024-11-21 10:35:52 lnkmtd::monitor_ppeer_fail(1847): ---> HUB-13-VIRTUAL_WAN_LINK-13(10.117.141.1 ping) is dead. 2024-11-21 10:35:52 lnkmtd::monitor_proute_cmdb_set(1121): ---> policy routes or internet service routes related to the monitor(HUB-13-VIRTUAL_WAN_LINK-13) may be removed ``` In this example, replies are received with approximately 519 ms of latency.

Environment

FortiGate with an SD-WAN health check using ICMP probes, particularly on links where latency can approach or exceed the configured probe timeout.

Root Cause

SD-WAN health checks validate probe replies by sequence number and by whether the reply arrives within the configured `probe-timeout`. The link can be marked down when latency exceeds this timeout, network delay is outside acceptable thresholds, or the probe sequence number grows beyond 32768. The ping sequence is a 16-bit integer. When it increments from 0 to 32768, it is interpreted as a negative number. The sequence can then be treated as outdated, causing the system to consider a valid packet lost. Likewise, a reply received outside the configured timeout window is treated as invalid even though it reached the FortiGate.

Solution

  1. Review the SD-WAN health-check state.

    Run the applicable health-check status diagnostics described in the SD-WAN-related diagnose commands. Examine:

    • Packet loss.
    • Latency.
    • Jitter.
    • Health status.

  2. Confirm that ICMP probes are transmitted and returned.

    Capture traffic on the SD-WAN member interface. Replace <interface_name> with the interface and x.x.x.x with the probe target:

    diagnose sniffer packet <interface_name> "host x.x.x.x and icmp" 4 0 l

    Verify that:

    • An ICMP echo request leaves the FortiGate.
    • An ICMP echo reply returns.
    • No packet drops are visible.

  3. Inspect sequence numbers and probe timing.

    Enable SD-WAN and link-monitor debugging:

    diagnose debug reset
diagnose debug console timestamp enable
diagnose debug application sdwan -1
diagnose debug application link-monitor -1
diagnose debug enable

    Review probe sequence numbers and response timing. A health check may fail if sequence numbers exceed 32768 or if replies arrive after the timeout.

    Disable and reset debugging after collecting the required output:

    diagnose debug disable
diagnose debug reset

  4. Increase the probe timeout when link latency is high.

    Configure a timeout appropriate for the observed latency. The default value is 500. Replace health_check_name and XYZ with the relevant health-check name and timeout value:

    config system sdwan
    config health-check
        edit "health_check_name"
            set probe-timeout XYZ
        next
    end
end

    Increase the probe timeout and failtime as required for the high-latency environment. Increasing probe-timeout prevents valid but delayed replies from being rejected solely because they arrive outside the previous timeout window.

Verification

Run:

diagnose sys sdwan health-check

Confirm the following expected results:

  • Health-check status is alive.
  • Fail count is 0 and does not continue increasing.
  • Latency remains visible but is within the configured timeout.
  • Probe replies are received.
  • The link remains operational despite the elevated latency.

Tags

No tags yet.

Community rating

— / 5 (0)

Discussion (0)

    No comments yet.