keepalived 的一些排查方法

发表于 Disqus：

汇总一些用过的 keepalived 问题的排查方法

  1. 统计信息
1
kill -USR2 $(cat /var/run/keepalived.pid)

向 keepalived 进程发 signal，其它可用的参考man keepalived

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
HUP or SIGFUNC=RELOAD
       This causes keepalived to close down all interfaces, reload its configuration, and start up with the new
       configuration.

TERM, INT or SIGFUNC=STOP
       keepalived will shut down.

USR1 or SIGFUNC=DATA
       Write configuration data to /tmp/keepalived.data

USR2 or SIGFUNC=STATS
       Write statistics info to /tmp/keepalived.stats

SIGFUNC=JSON
       Write configuration data in JSON format to /tmp/keepalived.json

默认会生成/tmp/keepalived.stats，包含各 Instance 的统计信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
VRRP Instance: VI_0
  Advertisements:
    Received: 7233644
    Sent: 8
  Became master: 2
  Released master: 2
  Packet Errors:
    Length: 0
    TTL: 0
    Invalid Type: 0
    Advertisement Interval: 0
    Address List: 0
  Authentication Errors:
    Invalid Type: 0
    Type Mismatch: 0
    Failure: 0
  Priority Zero:
    Received: 2
    Sent: 0
  1. 当前状态
1
snmpwalk -v2c -cpublic 127.0.0.1 -mALL .1.3.6.1.4.1.9586

利用 SNMP 查询，结果包含当前状态和一些配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
KEEPALIVED-MIB::version.0 = STRING: Keepalived v2.0.20 (01/22,2020)
KEEPALIVED-MIB::routerId.0 = STRING: ***
KEEPALIVED-MIB::smtpServerAddressType.0 = INTEGER: ipv4(1)
KEEPALIVED-MIB::smtpServerAddress.0 = Hex-STRING: 00 00 00 00
KEEPALIVED-MIB::smtpServerTimeout.0 = Gauge32: 0 seconds
KEEPALIVED-MIB::smtpServerPort.0 = Gauge32: 0
KEEPALIVED-MIB::smtpEmailFaults.0 = INTEGER: enabled(1)
KEEPALIVED-MIB::smtpEmailSmtpAlert.0 = INTEGER: unset(3)
KEEPALIVED-MIB::smtpEmailSmtpAlertVrrp.0 = INTEGER: unset(3)
KEEPALIVED-MIB::smtpEmailSmtpAlertChecker.0 = INTEGER: unset(3)
KEEPALIVED-MIB::trapEnable.0 = INTEGER: disabled(2)
KEEPALIVED-MIB::linkBeat.0 = INTEGER: netlink(1)
KEEPALIVED-MIB::lvsFlush.0 = INTEGER: disabled(2)
KEEPALIVED-MIB::netNamespace.0 = STRING:
KEEPALIVED-MIB::dynamicInterfaces.0 = INTEGER: false(2)
KEEPALIVED-MIB::lvsFlushOnStop.0 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceName.1 = STRING: VI_0
KEEPALIVED-MIB::vrrpInstanceVirtualRouterId.1 = Gauge32: 29
KEEPALIVED-MIB::vrrpInstanceState.1 = INTEGER: backup(1)
KEEPALIVED-MIB::vrrpInstanceInitialState.1 = INTEGER: backup(1)
KEEPALIVED-MIB::vrrpInstanceWantedState.1 = INTEGER: backup(1)
KEEPALIVED-MIB::vrrpInstanceBasePriority.1 = INTEGER: 100
KEEPALIVED-MIB::vrrpInstanceEffectivePriority.1 = INTEGER: 100
KEEPALIVED-MIB::vrrpInstanceVipsStatus.1 = INTEGER: notAllSet(2)
KEEPALIVED-MIB::vrrpInstancePrimaryInterface.1 = STRING: eth0
KEEPALIVED-MIB::vrrpInstanceTrackPrimaryIf.1 = INTEGER: tracked(1)
KEEPALIVED-MIB::vrrpInstanceAdvertisementsInt.1 = Gauge32: 1 seconds(VRRPv2) / centiseconds(VRRPv3)
KEEPALIVED-MIB::vrrpInstancePreempt.1 = INTEGER: preempt(1)
KEEPALIVED-MIB::vrrpInstancePreemptDelay.1 = Gauge32: 0 seconds
KEEPALIVED-MIB::vrrpInstanceAuthType.1 = INTEGER: password(1)
KEEPALIVED-MIB::vrrpInstanceLvsSyncDaemon.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceGarpDelay.1 = Gauge32: 5 seconds
KEEPALIVED-MIB::vrrpInstanceSmtpAlert.1 = INTEGER: disabled(2)
KEEPALIVED-MIB::vrrpInstanceNotifyExec.1 = INTEGER: enabled(1)
KEEPALIVED-MIB::vrrpInstanceScriptFault.1 = STRING: ***
KEEPALIVED-MIB::vrrpInstanceAccept.1 = INTEGER: 0
KEEPALIVED-MIB::vrrpInstancePromoteSecondaries.1 = INTEGER: false(2)
KEEPALIVED-MIB::vrrpInstanceUseLinkbeat.1 = INTEGER: false(2)
KEEPALIVED-MIB::vrrpInstanceVrrpVersion.1 = INTEGER: vrrpv2(2)
KEEPALIVED-MIB::vrrpTrackedScriptName.1.1 = STRING: ***
KEEPALIVED-MIB::vrrpTrackedScriptWeight.1.1 = INTEGER: 0
KEEPALIVED-MIB::vrrpTrackedScriptWgtRev.1.1 = INTEGER: false(2)
KEEPALIVED-MIB::vrrpAddressType.1.1 = INTEGER: ipv4(1)
KEEPALIVED-MIB::vrrpAddressValue.1.1 = STRING: "
{zE"
KEEPALIVED-MIB::vrrpAddressBroadcast.1.1 = Hex-STRING: 00 00 00 00
KEEPALIVED-MIB::vrrpAddressMask.1.1 = Gauge32: 32
KEEPALIVED-MIB::vrrpAddressScope.1.1 = INTEGER: global(14)
KEEPALIVED-MIB::vrrpAddressIfIndex.1.1 = INTEGER: 2
KEEPALIVED-MIB::vrrpAddressIfName.1.1 = STRING: eth0
KEEPALIVED-MIB::vrrpAddressStatus.1.1 = INTEGER: unset(2)
KEEPALIVED-MIB::vrrpAddressAdvertising.1.1 = INTEGER: advertised(1)
KEEPALIVED-MIB::vrrpScriptName.1 = STRING: ***
KEEPALIVED-MIB::vrrpScriptCommand.1 = STRING: ***
KEEPALIVED-MIB::vrrpScriptInterval.1 = INTEGER: 5 seconds
KEEPALIVED-MIB::vrrpScriptWeight.1 = INTEGER: 0
KEEPALIVED-MIB::vrrpScriptResult.1 = INTEGER: good(3)
KEEPALIVED-MIB::vrrpScriptRise.1 = Gauge32: 2
KEEPALIVED-MIB::vrrpScriptFall.1 = Gauge32: 2
KEEPALIVED-MIB::vrrpScriptWgtRev.1 = INTEGER: false(2)
  1. 抓包

一般来说，就以往遇到的问题，往往都是 VRRP 包的收发不对劲，所以可以通过 tcpdump 抓包查看

同时结合 keepalived 的配置，关注抓包的网口，进出方向，vrid, prio

1
tcpdump -i any -n vrrp and inbound