把9306的arp表项转存到elastic

具体流程如下：

1. zabbix server通过shell脚本snmpwalk获取到华为9306各接口下的arp表项
2. zabbix server上的filebeat收集好arp表项传给logstash
3. logstash对数据进行结构化和丰富mac厂家信息后传给elastic

zabbix web配置

• zabbix web配置自动发现规则，自动发现9306端口运行状态为UP的端口且获取UP端口的index

discovery[{#IFOPERSTATUS},1.3.6.1.2.1.2.2.1.8,{#IFALIAS},1.3.6.1.2.1.31.1.1.1.18,{#IFNAME},.1.3.6.1.2.1.2.2.1.2]

9306引用配置好的自动发现规则后，自动获取到up接口的监控项如下

• 键值 net.if.status[ifOperStatus.39(GigabitEthernet1/0/33)]中的39就对应接口的索引（{#SNMPINDEX}）

zabbix server相关配置

• crontab计划任务配置

# 每隔10分钟把9306的arp保存到/var/log/9306arp.log。xxx为接口对应的索引SNMPINDEX
*/10 * * * * bash -x /usr/lib/zabbix/externalscripts/ipaddr_discovery.sh xxx xxx... &> /tmp/arp.log

#每天凌晨3点执行删除9306arp日志；by:yujing 2024/9/2
00 03 * * * bash -x /opt/delete_9306arp_log.sh &> /tmp/delete.log

相关脚本配置如下：

cat /opt/delete_9306arp_log.sh
find /var/log/ -type f -name "9306arp*.log"  -mtime +3   | xargs rm -f

cat /usr/lib/zabbix/externalscripts/ipaddr_discovery.sh
#!/usr/bin/bash

Date=$(date +%F)

for j in $@
do
array_ip=($(snmpwalk -v 2c -c xxx x.x.x.x ipNetToMediaNetAddress.$j | awk -F ' ' '{print $NF}' | sort -t "." -k4 -n))

        for i in ${array_ip[@]}
        do
                mac_addr=($(snmpwalk -v 2c -c xxx x.x.x.x ipNetToMediaPhysAddress.$j.$i | awk -F ' ' '{print $NF}'))
                full_mac=$(echo "STRING: $mac_addr" | perl -pe "s/(?<=:|\s)([a-f0-9A-F])(?=:|$)/0\1/g" | awk -F '[ ]' '{print $NF}')
                echo "$i $full_mac $full_mac"  >> /var/log/9306arp_${Date}.log
        done
done

• zabbix server上的filebeat配置

cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/9306arp*.log

output.logstash:
  hosts: ["x.x.x.x:5044"]

logstash上的配置

cat /etc/logstash/conf.d/network.logstash.conf
input {
  beats {
        add_field => {"beatType" => "filebeat"}
        port => 5044
        }
}
filter {
if [beatType] == "filebeat"{
        grok {
        match => { "message" => "%{IP:srcip}\s%{MAC:macaddr}\s(?<MAC1>[A-Fa-f0-9]{2}):(?<MAC2>[A-Fa-f0-9]{2}):(?<MAC3>[A-Fa-f0-9]{2})" }
        add_field => { "macprefix" => "%{MAC1}%{MAC2}%{MAC3}" }
        remove_field => ["MAC1","MAC2","MAC3"]
      }
        mutate {
        add_field => { "version" => "4" }
        uppercase => ["macprefix"]
        }
        translate {
        field => "macprefix"
        destination => "macoui"
        ##没查到就标一个no_match
        fallback => "no_match"
        ##下面标一下字典文件的位置
        dictionary_path => "/tmp/macoui.yml"
        ##字典文件可以更新后可以直接覆盖，logstash会间隔时间过来看看该文件是否发生改变了
        refresh_interval => 300
        ##如果发现文件变了就重新加载，默认动作是merge合并，本例中我们用replace，要确保该文件始终是最全的
        refresh_behaviour => "replace"
     }
  }
        date {
        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
        mutate {
        remove_field => ["message"]
        }
if [version] == "4"{
        elasticsearch {
        hosts => ["x.x.x.x:9200","x.x.x.x:9200","x.x.x.x:9200"]
        user => "elastic"
        password => "xxxxxx"
        query_template => "search-by-ipv4.json"
        index => "ipaddrfordep"
        fields => {
        "department" => "[department]"
    }
  }
 }
}
output {
 if [beatType] == "filebeat"{
    elasticsearch {
      index => "9306-arplog-%{+YYYY.MM.dd}"
      hosts => ["x.x.x.x:9200","x.x.x.x:9200","x.x.x.x:9200"]
      user => "elastic"
      password => "xxxxxx"
    }
  }
}

下载开源脚本获取mac地址的oui
https://github.com/Vigilant-LLC/logstash-oui-scraper
保存为mac.sh
运行
sh mac.sh oui-scraper -d /tmp
这条命令可以把IEEE里的OUI列表直接取回来，而且还可以自动进行格式处理
查看/tmp目录下的

cat /tmp/oui-logstash.txt | wc -l
36068

tail /tmp/oui-logstash.txt
BCB4FD  NXP Semiconductor (Tianjin) LTD.
580987  Amazon Technologies Inc.
94F717  CIG SHANGHAI CO LTD
7C5184  Unis Flash Memory Technology(Chengdu)Co.,Ltd.
E806EB  ShieldSOS LLC
78202E  Skychers Creations ShenZhen Limited
DC1ED5  Espressif Inc.
946C04  EM Microelectronic
E0C250  NETGEAR
AC04AA  GoPro

还需对oui-logstash.txt进行处理

# 选运行这个命令，把^M$去掉：
cat oui-logstash.txt | tr -d '\r' > macoui.txt
# 转为yml格式
sed -i  -e 's/^/"/1' -e 's/\t/": "/1' -e 's/$/"/g' macoui.txt
cp macoui.txt macoui.yml

tail macoui.yml
"585924": "Nanjing Simon Info Tech Co.,Ltd."
"909DAC": "Infinix mobility limited"
"24DE8A": "Nokia Solutions and Networks GmbH & Co. KG"
"C8E5E0": "HUAWEI TECHNOLOGIES CO.,LTD"
"BCB4FD": "NXP Semiconductor (Tianjin) LTD."
"580987": "Amazon Technologies Inc."
"94F717": "CIG SHANGHAI CO LTD"
"E806EB": "ShieldSOS LLC"
"7C5184": "Unis Flash Memory Technology(Chengdu)Co.,Ltd."
"78202E": "Skychers Creations ShenZhen Limited"

elastic配置（在kibana上配置）

1. 配置索引模板9306-arplog-template

• 映射如下配置：

• 生命周期如下配置

2. 索引管理
   配置好后在索引管理上会看到名称为9306-arplog-2024.09.02的索引

然后添加好9306-arplog*的索引模式

3. discover菜单