天御实验室官方博客
整合HIDS、NIDS和Elastic Stack，在此基础上实现SOC
The Elastic Stack delivers security analytics capabilities that are widely used for threat detection, visibility, and incident response. The speed and scale at which Elasticsearch can index and search security-related information enable security analysts to work more efficiently, while Kibana dashboards provide wide visibility and enable interactive threat hunting. And the machine learning engine can automate the analysis of complex datasets, making it possible to spot intruders that otherwise would’ve gone unnoticed.
Popular Intrusion Detection Systems (IDS), such as Wazuh or Snort/Suricata, use a signature-based approach to threat detection. That is, they compare patterns found in files, logs, and network traffic against a database of patterns known to be associated with malicious activity, alerting when a match is found. They provide useful rulesets to analyze and correlate data, usually generating thousands or millions of alerts per day in a production environment.
Casting a wide net can ensure that all potential security events are caught, but it also adds the work of sifting through thousands (or millions) of alerts every day. Elastic machine learning features help reduce the noise by automatically identifying unusual behaviors. This is a clear use case where anomaly-based and signature-based technologies complement each other, making threat detection easier and investigations more efficient.
1.部署 Snort/Suricata
Ubuntu
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata
RHEL/CentOS
yum install epel-release
yum install suricata
参考：
Suricata
https://github.com/tianyulab/dalton/blob/master/dalton-agent/Dockerfiles/Dockerfile_suricata
https://suricata.readthedocs.io/en/suricata-4.0.5/install.html
Snort
https://github.com/tianyulab/SnortCP/blob/master/Scripts/Snort_Wireshark.sh
2.配置Suricata Eve JSON Output
# 配置举例：
vi /etc/suricata/suricata.yaml
outputs:
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: eve.json
types:
- alert:
metadata: yes
tagged-packets: yes
xff:
enabled: yes
mode: extra-data
- http:
extended: yes
- dns:
query: yes # enable logging of DNS queries
answer: yes # enable logging of DNS answers
- tls:
extended: yes # enable this for extended logging information
- files:
force-magic: no # force logging magic on all logged files
- smtp:
extended: yes # enable this for extended logging information
- ssh
- flow
参考：
https://suricata.readthedocs.io/en/suricata-4.0.5/configuration/suricata-yaml.html#eve-extensible-event-format
3.部署 Wazuh Stack
Wazuh stack包含3个组件：
1.Wazuh server: 包含Wazuh manager，API 和 Filebeat（Filebeat仅在分布式架构下使用）
2.Elastic Stack: 包含Elasticsearch，Logstash，Kibana 和 Wazuh Kibana app，读取，解析，索引和存储Wazuh服务器生成的警报数据。
3.Wazuh agent
# 分布式架构：在不同主机上运行Wazuh服务器和Elastic Stack集群（一个或多个服务器）
https://documentation.wazuh.com/current/_images/installing_wazuh2.png
# 单主机架构：在同一系统上运行Wazuh服务器和Elastic Stack
https://documentation.wazuh.com/current/_images/installing_wazuh_singlehost2.png
# 本文采用分布式架构，分别在Ubuntu 16.04上部署Wazuh server，CentOS 7.x上部署Elastic Stack
Ubuntu 16.04
1.部署Wazuh server
# 1.添加Wazuh Repositories
## 1.1）安装依赖软件包
apt-get update
apt-get -y install curl apt-transport-https lsb-release
# if [ ! -f /usr/bin/python ]; then ln -s /usr/bin/python3 /usr/bin/python; fi # 可选
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
echo "deb https://packages.wazuh.com/3.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
# 2.安装wazuh-manager
apt-get update
apt-get -y install wazuh-manager
systemctl status wazuh-manager
systemctl enable wazuh-manager
# 3.安装Wazuh API
## 3.1）安装依赖软件包，NodeJS >= 4.6.1，Python >= 2.7
curl -sL https://deb.nodesource.com/setup_8.x | bash -
apt-get -y install nodejs
apt-get -y install wazuh-api
systemctl status wazuh-api
systemctl enable wazuh-api
# 4.配置Wazuh Kibana app
参考：
https://documentation.wazuh.com/current/user-manual/kibana-app/connect-kibana-app.html
# 5.安装Filebeat（分布式架构）
curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get -y install filebeat=6.4.2
curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/3.6/extensions/filebeat/filebeat.yml
修改/etc/filebeat/filebeat.yml中ELASTIC_SERVER_IP的值为Elastic Stack服务器IP
systemctl daemon-reload
systemctl enable filebeat.service
systemctl start filebeat.service
# 6.验证
systemctl status wazuh-manager
systemctl status wazuh-api
systemctl status filebeat.service
# 7.一键部署脚本
https://github.com/tianyulab/Threat_Hunting_with_ELK/tree/master/HIDS_NIDS_ELK/Deploy_Wazuh_server.sh
2.部署Elastic Stack
CentOS 7.x
# 1.安装依赖软件包，JRE
curl -Lo jre-8-linux-x64.rpm --header "Cookie: oraclelicense=accept-securebackup-cookie" "https://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jre-8u191-linux-x64.rpm"
rpm -qlp jre-8-linux-x64.rpm > /dev/null 2>&1 && echo "Java package downloaded successfully" || echo "Java package did not download successfully"
yum -y install jre-8-linux-x64.rpm
rm -f jre-8-linux-x64.rpm
# 2.安装elasticsearch、logstash、kibana
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elastic.repo << EOF
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum install elasticsearch-6.4.2
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
curl "localhost:9200/?pretty"
# Load the Wazuh template for Elasticsearch:
curl https://raw.githubusercontent.com/wazuh/wazuh/3.6/extensions/elasticsearch/wazuh-elastic6-template-alerts.json | curl -XPUT 'http://localhost:9200/_template/wazuh' -H 'Content-Type: application/json' -d @-
yum install logstash-6.4.2
# Download the Wazuh configuration file for Logstash:
curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/3.6/extensions/logstash/01-wazuh-remote.conf
systemctl daemon-reload
systemctl enable logstash.service
systemctl start logstash.service
yum install kibana-6.4.2
export NODE_OPTIONS="--max-old-space-size=3072"
sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.6.1_6.4.2.zip
# 参考：https://github.com/wazuh/wazuh-kibana-app#installation
# /etc/kibana/kibana.yml # 可选
# server.host: "0.0.0.0"
systemctl daemon-reload
systemctl enable kibana.service
systemctl start kibana.service
# 3.验证
curl "localhost:9200/?pretty"
systemctl status logstash.service
systemctl status kibana.service
# 4.一键部署脚本
https://github.com/tianyulab/Threat_Hunting_with_ELK/tree/master/HIDS_NIDS_ELK/Deploy_Elastic_Stack.sh
4.在Suricata服务器上安装Wazuh Agent
Ubuntu 16.04
# 1.部署Wazuh Agent
apt-get -y install curl apt-transport-https lsb-release
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
echo "deb https://packages.wazuh.com/3.x/apt/ stable main" | tee /etc/apt/sources.list.d/wazuh.list
apt-get update
apt-get -y install wazuh-agent
# 2.注册Wazuh Agent
# Wazuh Manager 上执行：
openssl req -x509 -batch -nodes -days 365 -newkey rsa:2048 -keyout /var/ossec/etc/sslmanager.key -out /var/ossec/etc/sslmanager.cert
/var/ossec/bin/ossec-authd -i
# Wazuh Agent 上执行：
sed -i "s/MANAGER_IP/8.8.8.8/" /var/ossec/etc/ossec.conf
/var/ossec/bin/agent-auth -m 8.8.8.8
systemctl restart wazuh-agent
# 参考：
https://documentation.wazuh.com/current/user-manual/registering/index.html
# 3.验证
systemctl status wazuh-agent
# 4.一键部署脚本
https://github.com/tianyulab/Threat_Hunting_with_ELK/tree/master/HIDS_NIDS_ELK/Deploy_Wazuh_agent.sh
# 注：此脚本为交互模式
5.在Wazuh Manager服务器上配置Wazuh rules处理Suricata日志
sed -i 's/id="86600" level="0"/id="86600" level="4"/g' /var/ossec/ruleset/rules/0475-suricata_rules.xml
sed -i 's/id="86602" level="0"/id="86602" level="4"/g' /var/ossec/ruleset/rules/0475-suricata_rules.xml
sed -i 's/id="86603" level="0"/id="86603" level="4"/g' /var/ossec/ruleset/rules/0475-suricata_rules.xml
sed -i 's/id="86604" level="0"/id="86604" level="4"/g' /var/ossec/ruleset/rules/0475-suricata_rules.xml
systemctl restart wazuh-manager.service
# /var/ossec/bin/ossec-control restart
6.在Suricata服务器上配置Wazuh Agent读取Suricata的eve.json文件
vi /var/ossec/etc/ossec.conf
# 在<ossec_config> tag里添加如下内容
# Modify ossec.conf - read localfile suricata EVE json log
<localfile>
<log_format>syslog</log_format>
<location>/var/log/suricata/eve.json</location>
</localfile>
# 验证
systemctl restart wazuh-agent
systemctl status wazuh-agent
7.在Elastic Stack上配置wazuh logstash filter
# 1.wazuh logstash filter 配置
在Elastic Stack服务器上执行，
vi /etc/logstash/conf.d/01-wazuh.conf
# 新增以下内容
filter {
if [data][src_ip] {
mutate{
add_field => [ "[data][srcip]","%{[data][src_ip]}"]
remove_field => [ "[data][src_ip]" ]
}
}
if [data][dest_ip] {
mutate{
add_field => [ "[data][dstip]","%{[data][dest_ip]}"]
remove_field => [ "[data][dest_ip]" ]
}
}
if [data][dest_port] {
mutate{
add_field => [ "[data][dstport]","%{[data][dest_port]}"]
remove_field => [ "[data][dest_port]" ]
}
}
if [data][src_port] {
mutate{
add_field => [ "[data][srcport]","%{[data][src_port]}"]
remove_field => [ "[data][src_port]" ]
}
}
}
# 配置验证
/usr/share/logstash/bin/logstash -f 01-wazuh.conf --config.test_and_exit
# 重启Logstash服务
systemctl restart logstash.service
8.Wazuh + Snort/Suricata 联动（active response）
# 1.Snort
# 修改snort输出为alert_fast：
vi /etc/snort/snort.conf
output alert_fast: snort.log 128M
systemctl restart snort
# 配置Wazuh agent
vi /var/ossec/etc/ossec.conf
<localfile>
<log_format>snort-full</log_format>
<location>/var/log/snort/snort.log</location>
</localfile>
systemctl restart wazuh-agent
# 配置Wazuh Manager
vi /var/ossec/etc/ossec.conf
# 添加如下内容：
<!-- Active response -->
<global>
<white_list>127.0.0.1</white_list>
<white_list>^localhost.localdomain$</white_list>
<white_list>8.8.8.8</white_list>
</global>
<command>
<name>disable-account</name>
<executable>disable-account.sh</executable>
<expect>user</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>restart-ossec</name>
<executable>restart-ossec.sh</executable>
<expect></expect>
</command>
<command>
<name>firewall-drop</name>
<executable>default-firewall-drop.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>host-deny</name>
<executable>host-deny.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>route-null</name>
<executable>route-null.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>win_route-null</name>
<executable>route-null.cmd</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>win_route-null-2012</name>
<executable>route-null-2012.cmd</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>netsh</name>
<executable>netsh.cmd</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<command>
<name>netsh-win-2016</name>
<executable>netsh-win-2016.cmd</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
<!-- Active Response Config -->
<active-response>
<!-- This response is going to execute the host-deny
- command for every event that fires a rule with
- level (severity) >= 6.
- The IP is going to be blocked for 600 seconds.
-->
<command>host-deny</command>
<location>local</location>
<level>12</level>
<timeout>600</timeout>
</active-response>
<active-response>
<!-- Firewall Drop response. Block the IP for
- 600 seconds on the firewall (iptables,
- ipfilter, etc).
-->
<command>firewall-drop</command>
<location>local</location>
<level>12</level>
<timeout>600</timeout>
</active-response>
<!-- Snort active response 配置 -->
<active-response>
<command>firewall-drop</command>
<location>local</location>
<rules_id>20101</rules_id>
<timeout>600</timeout>
</active-response>
<active-response>
<command>host-deny</command>
<location>local</location>
<rules_id>20101</rules_id>
<timeout>600</timeout>
</active-response>
# 重启服务，
systemctl restart wazuh-manager.service
参考：
https://groups.google.com/forum/#!msg/wazuh/8cu1hZ9PHCM/RiPK41gWAgAJ
https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html?highlight=localfile#log-format
/var/ossec/ruleset/decoders/0285-snort_decoders.xml
/var/ossec/ruleset/rules/0240-ids_rules.xml
https://github.com/wazuh/wazuh-ruleset/blob/master/decoders/0285-snort_decoders.xml
# 2.Suricata
省略
参考：
https://github.com/wazuh/wazuh/issues/202
https://documentation.wazuh.com/current/user-manual/ruleset/custom.html
https://documentation.wazuh.com/current/user-manual/ruleset/json-decoder.html
调试/测试工具：
/var/ossec/bin/ossec-logtest
效果图：
9.未完待续
1.机器学习实例
2.Wazuh和Snort/Suricata事件关联
10.致谢
Santiago Basset@Wazuh Team
作者：Zer0d0y
微信关注：Zer0d0y
本文出处：https://blog.tianyulab.com/post/ty-practical-guide-to-threat-hunting-03/
本站评论使用Disqus，如果长时间无法加载，请切换至“自由”互联网。
文章版权归本人所有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接，否则保留追究法律责任的权利。