ELK的用途和定义
elk是三个中间件的缩写。Elasticsearch、Logstash、Kibana。
具体的用途和定义可参考这篇文章
https://blog.csdn.net/m0_65196233/article/details/140603248?spm=1001.2014.3001.5501
Elasticsearch：是一个搜索引擎。它主要用于全文搜索、结构化搜索、分析以及实时地存储、检索数据。
Logstash：是一个日志收集引擎。它主要负责从各种数据源采集数据，对数据进行过滤、分析、丰富和转换，然后输出到指定的目的地（如Elasticsearch）。
Kibana：开源的可视化平台。它主要与Elasticsearch一起工作，用于搜索、查看和与存储在Elasticsearch索引中的数据进行交互。
说明
我准备了两台机器。分别用于安装ELK和模拟应用服务器。这个文档是单节点的安装文档，后面会出一个集群的安装文档。
ELK服务器：192.168.3.75（CentOS Linux release 7.9.2009 (Core)） 2C4T 16G 200G（ELK服务器，至少要16G内存，内存高，搜索性能会更高，硬盘容量也要尽可能高一点，因为这里是搜集了所有的日志数据，会占用大量磁盘空间）
应用服务器：192.168.3.99（CentOS Linux release 7.9.2009 (Core)） 4C4T 32G 1T
我下面的步骤都是会将离线包下载好，提前上传到192.168.3.75的/home/packages/elk目录下
安装Elasticsearch
安装Elasticsearch需要先安装jdk，版本最低要求1.8。
可以先检查系统中的jdk版本
[root@centos ~]# java -version
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)如果版本低于1.8，则可以执行下面的命令删除jdk之后，重新安装1.8版本的jdk。然后再次检查是否安装成功
rpm -qa | grep java | xargs rpm -e --nodeps
cd /home/packages/elk
rpm -ivh openlogic-openjdk-8u422-b05-linux-x64-el.rpm
[root@centos ~]# rpm -qa | grep java | xargs rpm -e --nodeps
[root@centos ~]# cd /home/packages/elk
[root@centos elk]# rpm -ivh openlogic-openjdk-8u422-b05-linux-x64-el.rpm
警告：openlogic-openjdk-8u422-b05-linux-x64-el.rpm: 头V4 RSA/SHA256 Signature, 密钥 ID 2aa04dbd: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:openlogic-openjdk-8-hotspot-8u422################################# [100%]
装好之后并不能直接使用，要设置环境变量
vi /etc/profile
# 再最后一行增加下面的内容，然后保存退出
export PATH=$PATH:/usr/lib/jvm/openlogic-openjdk-8-hotspot/bin/java
# 执行下面的命令使更改生效
source /etc/profile下面就可以安装Elasticsearch了，
es离线包地址：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-x86_64.rpm
sudo rpm --install elasticsearch-7.12.0-x86_64.rpm
[root@centos elk]# sudo rpm --install elasticsearch-7.12.0-x86_64.rpm
警告：elasticsearch-7.12.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore然后修改vm.max_map_count值
[root@centos elk]# vi /etc/sysctl.conf
#在最后面一行加上这一句
vm.max_map_count=262144保存后执行以下命令让配置生效
[root@centos elk]# sysctl -p
vm.max_map_count = 262144修改es数据和日志存储路径
[root@centos elk]# mkdir -p /data/elasticsearch/datadir
[root@centos elk]# chown -R elasticsearch:elasticsearch /data/elasticsearch/datadir
[root@centos elk]# mkdir -p /data/elasticsearch/logdir
[root@centos elk]# chown -R elasticsearch:elasticsearch /data/elasticsearch/logdir修改es配置文件
[root@centos elk]# vi /etc/elasticsearch/elasticsearch.yml修改以下信息
# 节点名称
node.name: node-1
# 数据文件存放路径
path.data: /data/elasticsearch/datadir
# 日志文件存放路径
path.logs: /data/elasticsearch/logdir
# 是否使用虚拟内存
bootstrap.memory_lock: false
# 绑定地址
network.host: 0.0.0.0
# 端口号
http.port: 9200
# 注册类型（单节点）
discovery.type: single-node
# 是否启用安全模块
xpack.security.enabled: true
启动es并设置开启自启
[root@centos elk]# systemctl start elasticsearch
[root@centos elk]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since 日 2024-08-04 16:53:04 CST; 20s ago
Docs: https://www.elastic.co
Main PID: 103922 (java)
Tasks: 54
Memory: 8.2G
CGroup: /system.slice/elasticsearch.service
├─103922 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt....
└─104917 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
8月 04 16:52:16 centos systemd[1]: Starting Elasticsearch...
8月 04 16:53:04 centos systemd[1]: Started Elasticsearch.
[root@centos elk]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
注：如果是通过SSH远程登录的服务器，启动ES服务时会报错，原因是部署ES时创建的elasticsearch用户默认是禁止shell登录的，通过切换用户命令可以看到提示：this account is currently not avaliable。
[root@centos elk]# cat /etc/passwd | grep elasticsearch
elasticsearch:x:986:979:elasticsearch user:/nonexistent:/sbin/nologin解决方案：将elasticsearch用户的shell从“/sbin /nologin”修改为“/bin/bash”即可。
# 修改shell配置，将elasticsearch用户的shell从“/sbin/nologin”修改为“/bin/bash”
vi /etc/passwd然后再重启es，问题就解决了
启动之后设置账号密码，这里会让你设置很多个账号的密码，设置完之后你要记住每个账号设置的密码是什么，后面需要用，这里我把所有的账号密码都设置成了123456
[root@centos elk]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]
安装Logstash
下载离线包，进行安装
离线包地址：https://artifacts.elastic.co/downloads/logstash/logstash-7.12.0-x86_64.rpm
[root@centos elk]# sudo rpm --install logstash-7.12.0-x86_64.rpm
警告：logstash-7.12.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
Using bundled JDK: /usr/share/logstash/jdk
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash新建数据文件目录和日志文件目录并授权
# 创建目录，用于存储logstash数据
mkdir -p /data/logstash/datadir
# 修改目录权限
chown -R logstash:logstash /data/logstash/datadir
# 创建日志目录
mkdir -p /data/logstash/logdir
# 修改目录权限
chown -R logstash:logstash /data/logstash/logdir修改logstash的数据和日志存储路径，vi /etc/logstash/logstash.yml
# 数据文件存放路径
path.data: /data/logstash/datadir
# 配置文件存放路径
path.config: /etc/logstash/conf.d
# 日志文件存放路径
path.logs: /data/logstash/logdir
启动logstash并设置开机自启
[root@centos elk]# systemctl start logstash
[root@centos elk]# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since 日 2024-08-04 17:37:48 CST; 3s ago
Main PID: 37152 (java)
Tasks: 18
Memory: 551.9M
CGroup: /system.slice/logstash.service
└─37152 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccup...
8月 04 17:37:48 centos systemd[1]: Started logstash.
8月 04 17:37:48 centos logstash[37152]: Using bundled JDK: /usr/share/logstash/jdk
8月 04 17:37:49 centos logstash[37152]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likel... release.
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos elk]# systemctl enable logstash
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
安装Kibana
下载离线包，进行安装
离线包地址：https://artifacts.elastic.co/downloads/kibana/kibana-7.12.0-x86_64.rpm
[root@centos elk]# sudo rpm --install kibana-7.12.0-x86_64.rpm
警告：kibana-7.12.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
修改kibana配置文件，vi /etc/kibana/kibana.yml
# 端口号
server.port: 5601
# 主机ip
server.host: "0.0.0.0"
# 索引名
kibana.index: ".kibana"
# es账号
elasticsearch.username: "kibana_system"
# es密码
elasticsearch.password: "123456"
# 设置语言为中文
i18n.locale: "zh-CN"
启动kibana并设置开机自启
[root@centos elk]# systemctl start kibana
[root@centos elk]# systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
Active: active (running) since 日 2024-08-04 17:53:45 CST; 4s ago
Docs: https://www.elastic.co
Main PID: 65172 (node)
Tasks: 11
Memory: 230.3M
CGroup: /system.slice/kibana.service
└─65172 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/r...
8月 04 17:53:45 centos systemd[1]: Started Kibana.
[root@centos elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
安装好之后，我们通过浏览器访问http://192.168.3.75:5601，查看elk是否正常安装，若能通过之前设置的账号密码正常登录，则说明elk已经安装成功了
.png)
.png)
安装Filebeat（日志数据收集）
这个组件是收集日志的，所以要在每个存放日志的服务器上各自安装。我这里用192.168.3.99这台机器当作应用服务器。
下载离线包，进行安装：
离线包地址：https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-x86_64.rpm
[root@n5105 elk]# sudo rpm -vi filebeat-7.12.0-x86_64.rpm
警告：filebeat-7.12.0-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
软件包准备中...
filebeat-7.12.0-1.x86_64安装好之后，修改配置文件，vi /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
# 需要收集的日志文件路径，可以填写多个，格式如下
# - /home/nacos/logs/*.log
# - /home/nacos2/logs/*.log
# - /home/nacos3/logs/*.log
- /home/nacos/logs/*.log
# 自定义的字段
fields:
# 自定义字段名字和值（用于收集日志之后，进行分类，分类之后可以根据不同类型创建不同的索引）
log_type: nacos
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/*.log
fields:
log_type: nginx
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# 日志输出到logstash
output.logstash:
# The Logstash hosts
hosts: ["192.168.3.75:5044"]
修改好之后，先别启动filebeat，我们来到elk服务器，找到/etc/logstash/conf.d目录，在这个目录下面新建一个配置文件，用于接收处理收集的日志，input.conf，里面添加以下内容。配置中的账号密码，自定义字段，以及es的地址，要根据自己的实际情况进行修改，改好之后重启logstash（systemctl restart logstash）
input {
beats {
port => 5044
}
}
filter {
if [fields][log_type] == "nginx" {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
}
}
if [fileset][name] == "error" {
grok {
match => { "message" => "\[%{HTTPDATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] \[%{NUMBER:pid}\#%{NUMBER:tid}\] \[%{DATA:module}\] (?:%{DATA:message}, )?client: %{IPORHOST:client}, server: %{DATA:server}, request: \"%{DATA:request}\", upstream: \"%{DATA:upstream}\", host: \"%{DATA:host}\"" }
}
}
}
}
if [fields][log_type] == "nacos" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}
geoip {
source => "clientip"
}
mutate {
convert => [ "[geoip]", "float" ]
}
}
}
output {
if [fields][log_type] == "nginx" {
elasticsearch {
hosts => ["192.168.3.75:9200"]
index => "nginx-%{+YYYY.MM.dd}"
user => "elastic"
password => "123456"
}
}
if [fields][log_type] == "nacos" {
elasticsearch {
hosts => ["192.168.3.75:9200"]
index => "nacos-%{+YYYY.MM.dd}"
user => "elastic"
password => "123456"
}
}
}
然后在转到应用服务器，启动filebeat（systemctl start filebeat）
建立索引以及简单使用方法
安装好elk，并且配置好filebeat之后，filebeat会自动收集应用服务器中的日志，发送到es服务器。我们登录Kibana。
然后我们点击首页的管理，或者点击左侧菜单
.png)
然后点击索引管理，这里就会出现你刚刚收集的日志
.png)
查看到有数据之后，我们新建索引模式，如下图
.png)
选择时间，点击创建
.png)
索引创建好之后，我们就可以点击Discover，进入
.png)
根据图片内容，可以进行搜索
.png)
其实ELK可以集成到Java项目中，还有很多其他的功能，可以查看请求ip所在地区，对数据进行分析得到图表等等。其他功能可自行探索
.jpeg)
评论区