centos oracle 安装,ELK单机安装与集群安装 | 运维社区

Elasticsearch 安装

在跳板机上配置 yum 仓库

拷贝云盘 rpm-package/elk 目录到跳板机

(链接: https://pan.baidu.com/s/1JN58RIr8C9IBwRXsn2AKBw?pwd=yrmf 提取码: yrmf)

[root@ecs-proxy ~]# cp -a elk /var/ftp/localrepo/elk[root@ecs-proxy ~]# cd /var/ftp/localrepo/[root@ecs-proxy localrepo]# createrepo –update .

购买云主机

单机安装

[root@es-0001 ~]# vim /etc/hosts192.168.1.41 es-0001[root@es-0001 ~]# yum install -y java-1.8.0-openjdk elasticsearch[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml55: network.host: 0.0.0.0[root@es-0001 ~]# systemctl enable –now elasticsearch[root@es-0001 ~]# curl http://192.168.1.41:9200/{“name” : “War Eagle”,“cluster_name” : “elasticsearch”,“version” : {“number” : “2.3.4”,“build_hash” : “e455fd0c13dceca8dbbdbb1665d068ae55dabe3f”,“build_timestamp” : “2016-06-30T11:24:31Z”,“build_snapshot” : false,“lucene_version” : “5.5.0”},“tagline” : “You Know, for Search”}

集群安装

es-0001 … es-0005 所有主机，都要执行以下操作

[root@es-0001 ~]# vim /etc/hosts192.168.1.41 es-0001192.168.1.42 es-0002192.168.1.43 es-0003192.168.1.44 es-0004192.168.1.45 es-0005[root@es-0001 ~]# yum install -y java-1.8.0-openjdk elasticsearch[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml17: cluster.name: my-es23: node.name: es-0001 # 本机主机名55: network.host: 0.0.0.068: discovery.zen.ping.unicast.hosts: [“es-0001”, “es-0002”][root@es-0001 ~]# systemctl enable –now elasticsearch[root@es-0001 ~]# curl http://192.168.1.41:9200/_cluster/health?pretty{“cluster_name” : “my-es”,“status” : “green”,“timed_out” : false,“number_of_nodes” : 5,“number_of_data_nodes” : 5,… …}

插件安装

插件原理图

购买云主机

安装 apache，并部署 head 插件

[root@web ~]# yum install -y httpd[root@web ~]# tar zxf head.tar.gz[root@web ~]# mv elasticsearch-head /var/www/html/head[root@web ~]# systemctl enable –now httpdCreated symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

使用华为 ELB，把 web 服务和 es-0001 的 9200 端口发布到互联网，并通过浏览器访问

es-0001 访问授权

[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml# 配置文件最后追加http.cors.enabled : truehttp.cors.allow-origin : “*”http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETEhttp.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length[root@es-0001 ~]# systemctl restart elasticsearch.service

使用插件访问 es 集群服务

Elasticsearch基本操作

查询_cat方法

# 查询支持的关键字[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/# 查具体的信息[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master# 显示详细信息 ?v[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master?v# 显示帮助信息 ?help[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cat/master?help

创建索引

指定索引的名称，指定分片数量，指定副本数量

创建索引使用 PUT 方法，创建完成以后通过 head 插件验证

增加数据

查询数据

[root@es-0001 ~]# curl -XGET http://es-0001:9200/tedu/teacher/1?pretty

修改数据

[root@es-0001 ~]# curl -XPOST -H “Content-Type: application/json” \http://es-0001:9200/tedu/teacher/1/_update -d ‘{ “doc”: {“年代”:”公元701″}}’

删除数据

# 删除一条[root@es-0001 ~]# curl -XDELETE http://es-0001:9200/tedu/teacher/1# 删除索引[root@es-0001 ~]# curl -XDELETE http://es-0001:9200/tedu

kibana安装

购买云主机

安装kibana

[root@kibana ~]# vim /etc/hosts192.168.1.41 es-0001192.168.1.42 es-0002192.168.1.43 es-0003192.168.1.44 es-0004192.168.1.45 es-0005192.168.1.46 kibana[root@kibana ~]# yum install -y kibana[root@kibana ~]# vim /etc/kibana/kibana.yml02 server.port: 560107 server.host: “0.0.0.0”28 elasticsearch.hosts: [“http://es-0002:9200”, “http://es-0003:9200”]37 kibana.index: “.kibana”40 kibana.defaultAppId: “home”113 i18n.locale: “zh-CN”[root@kibana ~]# systemctl enable –now kibana

绑定弹性公网IP，通过 WEB 浏览器验证

导入日志数据

拷贝云盘 logs.jsonl.gz 到跳板机

[root@ecs-proxy ~]# gunzip logs.jsonl.gz[root@ecs-proxy ~]# curl -XPOST -H “Content-Type: application/json” http://192.168.1.41:9200/_bulk –data-binary @logs.jsonl

logstash安装

购买云主机

logstash云主机安装

[root@logstash ~]# vim /etc/hosts192.168.1.41 es-0001192.168.1.42 es-0002192.168.1.43 es-0003192.168.1.44 es-0004192.168.1.45 es-0005192.168.1.47 logstash[root@logstash ~]# yum install -y java-1.8.0-openjdk logstash

基础配置样例

[root@logstash ~]# ln -s /etc/logstash /usr/share/logstash/config[root@logstash ~]# vim /etc/logstash/conf.d/my.confinput {stdin {}}

filter{ }

output{stdout{}}[root@logstash ~]# /usr/share/logstash/bin/logstash

插件与调试格式

使用json格式字符串测试 {“a”:”1″, “b”:”2″, “c”:”3″}

[root@logstash ~]# vim /etc/logstash/conf.d/my.confinput {stdin { codec => “json” }}

filter{ }

output{stdout{ codec => “rubydebug” }}[root@logstash ~]# /usr/share/logstash/bin/logstash

官方手册地址

https://www.elastic.co/guide/en/logstash/current/index.html

input file插件

[root@logstash ~]# vim /etc/logstash/conf.d/my.confinput {file {path => [“/tmp/c.log”]type => “test”start_position => “beginning”sincedb_path => “/var/lib/logstash/sincedb”}}filter{ }output{stdout{ codec => “rubydebug” }}[root@logstash ~]# rm -rf /var/lib/logstash/plugins/inputs/file/.sincedb_*[root@logstash ~]# /usr/share/logstash/bin/logstash

filter grok插件

正则表达式分组匹配格式: (?正则表达式)

正则表达式宏调用格式: %{宏名称:名字}

宏文件路径

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns

[root@logstash ~]# echo ‘192.168.1.252 – – [29/Jul/2020:14:06:57 +0800] “GET /info.html HTTP/1.1” 200 119 “-” “curl/7.29.0″‘ >/tmp/c.log[root@logstash ~]# vim /etc/logstash/conf.d/my.confinput {file {path => [“/tmp/c.log”]type => “test”start_position => “beginning”sincedb_path => “/dev/null”}}filter{grok {match => { “message” => “%{HTTPD_COMBINEDLOG}” }}}output{stdout{ codec => “rubydebug” }}[root@logstash ~]# /usr/share/logstash/bin/logstash

output elasticsearch插件