EFK 监控postgresql日志-CFANZ编程社区

服务端（192.168.100.226）
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.0-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/kibana/kibana-7.13.0-linux-x86_64.tar.gz



## elasticsearch
vi /usr/local/elasticsearch/config/elasticsearch.yml
cluster.name: rao-cluster
node.name: node1
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
network.host: 192.168.100.226
http.port: 9200
discovery.seed_hosts: ["192.168.100.226"]
cluster.initial_master_nodes: ["node1"]
# 允许跨域访问，head访问时需要开启
http.cors.enabled: true
http.cors.allow-origin: "*"

useradd es
echo es|passwd --stdin es

解决服务器内存过小而导致启动报错：
vi /etc/security/limits.conf
* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536
root soft nproc 65536
root hard nproc 65536
root soft nofile 65536
root hard nofile 65536

# 查看当前值
ulimit -Hn



vi /etc/sysctl.conf 
vm.max_map_count=655360
生效：sysctl -p


vi /usr/local/elasticsearch/bin/start.sh
su - es -c "nohup /usr/local/elasticsearch/bin/elasticsearch &"


结束进程：  ps aux|grep elasticsearch|awk '{print $2}'|xargs kill

访问：http://192.168.100.226:9200/
{
  "name" : "node1",
  "cluster_name" : "rao-cluster",
  "cluster_uuid" : "tugYCx9bT_KVqnFWZihC4A",
  "version" : {
    "number" : "7.13.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",
    "build_date" : "2021-05-19T22:22:26.081971330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}


## kibana
vi /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.100.226"
server.name: "192.168.100.226"
elasticsearch.hosts: ["http://192.168.100.226:9200"]
i18n.locale: "zh-CN"



vi /usr/local/kibana/bin/start.sh
su - es -c "nohup /usr/local/kibana/bin/kibana &"

访问： http://192.168.100.226:5601/app/home#/



## filebeat 
postgresql数据库节点（192.68.100.220）
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.0-linux-x86_64.tar.gz


主要包含两个主要组件：input和harvesters

harvester: harvester用于按行读取单个文件的内容。每个文件都会启动一个harvester，harvester负责打开和关闭文件。filebeat中还有一个Registrar组件用于记录文件的偏移量，即上一次读取的位置，下一次打开文件时会从Registrar读取偏移量然后继续读取数据。

input：负责管理harvester并且找到所有符合读取条件的文件。如果输入类型为log，则input会在驱动器上找到与定义的路径符合的文件，并会给每个文件都启动一个harvester.


vi /usr/local/filebeat/filebeat.yml
setup.template.settings:
  index.number_of_shards: 1
  # 因为es是单节点，所以将副本分片设置为0.否则会报黄
  index.number_of_replicas: 0
output.elasticsearch:
  hosts: ["192.168.100.226:9200"]
  username: "es"
  password: "es"
setup.kibana:
  host: "192.168.100.226:5601"


启用模块：
cd /usr/local/filebeat/ && ./filebeat modules enable postgresql


编辑模块：
vi /usr/local/filebeat/modules.d/postgresql.yml
- module: postgresql

  log:
    enabled: true
    var.paths: ["/data/postgresql/data/pg_log/*.csv"]


加载kibana仪表盘，如果之前执行过该指令的就无需再执行：cd /usr/local/filebeat/ && ./filebeat setup
启动filebeat ：  cd /usr/local/filebeat/ && nohup ./filebeat -e -c filebeat.yml &


打开加载索引： http://192.168.100.226:5601/app/management/data/index_management/indices