nginx 엑세스 로그 포맷에서는 “request_time” 옵션이 빠져있다.

“request_time” metrics 수집을 위해서 설정해준다.

log_format를 아래와 같이 수정해준다.

sudo vi /etc/nginx/nginx.conf
... 중간 생략 ...
http {
    log_format  main  '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" "$request_time"';
    access_log  /var/log/nginx/access.log  main;
... 중간 생략 ...    
service nginx restart

2. nginx log 권한 설정

nginx log exporter가 로그를 수집하기 위해서 로그 접근 권한 설정

sudo chmod 777 /var/log/nginx/access.log

2. Nginx log exporter 설정

1. nginx log exporter 설치

nginx log exporter GitHub URL

GitHub - nginx log exporter 메뉴얼

GitHub - nginx log exporter 다운로드

nginx log exporter 다운로드

mkdir -p nginx_log cd nginx_log wget https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.10.0/prometheus-nginxlog-exporter_1.10.0_linux_amd64.tar.gz tar xvfz prometheus-nginxlog-exporter_1.10.0_linux_amd64.tar.gz

nginx log exporter yaml 설정

sudo mkdir -p /etc/prometheus-nginxlog-exporter
sudo vi /etc/prometheus-nginxlog-exporter/prometheus-nginxlog-exporter.yml
listen:
  port: 4040
  metrics_endpoint: "/metrics"
consul:
  enable: false
namespaces:
  - name: nginx
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\""
    source:
      files:
        - /var/log/nginx/access.log
    only_count: true
    relabel_configs:
    - target_label: request_uri
      from: request
      split: 2
      separator: ' '  
sudo chown -R prometheus_nginxlog_exporter.prometheus_nginxlog_exporter /etc/prometheus-nginxlog-exporter/

nginx log exporter 서비스 등록

sudo useradd -M -r -s /bin/false prometheus_nginxlog_exporter
sudo cp ./prometheus-nginxlog-exporter /usr/local/bin
sudo chown prometheus_nginxlog_exporter:prometheus_nginxlog_exporter /usr/local/bin/prometheus-nginxlog-exporter
sudo tee /etc/systemd/system/prometheus_nginxlog_exporter.service << EOF
[Unit]
Description=Prometheus Nginxlog Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus_nginxlog_exporter
Group=prometheus_nginxlog_exporter
Type=simple
ExecStart=/usr/local/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter/prometheus-nginxlog-exporter.yml
[Install]
WantedBy=multi-user.target
nginx log exporter 실행
sudo systemctl daemon-reload
sudo systemctl restart prometheus_nginxlog_exporter
sudo systemctl status prometheus_nginxlog_exporter
nginx expoter 확인
curl localhost:4040/metrics
# HELP nginx_http_response_count_total Amount of processed HTTP requests
# TYPE nginx_http_response_count_total counter
nginx_http_response_count_total{method="GET",request_uri="/metrics",status="200"} 4
# HELP nginx_http_response_size_bytes Total amount of transferred bytes
# TYPE nginx_http_response_size_bytes counter
nginx_http_response_size_bytes{method="GET",request_uri="/metrics",status="200"} 404
# HELP nginx_http_response_time_seconds Time needed by NGINX to handle requests
# TYPE nginx_http_response_time_seconds summary
nginx_http_response_time_seconds{method="GET",request_uri="/metrics",status="200",quantile="0.5"} 0
nginx_http_response_time_seconds{method="GET",request_uri="/metrics",status="200",quantile="0.9"} 0
nginx_http_response_time_seconds{method="GET",request_uri="/metrics",status="200",quantile="0.99"} 0
nginx_http_response_time_seconds_sum{method="GET",request_uri="/metrics",status="200"} 0
nginx_http_response_time_seconds_count{method="GET",request_uri="/metrics",status="200"} 4
# HELP nginx_http_response_time_seconds_hist Time needed by NGINX to handle requests
# TYPE nginx_http_response_time_seconds_hist histogram
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.005"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.01"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.025"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.05"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.1"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.25"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="0.5"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="1"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="2.5"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="5"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="10"} 4
nginx_http_response_time_seconds_hist_bucket{method="GET",request_uri="/metrics",status="200",le="+Inf"} 4
nginx_http_response_time_seconds_hist_sum{method="GET",request_uri="/metrics",status="200"} 0
nginx_http_response_time_seconds_hist_count{method="GET",request_uri="/metrics",status="200"} 4
# HELP nginx_parse_errors_total Total number of log file lines that could not be parsed
# TYPE nginx_parse_errors_total counter
nginx_parse_errors_total 0
# HELP prometheus_nginxlog_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus_nginxlog_exporter was built.
# TYPE prometheus_nginxlog_exporter_build_info gauge
prometheus_nginxlog_exporter_build_info{branch="v1.10.0",goversion="go1.18.6",revision="9274be256a47378a9930164023d16e1958e96292",version="1.10.0"} 1
        metrics_path: /metrics
        static_configs:
          - targets: ["10.223.64.23:9100","10.223.64.23:8080","10.223.64.23:4040"]

2. prometheus 재배포

helm upgrade --install prometheus . -f values-dev.yaml -n monitor