📂Exporting Nginx Access Logs to an ELK Cluster

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

玩篮球的跑步机 · Common MIME types - ...· 15 小时前 ·

爱玩的豌豆 · Access Denied· 2 天前 ·

紧张的小熊猫 · Access Denied· 2 天前 ·

坏坏的凳子 · Access Denied· 2 天前 ·

含蓄的四季豆 · Mautic problem with ...· 2 天前 ·

纯真的毛衣 · How to print a int64 ...· 1 月前 ·

帅呆的电脑桌 · JSON.RESP - 亚马逊 ...· 3 月前 ·

威武的帽子 · SQL ...· 3 月前 ·

体贴的风衣 · matplotlib设置colorbar范围 ...· 4 月前 ·

眼睛小的乌冬面 · js为什么单线程实现异步编程 • ...· 5 月前 ·

The Wallarm user interface provides the ability to search for a number of different types of attack traffic within alert data. More information about the supported types of searches can be found in the Wallarm documentation .

However, the search functionality of the Wallarm UI does not provide full visibility into every type of potential attack or full details of a particular alert. If this level of visibility is necessary, then the access logs can be exported from Nginx into an ELK cluster where it is possible to perform full-text searches and other data analytics against them. This also enables this information to be fed into other systems that the organization may be using for threat detection or threat intelligence generation and analysis.

A couple of different options exist for exporting this data from Nginx. This article focuses on the approach of sending Nginx access logs to an ELK cluster. Alternatively, it is possible to use the Wallarm API to extract request information, as described in this blog.

How to Export Nginx Access Logs

Sending access logs from Nginx to an ELK cluster is a multi-stage process. In this article, we’ll focus on configuring Nginx to export the logs in the correct format and send them to a listening TCP/UDP port. The remaining steps of the process (configuring Logstash and Elasticsearch) are well-described on the Internet, and links are provided that contain more detail on how to accomplish these steps correctly.

Step 1: Configure JSON Access Log Output

The first step in setting up the export of Nginx access logs is configuring Nginx to send the data out in the desired format. For ease of use, we’ll be defining a JSON-based log format for the export.

The following shows the structure of a custom log format that combines some standard Nginx attributes, such as number of connection requests and response status, with some Wallarm-specific data, such as the Wallarm attack type. More information about the standard attributes available with Nginx is available in their documentation , and the Wallarm documentation describes the set of custom attributes that Wallarm defines.

<code>log_format json_log escape=json '{
"connection_serial_number":$connection,'
'"number_of_requests":$connection_requests,'
'"response_status":"$status",'
'"body_bytes_sent":$body_bytes_sent,'
'"content_type":"$content_type",'
'"host":"$host",'
'"host_name":"$hostname",'
'"http_name":"$http_name",'
'"https":"$https",'
'"proxy_protocol_addr":"$proxy_protocol_addr",'
'"proxy_protocol_port":"$proxy_protocol_port",'
'"query_string":"$query_string",'
'"client_address":"$remote_addr",'
'"http_ar_real_proto":"$http_ar_real_proto",'
'"http_ar_real_ip":"$http_ar_real_ip",'
'"http_ar_real_country":"$http_ar_real_country",'
'"http_x_real_ip":"$http_x_real_ip",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"http_config":"$http_config",'
'"client_port":"$remote_port",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"request_time":$request_time,'
'"request_id":"$request_id",'
'"request_length":$request_length,'
'"request_method":"$request_method",'
'"request_uri":"$request_uri",'
'"request_body":"$request_body",'
'"scheme":"$scheme",'
'"server_addr":"$server_addr",'
'"server_name":"$server_name",'
'"server_port":"$server_port",'
'"server_protocol":"$server_protocol",'
'"http_user_agent":"$http_user_agent",'
'"time_local":"$time_local",'
'"time_iso":"$time_iso8601",'
'"attack":"$wallarm_attack_type",'
'"route_uri": "$upstream_http_route_uri",'
'"route_cookie": "$upstream_http_route_cookie",'
'"url":"$scheme://$host$request_uri",'
'"uri":"$uri"}';</code>

One option is to use fluentd to perform the transfer. After saving the log file as a text file, a local copy of fluentd can be used to perform the transfer. For more information regarding this option, check out the fluentd documentation .

Alternatively, it is possible to use Nginx’s built-in syslog client to perform the log file export. However, this option requires the receiving Logstash instance to be configured to use the syslog plugin to receive input. If this is the case, use the following configuration to send the access logs via syslog:

access_log syslog:server=<ADDRESS_OR_DNS_OF_COLLECTOR>:<PORT>,tag=wallarm_access json_log;