添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接
相关文章推荐
憨厚的充值卡  ·  How to get rid of ...·  2 天前    · 
想发财的大脸猫  ·  Kibana ...·  1 周前    · 
飘逸的莲藕  ·  Send data from ...·  1 周前    · 
愤怒的西瓜  ·  Running Logstash from ...·  4 周前    · 
瘦瘦的柚子  ·  Shift-Cmd=C not ...·  3 周前    · 
鬼畜的柚子  ·  Note·  5 月前    · 

After upgrading to ELK 8.1, I noticed that every event has the "event.original" field containing all of the log data. This is highly unwanted, how to prevent this field from being sent from Filebeat?
I tried doing it on Filebeat level using processors:

  processors:
  - drop_fields:
      fields: ["event.original"]

and via Logstash remove_field.
None of this worked, the field is still visible.

Are you using any filebeat module?

The field event.original is normally created by the ingest pipeline used by some of the filebeat modules when parsing the original message, it does not exist in your original event, so you won't be able to remove it in filebeat nor in logstash, you would need to check the ingest pipeline for the module that you are using and remove the field there.

- multiline: type: pattern pattern: '^\[[0-9]{2}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}' negate: True match: after name: iob1_int tags: ["iobint1-logs"] ignore_older: 24h - type: filestream enabled: true paths: - /path/to/server.log parsers: - multiline: type: pattern pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}' negate: True match: after name: iob1_int tags: ["iobint1-logs_server"] ignore_older: 24h processors: - drop_fields: fields: ["event.original","agent.type"] output.logstash: hosts: ["hostname:5044"] ssl.certificate_authorities: - /path/to/ca.crt monitoring: enabled: true cluster_uuid: 123456789 elasticsearch: hosts: ["https://hostname:9200"] ssl.certificate_authorities: ["path/to/crt"] username: XXX password: XXX

After filebeat, Logstash parses the logs and sends to Elastic. Do you mean that the field is added on Logstash level? How to delete it?

What is your logstash pipeline? Please share it.

Logstash normally won't add any field unless explicitly configured in the pipeline, but I'm not running version 8.X and there was some changes regarding the ecs fields, so I'm not sure if this is being added by logstash or not.

How you tried to remove it in Logstash?

Did you have something like this?

mutate {
    remove_field => ["[event][original]"]
I tried to delete the field like below:
remove_field => ["event.original"]
and this didn't work.
This solves the issue for me.

Ah yes, this is confusing some times.

In filebeat and Elasticsearch you work with nested fields using top.nested, like event.original, but in logstash you need to use [top][nested], so it should be [event][original].

Using event.original in Logstash would make it try to work with a field with that literal name, with the dot in the name.