Hi all. I’m using the
logstash-output-opensearch
. It’s working, but I can’t get it to use my log timestamp as
@timestamp
.
This is what I want it to use, direct from the log:
"ACTUAL_TIME": "2023-09-03T08:06:51.946Z"
This is my logstash.conf filter:
filter{
date {
match => [ "ACTUAL_TIME", "yyyy-MM-dd HH:mm:ss.SSSZ" ]
remove_field => [ "ACTUAL_TIME" ]
In discover, I see both fields, and they don’t match on the milliseconds:
Any help would be, like, really helpful!
Maybe somethign like this
filter {
mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
filter {
mutate { remove_field => [ "ACTUAL_TIME"]
Is it posible to show your full Logstash config.
Thank you for responding!
This is the full (anonymized) logstash.conf. Note “contextMap” now prepends ACTUAL TIME. The “_app” above was to rename it. But still, it’s not working.
input {
file {
path => "/spring/logs/myApp/ELK/myLog.log"
filter{
json{
source => "message"
date {
match => [ "contextMap.ACTUAL_TIME", "yyyy-MM-dd HH:mm:ss.SSSZ" ]
remove_field => [ "contextMap.ACTUAL_TIME" ]
}output {
opensearch {
hosts => "http://myServer:9200"
user => "XXXXX"
password => "XXXXX"
index => "my-app-%{[agent.version]}-%{+yyyy.MM.dd}"
ssl_certificate_verification => false
OK. This issue seems to be with inner JSON. I told it to just remove “host”, and that works.
We are sending the Log4j2 MDC map, which generates a section like this:
"contextMap": {
"ACTUAL_TIME": "2023-09-03T08:06:51.946Z",
Does anyone know of a way to remove just ACTUAL_TIME? I’ve tried all combinations I can think of.
Thanks!