I'm sending data to logstash through kafka python client. this is the object i send
{"user_id":"kafka12","service_name":"kafka12","activity_type":"kafka","error":{"message":"Invalid username","component_id":"Y456"},"additional_data":{"key":"build"}}
and on the logstash log i get this warn
]
could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"user_activity_log_missing_data", :routing=>nil}, {"@timestamp"=>2023-09-29T07:38:38.823781508Z, "document"=>{"additional_data"=>{"key"=>"build"}, "user_id"=>"kafka12", "service_name"=>"kafka12", "activity_type"=>"kafka", "error"=>{"component_id"=>"Y456", "message"=>"Invalid username"}}}], :response=>{"index"=>{"_index"=>"user_activity_log_missing_data", "_id"=>"alrg34oBFJ5BviDnE3nj", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:222] failed to parse field [document.error] of type [text] in document with id 'alrg34oBFJ5BviDnE3nj'. Preview of field's value: '{component_id=Y456, message=Invalid username}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:171"}}}}}
this warn does not occur when i send the same data with http request
I don't think so. This is in your LS log.
"error"=>{"type"=>"
document_parsing_exception
", "reason"=>"[1:222] failed to parse field [document.error] of type [text] in document with id 'alrg34oBFJ5BviDnE3nj'. Preview of field's value: '{component_id=Y456, message=Invalid username}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:171"}}}}}