link管理
链接快照平台
- 输入网页链接，自动生成快照
- 标签化管理网页链接
相关文章推荐
|
潇洒的课本 · Error 400 - Rejected ...· 1 月前 · |
|
|
谦和的蘑菇 · 中国科学院生物与化学交叉研究中心· 7 月前 · |
|
|
暴躁的四季豆 · [Wireless Router] How ...· 8 月前 · |
|
|
英姿勃勃的鸡蛋面 · warning: ...· 11 月前 · |
|
|
打篮球的荒野 · Search Results – ...· 11 月前 · |
|
|
坚强的葫芦 · 炮灰美人就该被疯批免费阅读全文 - ...· 1 年前 · |
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Version used: 1.9.3
Configuration: See below
Environment name and version: AWS EC2 instance
Server type and version: Windows
Operating System and version: Server 2016
Filters and plugins: in_tail, filter_lua, output_es
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your accountBug Report
Describe the bug
After updating Fluent Bit from 1.5.2 to 1.9.3, I'm seeing random crashes in production.
Your Environment
fluent-bit.conf
:
[SERVICE]
daemon false
flush 1
log_level debug
# log_file fluent-bit.log
parsers_file parsers.conf
http_server true
http_listen 0.0.0.0
http_port 2020
health_check true
[INPUT]
name tail
tag pharos.*
path C:\PharosSystems\Logs\Json\PharosMpsLog.txt
db C:\PharosSystems\Logs\Json\FluentBit.db
parser json
buffer_chunk_size 64kb
buffer_max_size 64kb
mem_buf_limit 16mb
skip_long_lines true
refresh_interval 10
[FILTER]
name lua
match *
script filters.lua
call transform
[OUTPUT]
name es
match *
host logs
port 443
logstash_format true
replace_dots true
retry_limit 5
trace_error true
buffer_size 64kb
aws_auth true
tls true
filters.lua:
function transform(tag, timestamp, record)
if record["Payload"] then
record["Service"] = record["Payload"]["processName"]
if not record["Message"] and record["Payload"]["message"] then
record["Message"], record["Payload"]["message"] = record["Payload"]["message"], nil
if record["Properties"] and record["Properties"]["StatusCode"] then
record["Properties"]["StatusCode"] = tostring(record["Properties"]["StatusCode"])
return 1, timestamp, record
Additional context
Dump file captured with Sysinternals ProcDump using procdump64.exe -ma -e fluent-bit.exe:
fluent-bit.exe_220705_151325.zip
Looking around with WinDbg Preview:
Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_220705_151325\fluent-bit.exe_220705_151325.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION'
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK D:\SymbolsTemp
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\SymbolsTemp
Executable search path is:
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 10.0.14393.5125 (rs1_release.220429-1732)
Machine Name:
Debug session time: Wed Jul 6 03:13:25.000 2022 (UTC + 12:00)
System Uptime: 2 days 9:41:01.606
Process Uptime: 0 days 13:52:32.000
..........................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(16ec.16dc): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!RtlpAllocateHeap+0x12da:
00007ffc`ed175fda 488b08 mov rcx,qword ptr [rax] ds:2265646f`63704f22=????????????????
0:005> .kframes 0n100
Default stack trace depth is 0n100 frames
0:005> .prefer_dml 1
DML versions of commands on by default
0:005> .reload /f
..........................................
0:005> lm
start end module name
00007ff6`c0230000 00007ff6`c0a5a000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
00007ffc`dff50000 00007ffc`dff66000 NapiNSP (pdb symbols) d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffc`dff70000 00007ffc`dff7e000 winrnr (pdb symbols) d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffc`e2490000 00007ffc`e249a000 rasadhlp (pdb symbols) d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffc`e61b0000 00007ffc`e61ca000 dhcpcsvc (pdb symbols) d:\symbols\dhcpcsvc.pdb\896F47CEEF60427FBE41C39E04C01D9E1\dhcpcsvc.pdb
00007ffc`e6210000 00007ffc`e6226000 dhcpcsvc6 (pdb symbols) d:\symbols\dhcpcsvc6.pdb\BC7D98B165E241B7856EA8A03648BD371\dhcpcsvc6.pdb
00007ffc`e6910000 00007ffc`e6928000 nlaapi (pdb symbols) d:\symbols\nlaapi.pdb\55CC67AE0E7A423BB758846D361972001\nlaapi.pdb
00007ffc`e7180000 00007ffc`e71e1000 wevtapi (pdb symbols) d:\symbols\wevtapi.pdb\2F5835A7F06F43EE86ADF99297D785921\wevtapi.pdb
00007ffc`e8a00000 00007ffc`e8a33000 rsaenh (pdb symbols) d:\symbols\rsaenh.pdb\95B56C8731B44E9AA2858411CD4281E21\rsaenh.pdb
00007ffc`e8b80000 00007ffc`e8bb8000 IPHLPAPI (pdb symbols) d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffc`e8bc0000 00007ffc`e8c62000 dnsapi (pdb symbols) d:\symbols\dnsapi.pdb\5C6526F84E6C4878B0CCB26BA6E564A41\dnsapi.pdb
00007ffc`e8e50000 00007ffc`e8ead000 mswsock (pdb symbols) d:\symbols\mswsock.pdb\DAA810367BC04CDD8EBFE12DA2EB501C1\mswsock.pdb
00007ffc`e9000000 00007ffc`e9017000 cryptsp (pdb symbols) d:\symbols\cryptsp.pdb\677162669D6740539787653CD28E727F1\cryptsp.pdb
00007ffc`e9020000 00007ffc`e902b000 CRYPTBASE (pdb symbols) d:\symbols\cryptbase.pdb\A4945E336F8347DDA43A2E9E5FC0ED541\cryptbase.pdb
00007ffc`e94e0000 00007ffc`e950b000 bcrypt (pdb symbols) d:\symbols\bcrypt.pdb\C66626E3ED344020A87327CB85BC26D01\bcrypt.pdb
00007ffc`e9590000 00007ffc`e95a4000 profapi (pdb symbols) d:\symbols\profapi.pdb\CAC4E3CC2E9B4973B17D131D0EFC8ACE1\profapi.pdb
00007ffc`e95b0000 00007ffc`e95bf000 kernel_appcore (pdb symbols) d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffc`e95c0000 00007ffc`e95d0000 msasn1 (pdb symbols) d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffc`e95d0000 00007ffc`e961c000 powrprof (pdb symbols) d:\symbols\powrprof.pdb\74B38765BC2D4D1B9E53834273CE9ABF1\powrprof.pdb
00007ffc`e9620000 00007ffc`e9813000 crypt32 (pdb symbols) d:\symbols\crypt32.pdb\37224C862FEF48A78732D55DA43319AF1\crypt32.pdb
00007ffc`e98d0000 00007ffc`e9978000 SHCore (pdb symbols) d:\symbols\shcore.pdb\DD5B4BDC74714727A71E4F62D53DB41B1\shcore.pdb
00007ffc`e9980000 00007ffc`e999e000 win32u (pdb symbols) d:\symbols\win32u.pdb\CE777C7E22FC4E43A47A1D6EF5E1AF031\win32u.pdb
00007ffc`e99a0000 00007ffc`e99e2000 cfgmgr32 (pdb symbols) d:\symbols\cfgmgr32.pdb\0C85585E81E54222A3065E20B6C913D41\cfgmgr32.pdb
00007ffc`e99f0000 00007ffc`e9a5c000 bcryptPrimitives (pdb symbols) d:\symbols\bcryptprimitives.pdb\66FD402F0E2C4120A9CE02CE189B48E91\bcryptprimitives.pdb
00007ffc`e9a60000 00007ffc`ea133000 windows_storage (pdb symbols) d:\symbols\Windows.Storage.pdb\815F10C0A2804CF49A9BEAA4152EBB5F1\Windows.Storage.pdb
00007ffc`ea140000 00007ffc`ea2c4000 gdi32full (pdb symbols) d:\symbols\gdi32full.pdb\747C73435FBD4F1784BB0FB4F88D54961\gdi32full.pdb
00007ffc`ea2d0000 00007ffc`ea3c4000 ucrtbase (pdb symbols) d:\symbols\ucrtbase.pdb\FB16A8A30FFD4D868EA1F6402B3143F81\ucrtbase.pdb
00007ffc`ea4d0000 00007ffc`ea6ee000 KERNELBASE (pdb symbols) d:\symbols\kernelbase.pdb\162A8BD56C304693A8AB2468E50C5F6A1\kernelbase.pdb
00007ffc`ea890000 00007ffc`ebd96000 shell32 (pdb symbols) d:\symbols\shell32.pdb\C1EE097F73C64A229FFE4097566613111\shell32.pdb
00007ffc`ebe40000 00007ffc`ebf5d000 rpcrt4 (pdb symbols) d:\symbols\rpcrt4.pdb\C4E0FD0447934EC38FCB2D46A2A4BED21\rpcrt4.pdb
00007ffc`ebf60000 00007ffc`ec006000 advapi32 (pdb symbols) d:\symbols\advapi32.pdb\0EC476B28EA04617B741EDF5C605AEBC1\advapi32.pdb
00007ffc`ec070000 00007ffc`ec10e000 msvcrt (pdb symbols) d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffc`ec120000 00007ffc`ec17b000 sechost (pdb symbols) d:\symbols\sechost.pdb\F78401B62F914F38AEF989B6261E214B1\sechost.pdb
00007ffc`ec180000 00007ffc`ec22d000 kernel32 (pdb symbols) d:\symbols\kernel32.pdb\5098B486BCBA4FDFBE1B74CE5C48DC341\kernel32.pdb
00007ffc`ec830000 00007ffc`ec969000 ole32 (private pdb symbols) d:\symbols\ole32.pdb\0E805AFF195F4448B05AE219EAB08F071\ole32.pdb
00007ffc`ec970000 00007ffc`ec9da000 ws2_32 (pdb symbols) d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffc`ec9e0000 00007ffc`eca14000 gdi32 (pdb symbols) d:\symbols\gdi32.pdb\44AE0735CFE84F5D847786D34C02BC401\gdi32.pdb
00007ffc`ecae0000 00007ffc`ecb32000 shlwapi (pdb symbols) d:\symbols\shlwapi.pdb\BE622E17968E4B1CBFF14120667BC4C61\shlwapi.pdb
00007ffc`ecb40000 00007ffc`ecb48000 nsi (pdb symbols) d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffc`ecd10000 00007ffc`ecfd6000 combase (private pdb symbols) d:\symbols\combase.pdb\7F9766E264B5460DB6B97F055B10AB241\combase.pdb
00007ffc`ecfe0000 00007ffc`ed146000 user32 (pdb symbols) d:\symbols\user32.pdb\BB7C068C642C4ED2AA5E9DD14A009B311\user32.pdb
00007ffc`ed150000 00007ffc`ed31f000 ntdll (pdb symbols) d:\symbols\ntdll.pdb\13B5E8DAE6F540E49137EE219A7ED12D1\ntdll.pdb
0:005> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
DEBUG_FLR_EXCEPTION_CODE(c0000374) and the ".exr -1" ExceptionCode(c0000005) don't match
KEY_VALUES_STRING: 1
Key : AV.Fault
Value: Read
Key : Analysis.CPU.mSec
Value: 436
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 428
Key : Analysis.Init.CPU.mSec
Value: 452
Key : Analysis.Init.Elapsed.mSec
Value: 30315
Key : Analysis.Memory.CommitPeak.Mb
Value: 184
Key : Timeline.OS.Boot.DeltaSec
Value: 207661
Key : Timeline.Process.Start.DeltaSec
Value: 49952
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Timestamp
Value: 2022-04-29T17:32:00Z
Key : WER.OS.Version
Value: 10.0.14393.5125
Key : WER.Process.Version
Value: 1.9.3.0
FILE_IN_CAB: fluent-bit.exe_220705_151325.dmp
COMMENT:
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: (.ecxr)
rax=2265646f63704f22 rbx=0000026e07d80000 rcx=0000000076654c22
rdx=0000000000040000 rsi=0000026e08755da0 rdi=0000000000000301
rip=00007ffced175fda rsp=000000b2b2dfec70 rbp=000000b2b2dff029
r8=0000026e07d860b0 r9=000000000000001b r10=0000000000000000
r11=0000026e086a0080 r12=0000026e08755db0 r13=0000026e07d80150
r14=0000026e08a033a0 r15=0000000000000001
iopl=0 nv up ei ng nz ac pe cy
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010291
ntdll!RtlpAllocateHeap+0x12da:
00007ffc`ed175fda 488b08 mov rcx,qword ptr [rax] ds:2265646f`63704f22=????????????????
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffced175fda (ntdll!RtlpAllocateHeap+0x00000000000012da)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff
PROCESS_NAME: fluent-bit.exe
READ_ADDRESS: ffffffffffffffff
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: ffffffffffffffff
ADDITIONAL_DEBUG_TEXT: Enable Pageheap/AutoVerifer ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
FAULTING_THREAD: 000016dc
STACK_TEXT:
00000000`00000000 00000000`00000000 heap_corruption!fluent-bit.exe+0x0
STACK_COMMAND: ** Pseudo Context ** ManagedPseudo ** Value: ffffffff ** ; kb
SYMBOL_NAME: heap_corruption!fluent-bit.exe
MODULE_NAME: heap_corruption
IMAGE_NAME: heap_corruption
FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!fluent-bit.exe
OS_VERSION: 10.0.14393.5125
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {e4b4a773-236c-d32e-a7f9-5bb698111271}
Followup: MachineOwner
---------
0:005> kb
# RetAddr : Args to Child : Call Site
00 00007ffc`ed173ba5 : 0000026e`07d80000 00000000`00000002 00000000`00005000 00000000`00005010 : ntdll!RtlpAllocateHeap+0x12da
01 00007ffc`ed171090 : 0000026e`07d80000 00000000`00005000 0000026e`08a40fc0 00000000`00000000 : ntdll!RtlpAllocateHeapInternal+0x1115
02 00007ffc`ed170b4f : 00000000`00000000 00000000`00000000 00000000`00000000 0000026e`07b40000 : ntdll!RtlpReAllocateHeap+0x220
03 00007ffc`ed170a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00007ff6`c072902c : ntdll!RtlpReAllocateHeapInternal+0xdf
04 00007ff6`c07e1b53 : 00000000`00000300 000000b2`b2dff6c0 000000b2`b2dff7b0 000000b2`b2dff6c0 : ntdll!RtlReAllocateHeap+0x31
05 00007ff6`c02df2fb : 000000b2`b2dff550 000000b2`b2dff599 0000026e`08882f25 00000000`00002134 : fluent_bit!_realloc_base+0x73 [minkernel\crts\ucrt\src\appcrt\heap\realloc_base.cpp @ 46]
06 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!flb_realloc+0x17 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_mem.h @ 94]
07 00007ff6`c02e1682 : 0000026e`08882f25 000000b2`b2dff599 000000b2`b2dff680 00000000`00000000 : fluent_bit!flb_json_tokenise+0x5b [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 62]
08 00007ff6`c02df93d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!pack_json_to_msgpack+0xe2 [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 246]
09 00007ff6`c034f8a9 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!flb_pack_json+0x1d [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 280]
0a 00007ff6`c034f5cb : 0000026e`085a7990 00000000`00000000 00000000`0000e360 00007ff6`c0833ea0 : fluent_bit!elasticsearch_error_check+0x49 [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 662]
0b 00007ff6`c0306f61 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0000e360 : fluent_bit!cb_es_flush+0x4ab [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 884]
0c 00007ff6`c0765c1e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xc1 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_output.h @ 522]
0d 00007ffc`ea53e951 : 00007ff6`c0306ea0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe [C:\projects\fluent-bit-2e87g\lib\monkey\deps\flb_libco\fiber.c @ 23]
0e 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
0:005> dx Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[5].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[5].SwitchTo()
@rdi void * block = 0x0000026e`08a40fd0
@rbx unsigned int64 size = 0x5000
@rax void * new_block = 0x2265646f`63704f22
0:005> dx Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[6].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[6].SwitchTo()
0:005> dx Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[7].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[5868].Threads[5852].Stack.Frames[7].SwitchTo()
@rsi char * js = 0x0000026e`08882f25 "{"took":332,"errors":false,"items":[{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Ty_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041226,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"UC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040758,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"US_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040759,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Ui_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038876,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Uy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041227,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"VC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036203,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"VS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038877,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Vi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040760,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Vy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038878,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"WC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040761,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"WS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039421,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Wi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040762,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Wy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036204,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"XC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041228,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"XS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039422,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Xi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041229,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Xy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040763,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"YC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039423,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"YS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036205,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Yi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038879,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Yy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041230,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"ZC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036206,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"ZS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036207,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Zi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039424,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"Zy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039425,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"aC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039426,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"aS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041231,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"ai_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041232,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"ay_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3039427,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"bC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041233,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"bS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040764,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"bi_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036208,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"by_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040765,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"cC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3036209,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"cS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3040766,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"ci_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038880,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"cy_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":...2,"failed":0},"_seq_no":3041234,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"dC_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3041235,"_primary_term":1,"status":201}},{"create":{"_index":"logstash-2022.07.05","_type":"_doc","_id":"dS_szoEBF-nUXHgNWXaO","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3038881,"_primary_term":1,"status":201}}]}"
@rdi unsigned int64 len = 0x2134
@rbx struct flb_pack_state * state = 0x000000b2`b2dff550
<unavailable> int ret = <value unavailable>
<unavailable> unsigned int64 new_size = <value unavailable>
@rax void * tmp = 0x2265646f`63704f22
0:005> dx -r1 ((fluent_bit!flb_pack_state *)0xb2b2dff550)
((fluent_bit!flb_pack_state *)0xb2b2dff550) : 0xb2b2dff550 [Type: flb_pack_state *]
[+0x000] multiple : 0 [Type: int]
[+0x004] tokens_count : 0 [Type: int]
[+0x008] tokens_size : 768 [Type: int]
[+0x00c] last_byte : 0 [Type: int]
[+0x010] tokens : 0x26e08a40fd0 [Type: jsmntok *]
[+0x018] parser [Type: jsmn_parser]
[+0x028] buf_data : 0x26e087b64c0 : "2022-07-05T15:13:19.0028054Z" [Type: char *]
[+0x030] buf_size : 0x1400 [Type: unsigned __int64]
[+0x038] buf_len : 0x0 [Type: unsigned __int64]
0:005> dx -r1 tmp
tmp : 0x2265646f63704f22 [Type: void *]
0:005> dd 0x2265646f63704f22
2265646f`63704f22 ???????? ???????? ???????? ????????
2265646f`63704f32 ???????? ???????? ???????? ????????
2265646f`63704f42 ???????? ???????? ???????? ????????
2265646f`63704f52 ???????? ???????? ???????? ????????
2265646f`63704f62 ???????? ???????? ???????? ????????
2265646f`63704f72 ???????? ???????? ???????? ????????
2265646f`63704f82 ???????? ???????? ???????? ????????
2265646f`63704f92 ???????? ???????? ???????? ????????
PTAL @fujimotos 🙏
After updating to fluent bit 1.9.5 there were 8 crashes overnight across 2 different instances, and I captured 2 crash dumps:
fluent-bit.exe_220707_130801.zip
Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_220707_130801\fluent-bit.exe_220707_130801.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000374'
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK D:\SymbolsTemp
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\SymbolsTemp
Executable search path is:
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 10.0.14393.5125 (rs1_release.220429-1732)
Machine Name:
Debug session time: Fri Jul 8 01:08:02.000 2022 (UTC + 12:00)
System Uptime: 0 days 10:50:30.634
Process Uptime: 0 days 10:45:50.000
...........................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(c6c.d1c): Unknown exception - code c0000374 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!RtlReportCriticalFailure+0x97:
00007ffc`a3e07143 eb00 jmp ntdll!RtlReportCriticalFailure+0x99 (00007ffc`a3e07145)
0:005> .kframes 0n100
Default stack trace depth is 0n100 frames
0:005> .prefer_dml 1
DML versions of commands on by default
0:005> .reload /f
.....................
Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.
......................
0:005> lm
start end module name
00007ff7`aedd0000 00007ff7`af5fe000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
00007ffc`96b10000 00007ffc`96b26000 NapiNSP (pdb symbols) d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffc`96b30000 00007ffc`96b3e000 winrnr (pdb symbols) d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffc`98b60000 00007ffc`98b6a000 rasadhlp (pdb symbols) d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffc`9cd00000 00007ffc`9cd1a000 dhcpcsvc (pdb symbols) d:\symbols\dhcpcsvc.pdb\896F47CEEF60427FBE41C39E04C01D9E1\dhcpcsvc.pdb
00007ffc`9cd60000 00007ffc`9cd76000 dhcpcsvc6 (pdb symbols) d:\symbols\dhcpcsvc6.pdb\BC7D98B165E241B7856EA8A03648BD371\dhcpcsvc6.pdb
00007ffc`9d9a0000 00007ffc`9da01000 wevtapi (pdb symbols) d:\symbols\wevtapi.pdb\2F5835A7F06F43EE86ADF99297D785921\wevtapi.pdb
00007ffc`9da10000 00007ffc`9da8a000 apphelp (pdb symbols) d:\symbols\apphelp.pdb\A9C49D33ADEA4891B6CE6414A6BAAAA71\apphelp.pdb
00007ffc`9dfb0000 00007ffc`9dfc8000 nlaapi (pdb symbols) d:\symbols\nlaapi.pdb\55CC67AE0E7A423BB758846D361972001\nlaapi.pdb
00007ffc`9f5c0000 00007ffc`9f5f3000 rsaenh (pdb symbols) d:\symbols\rsaenh.pdb\95B56C8731B44E9AA2858411CD4281E21\rsaenh.pdb
00007ffc`9f740000 00007ffc`9f778000 IPHLPAPI (pdb symbols) d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffc`9f780000 00007ffc`9f822000 dnsapi (pdb symbols) d:\symbols\dnsapi.pdb\5C6526F84E6C4878B0CCB26BA6E564A41\dnsapi.pdb
00007ffc`9fa10000 00007ffc`9fa6d000 mswsock (pdb symbols) d:\symbols\mswsock.pdb\DAA810367BC04CDD8EBFE12DA2EB501C1\mswsock.pdb
00007ffc`9fbc0000 00007ffc`9fbd7000 cryptsp (pdb symbols) d:\symbols\cryptsp.pdb\677162669D6740539787653CD28E727F1\cryptsp.pdb
00007ffc`9fbe0000 00007ffc`9fbeb000 CRYPTBASE (pdb symbols) d:\symbols\cryptbase.pdb\A4945E336F8347DDA43A2E9E5FC0ED541\cryptbase.pdb
00007ffc`a0090000 00007ffc`a00bb000 bcrypt (pdb symbols) d:\symbols\bcrypt.pdb\C66626E3ED344020A87327CB85BC26D01\bcrypt.pdb
00007ffc`a0150000 00007ffc`a015f000 kernel_appcore (pdb symbols) d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffc`a0160000 00007ffc`a0170000 msasn1 (pdb symbols) d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffc`a0170000 00007ffc`a01bc000 powrprof (pdb symbols) d:\symbols\powrprof.pdb\74B38765BC2D4D1B9E53834273CE9ABF1\powrprof.pdb
00007ffc`a01c0000 00007ffc`a01d4000 profapi (pdb symbols) d:\symbols\profapi.pdb\CAC4E3CC2E9B4973B17D131D0EFC8ACE1\profapi.pdb
00007ffc`a01e0000 00007ffc`a03fe000 KERNELBASE (pdb symbols) d:\symbols\kernelbase.pdb\162A8BD56C304693A8AB2468E50C5F6A1\kernelbase.pdb
00007ffc`a0400000 00007ffc`a0ad3000 windows_storage (pdb symbols) d:\symbols\Windows.Storage.pdb\815F10C0A2804CF49A9BEAA4152EBB5F1\Windows.Storage.pdb
00007ffc`a0ae0000 00007ffc`a0b4c000 bcryptPrimitives (pdb symbols) d:\symbols\bcryptprimitives.pdb\66FD402F0E2C4120A9CE02CE189B48E91\bcryptprimitives.pdb
00007ffc`a0c60000 00007ffc`a0c7e000 win32u (pdb symbols) d:\symbols\win32u.pdb\CE777C7E22FC4E43A47A1D6EF5E1AF031\win32u.pdb
00007ffc`a0d20000 00007ffc`a0dc8000 SHCore (pdb symbols) d:\symbols\shcore.pdb\DD5B4BDC74714727A71E4F62D53DB41B1\shcore.pdb
00007ffc`a0dd0000 00007ffc`a0f54000 gdi32full (pdb symbols) d:\symbols\gdi32full.pdb\747C73435FBD4F1784BB0FB4F88D54961\gdi32full.pdb
00007ffc`a0f60000 00007ffc`a0fa2000 cfgmgr32 (pdb symbols) d:\symbols\cfgmgr32.pdb\0C85585E81E54222A3065E20B6C913D41\cfgmgr32.pdb
00007ffc`a0fb0000 00007ffc`a10a4000 ucrtbase (pdb symbols) d:\symbols\ucrtbase.pdb\FB16A8A30FFD4D868EA1F6402B3143F81\ucrtbase.pdb
00007ffc`a10b0000 00007ffc`a12a3000 crypt32 (pdb symbols) d:\symbols\crypt32.pdb\37224C862FEF48A78732D55DA43319AF1\crypt32.pdb
00007ffc`a1390000 00007ffc`a2896000 shell32 (pdb symbols) d:\symbols\shell32.pdb\C1EE097F73C64A229FFE4097566613111\shell32.pdb
00007ffc`a2db0000 00007ffc`a2de4000 gdi32 (pdb symbols) d:\symbols\gdi32.pdb\44AE0735CFE84F5D847786D34C02BC401\gdi32.pdb
00007ffc`a2df0000 00007ffc`a2df8000 nsi (pdb symbols) d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffc`a2e00000 00007ffc`a2ead000 kernel32 (pdb symbols) d:\symbols\kernel32.pdb\5098B486BCBA4FDFBE1B74CE5C48DC341\kernel32.pdb
00007ffc`a2eb0000 00007ffc`a3176000 combase (private pdb symbols) d:\symbols\combase.pdb\7F9766E264B5460DB6B97F055B10AB241\combase.pdb
00007ffc`a3210000 00007ffc`a3262000 shlwapi (pdb symbols) d:\symbols\shlwapi.pdb\BE622E17968E4B1CBFF14120667BC4C61\shlwapi.pdb
00007ffc`a3270000 00007ffc`a330e000 msvcrt (pdb symbols) d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffc`a3310000 00007ffc`a336b000 sechost (pdb symbols) d:\symbols\sechost.pdb\F78401B62F914F38AEF989B6261E214B1\sechost.pdb
00007ffc`a37b0000 00007ffc`a381a000 ws2_32 (pdb symbols) d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffc`a3890000 00007ffc`a39f6000 user32 (pdb symbols) d:\symbols\user32.pdb\BB7C068C642C4ED2AA5E9DD14A009B311\user32.pdb
00007ffc`a3a00000 00007ffc`a3aa6000 advapi32 (pdb symbols) d:\symbols\advapi32.pdb\0EC476B28EA04617B741EDF5C605AEBC1\advapi32.pdb
00007ffc`a3ab0000 00007ffc`a3bcd000 rpcrt4 (pdb symbols) d:\symbols\rpcrt4.pdb\C4E0FD0447934EC38FCB2D46A2A4BED21\rpcrt4.pdb
00007ffc`a3bd0000 00007ffc`a3d09000 ole32 (private pdb symbols) d:\symbols\ole32.pdb\0E805AFF195F4448B05AE219EAB08F071\ole32.pdb
00007ffc`a3d10000 00007ffc`a3edf000 ntdll (pdb symbols) d:\symbols\ntdll.pdb\13B5E8DAE6F540E49137EE219A7ED12D1\ntdll.pdb
0:005> lmDvmfluent_bit
Browse full module list
start end module name
00007ff7`aedd0000 00007ff7`af5fe000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
Loaded symbol image file: fluent-bit.exe
Image path: C:\PharosSystems\Tools\FluentBit\bin\fluent-bit.exe
Image name: fluent-bit.exe
Browse all global symbols functions data
Timestamp: Fri Jun 17 23:01:40 2022 (62AC5F14)
CheckSum: 00825963
ImageSize: 0082E000
File version: 1.9.5.0
Product version: 1.9.5.0
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Calyptia Inc.
ProductName: Fluent Bit - Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
InternalName: fluent-bit
ProductVersion: 1.9.5.0
FileVersion: 1.9.5.0
FileDescription: Compiled with MSVC 19.29.30145.0
All rights reserved.
0:005> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 468
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 475
Key : Analysis.Init.CPU.mSec
Value: 452
Key : Analysis.Init.Elapsed.mSec
Value: 113513
Key : Analysis.Memory.CommitPeak.Mb
Value: 186
Key : Timeline.OS.Boot.DeltaSec
Value: 39030
Key : Timeline.Process.Start.DeltaSec
Value: 38750
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Timestamp
Value: 2022-04-29T17:32:00Z
Key : WER.OS.Version
Value: 10.0.14393.5125
Key : WER.Process.Version
Value: 1.9.5.0
FILE_IN_CAB: fluent-bit.exe_220707_130801.dmp
COMMENT:
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000374
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: (.ecxr)
rax=0000000000000000 rbx=000000d9fd3fe2f0 rcx=000000d9fd3fe8c8
rdx=0000000000000008 rsi=000000d9fd3fde00 rdi=000000d9fd3fde00
rip=00007ffca3e07143 rsp=000000d9fd3febf0 rbp=000000000000000d
r8=fffffff800000021 r9=0000000000000001 r10=0000000000000000
r11=000000d9fd3fe950 r12=0000000000006400 r13=0000000000000001
r14=0000029c23fbb401 r15=0000029c24487820
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000204
ntdll!RtlReportCriticalFailure+0x97:
00007ffc`a3e07143 eb00 jmp ntdll!RtlReportCriticalFailure+0x99 (00007ffc`a3e07145)
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffca3e07143 (ntdll!RtlReportCriticalFailure+0x0000000000000097)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00007ffca3e5d6b0
PROCESS_NAME: fluent-bit.exe
ERROR_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.
EXCEPTION_CODE_STR: c0000374
EXCEPTION_PARAMETER1: 00007ffca3e5d6b0
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
FAULTING_THREAD: ffffffff
STACK_TEXT:
00000000`00000000 00000000`00000000 heap_corruption!fluent-bit.exe+0x0
STACK_COMMAND: !heap ; ** Pseudo Context ** ManagedPseudo ** Value: ffffffff ** ; kb
SYMBOL_NAME: heap_corruption!fluent-bit.exe
MODULE_NAME: heap_corruption
IMAGE_NAME: heap_corruption
FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000374_heap_corruption!fluent-bit.exe
OS_VERSION: 10.0.14393.5125
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {6fea311a-173d-fab0-9a0e-1adeaccaa2de}
Followup: MachineOwner
---------
0:005> kb
# RetAddr : Args to Child : Call Site
00 00007ffc`a3e07ae2 : 00000000`00000000 00007ffc`a3e5d6b0 0000029c`23430000 0000029c`23fbb420 : ntdll!RtlReportCriticalFailure+0x97
01 00007ffc`a3db574a : 0000029c`23430000 00000000`0000000d 0000029c`23fbb420 00000000`00000000 : ntdll!RtlpHeapHandleError+0x12
02 00007ffc`a3dc6d9c : 0000029c`23439bb0 00000000`00000000 00000000`02000002 00000000`00000000 : ntdll!RtlpLogHeapFailure+0x96
03 00007ffc`a3d595b8 : 0000029c`23430000 00000000`00000000 0000029c`23f49c90 0000029c`23f4eca0 : ntdll!RtlpHeapRemoveListEntry+0x8e710
04 00007ffc`a3d56956 : 0000029c`23430000 0000029c`23f49c90 0000029c`23f49c90 0000029c`23f49c90 : ntdll!RtlpFreeHeap+0x828
05 00007ffc`a3d31dca : 0000029c`23f49c90 00000000`00000002 0000029c`23f49ca0 00000000`00000000 : ntdll!RtlFreeHeap+0x846
06 00007ffc`a3d30b4f : 00000000`00000000 00000000`00000000 00000000`00000000 0000029c`233d0000 : ntdll!RtlpReAllocateHeap+0xf5a
07 00007ffc`a3d30a51 : 00000000`00000000 00000000`00000000 00000000`00000000 00007ff7`af2cc0dc : ntdll!RtlpReAllocateHeapInternal+0xdf
08 00007ff7`af384ac3 : 00000000`00000400 000000d9`fd3ff660 000000d9`fd3ff750 000000d9`fd3ff660 : ntdll!RtlReAllocateHeap+0x31
09 00007ff7`aee7f30b : 000000d9`fd3ff4f0 000000d9`fd3ff539 0000029c`24330165 00000000`0000220c : fluent_bit!_realloc_base+0x73 [minkernel\crts\ucrt\src\appcrt\heap\realloc_base.cpp @ 46]
0a (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!flb_realloc+0x17 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_mem.h @ 94]
0b 00007ff7`aee81692 : 0000029c`24330165 000000d9`fd3ff539 000000d9`fd3ff620 00000000`00000000 : fluent_bit!flb_json_tokenise+0x5b [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 62]
0c 00007ff7`aee7f94d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!pack_json_to_msgpack+0xe2 [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 246]
0d 00007ff7`aeef0cf9 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!flb_pack_json+0x1d [C:\projects\fluent-bit-2e87g\src\flb_pack.c @ 280]
0e 00007ff7`aeef0a1b : 0000029c`23db8ee0 00000000`00000000 00000000`0000b702 00007ff7`af3d6f40 : fluent_bit!elasticsearch_error_check+0x49 [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 662]
0f 00007ff7`aeea7ac1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0000b702 : fluent_bit!cb_es_flush+0x4ab [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 884]
10 00007ff7`af308b8e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xc1 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_output.h @ 522]
11 00007ffc`a024e951 : 00007ff7`aeea7a00 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe [C:\projects\fluent-bit-2e87g\lib\monkey\deps\flb_libco\fiber.c @ 23]
12 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
fluent-bit.exe_220707_132410.zip
Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_220707_132410\fluent-bit.exe_220707_132410.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000374'
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK D:\SymbolsTemp
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\SymbolsTemp
Executable search path is:
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 10.0.14393.5125 (rs1_release.220429-1732)
Machine Name:
Debug session time: Fri Jul 8 01:24:10.000 2022 (UTC + 12:00)
System Uptime: 0 days 10:47:54.053
Process Uptime: 0 days 10:45:24.000
...........................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(c44.3b8): Unknown exception - code c0000374 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!RtlReportCriticalFailure+0x97:
00007ffb`487d7143 eb00 jmp ntdll!RtlReportCriticalFailure+0x99 (00007ffb`487d7145)
0:005> .kframes 0n100
Default stack trace depth is 0n100 frames
0:005> .prefer_dml 1
DML versions of commands on by default
0:005> .reload /f
...........................................
0:005> lm
start end module name
00007ff6`a9230000 00007ff6`a9a5e000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
00007ffb`3b930000 00007ffb`3b946000 NapiNSP (pdb symbols) d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffb`3b950000 00007ffb`3b95e000 winrnr (pdb symbols) d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffb`40870000 00007ffb`4087a000 rasadhlp (pdb symbols) d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffb`417f0000 00007ffb`4180a000 dhcpcsvc (pdb symbols) d:\symbols\dhcpcsvc.pdb\896F47CEEF60427FBE41C39E04C01D9E1\dhcpcsvc.pdb
00007ffb`41830000 00007ffb`41846000 dhcpcsvc6 (pdb symbols) d:\symbols\dhcpcsvc6.pdb\BC7D98B165E241B7856EA8A03648BD371\dhcpcsvc6.pdb
00007ffb`41b70000 00007ffb`41bd1000 wevtapi (pdb symbols) d:\symbols\wevtapi.pdb\2F5835A7F06F43EE86ADF99297D785921\wevtapi.pdb
00007ffb`427f0000 00007ffb`4286a000 apphelp (pdb symbols) d:\symbols\apphelp.pdb\A9C49D33ADEA4891B6CE6414A6BAAAA71\apphelp.pdb
00007ffb`42cd0000 00007ffb`42ce8000 nlaapi (pdb symbols) d:\symbols\nlaapi.pdb\55CC67AE0E7A423BB758846D361972001\nlaapi.pdb
00007ffb`43f90000 00007ffb`43fc3000 rsaenh (pdb symbols) d:\symbols\rsaenh.pdb\95B56C8731B44E9AA2858411CD4281E21\rsaenh.pdb
00007ffb`44110000 00007ffb`44148000 IPHLPAPI (pdb symbols) d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffb`44150000 00007ffb`441f2000 dnsapi (pdb symbols) d:\symbols\dnsapi.pdb\5C6526F84E6C4878B0CCB26BA6E564A41\dnsapi.pdb
00007ffb`443e0000 00007ffb`4443d000 mswsock (pdb symbols) d:\symbols\mswsock.pdb\DAA810367BC04CDD8EBFE12DA2EB501C1\mswsock.pdb
00007ffb`44590000 00007ffb`445a7000 cryptsp (pdb symbols) d:\symbols\cryptsp.pdb\677162669D6740539787653CD28E727F1\cryptsp.pdb
00007ffb`445b0000 00007ffb`445bb000 CRYPTBASE (pdb symbols) d:\symbols\cryptbase.pdb\A4945E336F8347DDA43A2E9E5FC0ED541\cryptbase.pdb
00007ffb`44a70000 00007ffb`44a9b000 bcrypt (pdb symbols) d:\symbols\bcrypt.pdb\C66626E3ED344020A87327CB85BC26D01\bcrypt.pdb
00007ffb`44b20000 00007ffb`44b34000 profapi (pdb symbols) d:\symbols\profapi.pdb\CAC4E3CC2E9B4973B17D131D0EFC8ACE1\profapi.pdb
00007ffb`44b40000 00007ffb`44b4f000 kernel_appcore (pdb symbols) d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffb`44b50000 00007ffb`44b60000 msasn1 (pdb symbols) d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffb`44b60000 00007ffb`44bac000 powrprof (pdb symbols) d:\symbols\powrprof.pdb\74B38765BC2D4D1B9E53834273CE9ABF1\powrprof.pdb
00007ffb`44bb0000 00007ffb`44bf2000 cfgmgr32 (pdb symbols) d:\symbols\cfgmgr32.pdb\0C85585E81E54222A3065E20B6C913D41\cfgmgr32.pdb
00007ffb`44c00000 00007ffb`44c6c000 bcryptPrimitives (pdb symbols) d:\symbols\bcryptprimitives.pdb\66FD402F0E2C4120A9CE02CE189B48E91\bcryptprimitives.pdb
00007ffb`44d20000 00007ffb`44e14000 ucrtbase (pdb symbols) d:\symbols\ucrtbase.pdb\FB16A8A30FFD4D868EA1F6402B3143F81\ucrtbase.pdb
00007ffb`44e20000 00007ffb`45013000 crypt32 (pdb symbols) d:\symbols\crypt32.pdb\37224C862FEF48A78732D55DA43319AF1\crypt32.pdb
00007ffb`450c0000 00007ffb`45244000 gdi32full (pdb symbols) d:\symbols\gdi32full.pdb\747C73435FBD4F1784BB0FB4F88D54961\gdi32full.pdb
00007ffb`45250000 00007ffb`4546e000 KERNELBASE (pdb symbols) d:\symbols\kernelbase.pdb\162A8BD56C304693A8AB2468E50C5F6A1\kernelbase.pdb
00007ffb`45470000 00007ffb`4548e000 win32u (pdb symbols) d:\symbols\win32u.pdb\CE777C7E22FC4E43A47A1D6EF5E1AF031\win32u.pdb
00007ffb`45490000 00007ffb`45538000 SHCore (pdb symbols) d:\symbols\shcore.pdb\DD5B4BDC74714727A71E4F62D53DB41B1\shcore.pdb
00007ffb`45540000 00007ffb`45c13000 windows_storage (pdb symbols) d:\symbols\Windows.Storage.pdb\815F10C0A2804CF49A9BEAA4152EBB5F1\Windows.Storage.pdb
00007ffb`45d60000 00007ffb`47266000 shell32 (pdb symbols) d:\symbols\shell32.pdb\C1EE097F73C64A229FFE4097566613111\shell32.pdb
00007ffb`47270000 00007ffb`47278000 nsi (pdb symbols) d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffb`47280000 00007ffb`472b4000 gdi32 (pdb symbols) d:\symbols\gdi32.pdb\44AE0735CFE84F5D847786D34C02BC401\gdi32.pdb
00007ffb`47430000 00007ffb`476f6000 combase (private pdb symbols) d:\symbols\combase.pdb\7F9766E264B5460DB6B97F055B10AB241\combase.pdb
00007ffb`47700000 00007ffb`47866000 user32 (pdb symbols) d:\symbols\user32.pdb\BB7C068C642C4ED2AA5E9DD14A009B311\user32.pdb
00007ffb`47870000 00007ffb`4798d000 rpcrt4 (pdb symbols) d:\symbols\rpcrt4.pdb\C4E0FD0447934EC38FCB2D46A2A4BED21\rpcrt4.pdb
00007ffb`47990000 00007ffb`47a2e000 msvcrt (pdb symbols) d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffb`47a40000 00007ffb`47a9b000 sechost (pdb symbols) d:\symbols\sechost.pdb\F78401B62F914F38AEF989B6261E214B1\sechost.pdb
00007ffb`47aa0000 00007ffb`47b0a000 ws2_32 (pdb symbols) d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffb`47b40000 00007ffb`47be6000 advapi32 (pdb symbols) d:\symbols\advapi32.pdb\0EC476B28EA04617B741EDF5C605AEBC1\advapi32.pdb
00007ffb`47c60000 00007ffb`47d99000 ole32 (private pdb symbols) d:\symbols\ole32.pdb\0E805AFF195F4448B05AE219EAB08F071\ole32.pdb
00007ffb`48410000 00007ffb`48462000 shlwapi (pdb symbols) d:\symbols\shlwapi.pdb\BE622E17968E4B1CBFF14120667BC4C61\shlwapi.pdb
00007ffb`484d0000 00007ffb`4857d000 kernel32 (pdb symbols) d:\symbols\kernel32.pdb\5098B486BCBA4FDFBE1B74CE5C48DC341\kernel32.pdb
00007ffb`486e0000 00007ffb`488af000 ntdll (pdb symbols) d:\symbols\ntdll.pdb\13B5E8DAE6F540E49137EE219A7ED12D1\ntdll.pdb
0:005> lmDvmfluent_bit
Browse full module list
start end module name
00007ff6`a9230000 00007ff6`a9a5e000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
Loaded symbol image file: fluent-bit.exe
Image path: C:\PharosSystems\Tools\FluentBit\bin\fluent-bit.exe
Image name: fluent-bit.exe
Browse all global symbols functions data
Timestamp: Fri Jun 17 23:01:40 2022 (62AC5F14)
CheckSum: 00825963
ImageSize: 0082E000
File version: 1.9.5.0
Product version: 1.9.5.0
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Calyptia Inc.
ProductName: Fluent Bit - Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
InternalName: fluent-bit
ProductVersion: 1.9.5.0
FileVersion: 1.9.5.0
FileDescription: Compiled with MSVC 19.29.30145.0
All rights reserved.
0:005> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 437
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 428
Key : Analysis.Init.CPU.mSec
Value: 405
Key : Analysis.Init.Elapsed.mSec
Value: 20291
Key : Analysis.Memory.CommitPeak.Mb
Value: 185
Key : Timeline.OS.Boot.DeltaSec
Value: 38874
Key : Timeline.Process.Start.DeltaSec
Value: 38724
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Timestamp
Value: 2022-04-29T17:32:00Z
Key : WER.OS.Version
Value: 10.0.14393.5125
Key : WER.Process.Version
Value: 1.9.5.0
FILE_IN_CAB: fluent-bit.exe_220707_132410.dmp
COMMENT:
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000374
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: (.ecxr)
rax=0000000000000000 rbx=000000a48d7fe470 rcx=000000a48d7fea38
rdx=0000000000000008 rsi=000000a48d7fdf80 rdi=000000a48d7fdf80
rip=00007ffb487d7143 rsp=000000a48d7fed60 rbp=000000a48d7ff289
r8=fffffff800000021 r9=0000000000000000 r10=0000000000000000
r11=000000a48d7feac0 r12=000001b225bb0150 r13=0000000000000901
r14=000001b225bb9bb0 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000200
ntdll!RtlReportCriticalFailure+0x97:
00007ffb`487d7143 eb00 jmp ntdll!RtlReportCriticalFailure+0x99 (00007ffb`487d7145)
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffb487d7143 (ntdll!RtlReportCriticalFailure+0x0000000000000097)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00007ffb4882d6b0
PROCESS_NAME: fluent-bit.exe
ERROR_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.
EXCEPTION_CODE_STR: c0000374
EXCEPTION_PARAMETER1: 00007ffb4882d6b0
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
FAULTING_THREAD: ffffffff
STACK_TEXT:
00000000`00000000 00000000`00000000 heap_corruption!fluent-bit.exe+0x0
STACK_COMMAND: !heap ; ** Pseudo Context ** ManagedPseudo ** Value: ffffffff ** ; kb
SYMBOL_NAME: heap_corruption!fluent-bit.exe
MODULE_NAME: heap_corruption
IMAGE_NAME: heap_corruption
FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000374_heap_corruption!fluent-bit.exe
OS_VERSION: 10.0.14393.5125
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {6fea311a-173d-fab0-9a0e-1adeaccaa2de}
Followup: MachineOwner
---------
0:005> kb
# RetAddr : Args to Child : Call Site
00 00007ffb`487d7ae2 : 00000000`00000000 00007ffb`4882d6b0 000001b2`25bb0000 00000000`0000037f : ntdll!RtlReportCriticalFailure+0x97
01 00007ffb`4878574a : 000001b2`25bb0000 000000a4`8d7ff289 00000000`0000037f 00000000`00000000 : ntdll!RtlpHeapHandleError+0x12
02 00007ffb`487053f3 : 000001b2`25bb0000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlpLogHeapFailure+0x96
03 00007ffb`48703ba5 : 000001b2`25bb0000 000001b2`00000002 00000000`00009000 00000000`00009010 : ntdll!RtlpAllocateHeap+0x6f3
04 00007ffb`48701090 : 000001b2`25bb0000 00000000`00009000 000001b2`269c40d0 000000a4`8d7f0000 : ntdll!RtlpAllocateHeapInternal+0x1115
05 00007ffb`48700b4f : 000001b2`26543840 000000a4`8d7ff5d0 00000000`00000004 cccccccc`cccccccc : ntdll!RtlpReAllocateHeap+0x220
06 00007ffb`48700a51 : 000001b2`269c40e0 00007ff6`a9315a76 000001b2`264865a0 00000000`00000fff : ntdll!RtlpReAllocateHeapInternal+0xdf
07 00007ff6`a97e4ac3 : 00000000`00000000 000001b2`26475180 0000b76a`eacfcd27 000001b2`26475180 : ntdll!RtlReAllocateHeap+0x31
08 00007ff6`a9318a76 : 000001b2`26543840 000001b2`26543840 000000a4`8d7ff7f8 000001b2`26543840 : fluent_bit!_realloc_base+0x73 [minkernel\crts\ucrt\src\appcrt\heap\realloc_base.cpp @ 46]
09 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!flb_realloc+0xf [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_mem.h @ 94]
0a (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!flb_http_buffer_increase+0x75 [C:\projects\fluent-bit-2e87g\src\flb_http_client.c @ 857]
0b 00007ff6`a935084e : 000001b2`25ca52d0 00000000`00000000 00000000`0000b702 00000000`00000077 : fluent_bit!flb_http_do+0x1e6 [C:\projects\fluent-bit-2e87g\src\flb_http_client.c @ 1216]
0c 00007ff6`a9307ac1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0000b702 : fluent_bit!cb_es_flush+0x2de [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 858]
0d 00007ff6`a9768b8e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xc1 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_output.h @ 522]
0e 00007ffb`452be951 : 00007ff6`a9307a00 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe [C:\projects\fluent-bit-2e87g\lib\monkey\deps\flb_libco\fiber.c @ 23]
0f 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
Since the issue is upstream due to heap corruption, I installed debugging tools for windows and application verifier, configured gflags for full page heap verification using gflags.exe /p /enable fluent-bit.exe /full, then restarted the fluent bit windows service and monitored it again using procdump64.exe -ma -b -e fluent-bit.exe, which caught another crash dump:
fluent-bit.exe_220708_132010.zip
Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_220708_132010\fluent-bit.exe_220708_132010.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -b -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION'
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK D:\SymbolsTemp
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\SymbolsTemp
Executable search path is:
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 10.0.14393.5125 (rs1_release.220429-1732)
Machine Name:
Debug session time: Sat Jul 9 01:20:10.000 2022 (UTC + 12:00)
System Uptime: 1 days 11:02:39.229
Process Uptime: 0 days 8:27:37.000
...........................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1304.e94): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
fluent_bit!memcpy+0x2ef:
00007ff7`af30e5bf c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:00000215`c9e66fee=32
0:006> .kframes 0n100
Default stack trace depth is 0n100 frames
0:006> .prefer_dml 1
DML versions of commands on by default
0:006> .reload /f
...........................................
0:006> lm
start end module name
00007ff7`aedd0000 00007ff7`af5fe000 fluent_bit (private pdb symbols) d:\symbolstemp\fluent-bit.pdb
00007ffc`89550000 00007ffc`895be000 verifier (pdb symbols) d:\symbols\verifier.pdb\DCFC37DA2F6E4567B06FC84BCF9BEDAF1\verifier.pdb
00007ffc`96b10000 00007ffc`96b26000 NapiNSP (pdb symbols) d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffc`96b30000 00007ffc`96b3e000 winrnr (pdb symbols) d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffc`98b60000 00007ffc`98b6a000 rasadhlp (pdb symbols) d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffc`9cd00000 00007ffc`9cd1a000 dhcpcsvc (pdb symbols) d:\symbols\dhcpcsvc.pdb\896F47CEEF60427FBE41C39E04C01D9E1\dhcpcsvc.pdb
00007ffc`9cd60000 00007ffc`9cd76000 dhcpcsvc6 (pdb symbols) d:\symbols\dhcpcsvc6.pdb\BC7D98B165E241B7856EA8A03648BD371\dhcpcsvc6.pdb
00007ffc`9d9a0000 00007ffc`9da01000 wevtapi (pdb symbols) d:\symbols\wevtapi.pdb\2F5835A7F06F43EE86ADF99297D785921\wevtapi.pdb
00007ffc`9dfb0000 00007ffc`9dfc8000 nlaapi (pdb symbols) d:\symbols\nlaapi.pdb\55CC67AE0E7A423BB758846D361972001\nlaapi.pdb
00007ffc`9f5c0000 00007ffc`9f5f3000 rsaenh (pdb symbols) d:\symbols\rsaenh.pdb\95B56C8731B44E9AA2858411CD4281E21\rsaenh.pdb
00007ffc`9f740000 00007ffc`9f778000 IPHLPAPI (pdb symbols) d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffc`9f780000 00007ffc`9f822000 dnsapi (pdb symbols) d:\symbols\dnsapi.pdb\5C6526F84E6C4878B0CCB26BA6E564A41\dnsapi.pdb
00007ffc`9fa10000 00007ffc`9fa6d000 mswsock (pdb symbols) d:\symbols\mswsock.pdb\DAA810367BC04CDD8EBFE12DA2EB501C1\mswsock.pdb
00007ffc`9fbc0000 00007ffc`9fbd7000 cryptsp (pdb symbols) d:\symbols\cryptsp.pdb\677162669D6740539787653CD28E727F1\cryptsp.pdb
00007ffc`9fbe0000 00007ffc`9fbeb000 CRYPTBASE (pdb symbols) d:\symbols\cryptbase.pdb\A4945E336F8347DDA43A2E9E5FC0ED541\cryptbase.pdb
00007ffc`a0090000 00007ffc`a00bb000 bcrypt (pdb symbols) d:\symbols\bcrypt.pdb\C66626E3ED344020A87327CB85BC26D01\bcrypt.pdb
00007ffc`a0150000 00007ffc`a015f000 kernel_appcore (pdb symbols) d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffc`a0160000 00007ffc`a0170000 msasn1 (pdb symbols) d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffc`a0170000 00007ffc`a01bc000 powrprof (pdb symbols) d:\symbols\powrprof.pdb\74B38765BC2D4D1B9E53834273CE9ABF1\powrprof.pdb
00007ffc`a01c0000 00007ffc`a01d4000 profapi (pdb symbols) d:\symbols\profapi.pdb\CAC4E3CC2E9B4973B17D131D0EFC8ACE1\profapi.pdb
00007ffc`a01e0000 00007ffc`a03fe000 KERNELBASE (pdb symbols) d:\symbols\kernelbase.pdb\162A8BD56C304693A8AB2468E50C5F6A1\kernelbase.pdb
00007ffc`a0400000 00007ffc`a0ad3000 windows_storage (pdb symbols) d:\symbols\Windows.Storage.pdb\815F10C0A2804CF49A9BEAA4152EBB5F1\Windows.Storage.pdb
00007ffc`a0ae0000 00007ffc`a0b4c000 bcryptPrimitives (pdb symbols) d:\symbols\bcryptprimitives.pdb\66FD402F0E2C4120A9CE02CE189B48E91\bcryptprimitives.pdb
00007ffc`a0c60000 00007ffc`a0c7e000 win32u (pdb symbols) d:\symbols\win32u.pdb\CE777C7E22FC4E43A47A1D6EF5E1AF031\win32u.pdb
00007ffc`a0d20000 00007ffc`a0dc8000 SHCore (pdb symbols) d:\symbols\shcore.pdb\DD5B4BDC74714727A71E4F62D53DB41B1\shcore.pdb
00007ffc`a0dd0000 00007ffc`a0f54000 gdi32full (pdb symbols) d:\symbols\gdi32full.pdb\747C73435FBD4F1784BB0FB4F88D54961\gdi32full.pdb
00007ffc`a0f60000 00007ffc`a0fa2000 cfgmgr32 (pdb symbols) d:\symbols\cfgmgr32.pdb\0C85585E81E54222A3065E20B6C913D41\cfgmgr32.pdb
00007ffc`a0fb0000 00007ffc`a10a4000 ucrtbase (pdb symbols) d:\symbols\ucrtbase.pdb\FB16A8A30FFD4D868EA1F6402B3143F81\ucrtbase.pdb
00007ffc`a10b0000 00007ffc`a12a3000 crypt32 (pdb symbols) d:\symbols\crypt32.pdb\37224C862FEF48A78732D55DA43319AF1\crypt32.pdb
00007ffc`a1390000 00007ffc`a2896000 shell32 (pdb symbols) d:\symbols\shell32.pdb\C1EE097F73C64A229FFE4097566613111\shell32.pdb
00007ffc`a2db0000 00007ffc`a2de4000 gdi32 (pdb symbols) d:\symbols\gdi32.pdb\44AE0735CFE84F5D847786D34C02BC401\gdi32.pdb
00007ffc`a2df0000 00007ffc`a2df8000 nsi (pdb symbols) d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffc`a2e00000 00007ffc`a2ead000 kernel32 (pdb symbols) d:\symbols\kernel32.pdb\5098B486BCBA4FDFBE1B74CE5C48DC341\kernel32.pdb
00007ffc`a2eb0000 00007ffc`a3176000 combase (private pdb symbols) d:\symbols\combase.pdb\7F9766E264B5460DB6B97F055B10AB241\combase.pdb
00007ffc`a3210000 00007ffc`a3262000 shlwapi (pdb symbols) d:\symbols\shlwapi.pdb\BE622E17968E4B1CBFF14120667BC4C61\shlwapi.pdb
00007ffc`a3270000 00007ffc`a330e000 msvcrt (pdb symbols) d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffc`a3310000 00007ffc`a336b000 sechost (pdb symbols) d:\symbols\sechost.pdb\F78401B62F914F38AEF989B6261E214B1\sechost.pdb
00007ffc`a37b0000 00007ffc`a381a000 ws2_32 (pdb symbols) d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffc`a3890000 00007ffc`a39f6000 user32 (pdb symbols) d:\symbols\user32.pdb\BB7C068C642C4ED2AA5E9DD14A009B311\user32.pdb
00007ffc`a3a00000 00007ffc`a3aa6000 advapi32 (pdb symbols) d:\symbols\advapi32.pdb\0EC476B28EA04617B741EDF5C605AEBC1\advapi32.pdb
00007ffc`a3ab0000 00007ffc`a3bcd000 rpcrt4 (pdb symbols) d:\symbols\rpcrt4.pdb\C4E0FD0447934EC38FCB2D46A2A4BED21\rpcrt4.pdb
00007ffc`a3bd0000 00007ffc`a3d09000 ole32 (private pdb symbols) d:\symbols\ole32.pdb\0E805AFF195F4448B05AE219EAB08F071\ole32.pdb
00007ffc`a3d10000 00007ffc`a3edf000 ntdll (pdb symbols) d:\symbols\ntdll.pdb\13B5E8DAE6F540E49137EE219A7ED12D1\ntdll.pdb
0:006> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : AV.Fault
Value: Write
Key : Analysis.CPU.mSec
Value: 437
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 2436
Key : Analysis.Init.CPU.mSec
Value: 562
Key : Analysis.Init.Elapsed.mSec
Value: 20604
Key : Analysis.Memory.CommitPeak.Mb
Value: 196
Key : Timeline.OS.Boot.DeltaSec
Value: 126159
Key : Timeline.Process.Start.DeltaSec
Value: 30457
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Timestamp
Value: 2022-04-29T17:32:00Z
Key : WER.OS.Version
Value: 10.0.14393.5125
Key : WER.Process.Version
Value: 1.9.5.0
FILE_IN_CAB: fluent-bit.exe_220708_132010.dmp
COMMENT:
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -b -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION
NTGLOBALFLAG: 2000000
APPLICATION_VERIFIER_FLAGS: 0
APPLICATION_VERIFIER_LOADED: 1
CONTEXT: (.ecxr)
rax=00000215c9e65545 rbx=00000215be521ff0 rcx=00000215c9e66f60
rdx=00000215c9e5862b rsi=000000000000010a rdi=00000000000019de
rip=00007ff7af30e5bf rsp=00000040a1bff338 rbp=0000000000001ac9
r8=00000000000000ae r9=00000000000000c0 r10=00007ff7aedd0000
r11=00007ff7af30e56d r12=00000215ca1fcf50 r13=00000215c4ce2df0
r14=000000000000003b r15=00000215c9e56c10
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010200
fluent_bit!memcpy+0x2ef:
00007ff7`af30e5bf c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:00000215`c9e66fee=32
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ff7af30e5bf (fluent_bit!memcpy+0x00000000000002ef)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 00000215c9e67000
Attempt to write to address 00000215c9e67000
PROCESS_NAME: fluent-bit.exe
WRITE_ADDRESS: 00000215c9e67000
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000001
EXCEPTION_PARAMETER2: 00000215c9e67000
STACK_TEXT:
00000040`a1bff338 00007ff7`aeef2f75 : 00000215`be521ff0 00007ff7`af37dabc 00000040`a1bff5e9 00000000`00000000 : fluent_bit!memcpy+0x2ef
00000040`a1bff340 00007ff7`aeef02d7 : 00000040`a1bff5e9 00000040`a1bff480 00000215`be521ff0 00000215`c9e56c10 : fluent_bit!es_bulk_append+0x105
00000040`a1bff380 00007ff7`aeef05f1 : 00000215`bedbedd0 00000215`c4ce2df0 00000215`bedbedd0 00000215`bae6efd0 : fluent_bit!elasticsearch_format+0x17a7
00000040`a1bff920 00007ff7`aeea7ac1 : 00000215`c4e9aad0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!cb_es_flush+0x81
00000040`a1bff9b0 00007ff7`af308b8e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xc1
00000040`a1bffa60 00007ffc`a024e951 : 00007ff7`aeea7a00 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe
00000040`a1bffa90 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
STACK_COMMAND: ~6s; .ecxr ; kb
FAULTING_SOURCE_LINE: D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm
FAULTING_SOURCE_FILE: D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm
FAULTING_SOURCE_LINE_NUMBER: 405
FAULTING_SOURCE_CODE:
No source found for 'D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm'
SYMBOL_NAME: fluent_bit!memcpy+2ef
MODULE_NAME: fluent_bit
IMAGE_NAME: fluent-bit.exe
FAILURE_BUCKET_ID: INVALID_POINTER_WRITE_CONTEXT_MISMATCH_AVRF_c0000005_fluent-bit.exe!memcpy
OS_VERSION: 10.0.14393.5125
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 1.9.5.0
FAILURE_ID_HASH: {0d85db40-15bc-ec4a-c8fd-20b5102ced04}
Followup: MachineOwner
---------
0:006> ~6s; .ecxr ; kb
fluent_bit!memcpy+0x2ef:
00007ff7`af30e5bf c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:00000215`c9e66fee=32
rax=00000215c9e65545 rbx=00000215be521ff0 rcx=00000215c9e66f60
rdx=00000215c9e5862b rsi=000000000000010a rdi=00000000000019de
rip=00007ff7af30e5bf rsp=00000040a1bff338 rbp=0000000000001ac9
r8=00000000000000ae r9=00000000000000c0 r10=00007ff7aedd0000
r11=00007ff7af30e56d r12=00000215ca1fcf50 r13=00000215c4ce2df0
r14=000000000000003b r15=00000215c9e56c10
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010200
fluent_bit!memcpy+0x2ef:
00007ff7`af30e5bf c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:00000215`c9e66fee=32
*** Stack trace for last set context - .thread/.cxr resets it
# RetAddr : Args to Child : Call Site
00 00007ff7`aeef2f75 : 00000215`be521ff0 00007ff7`af37dabc 00000040`a1bff5e9 00000000`00000000 : fluent_bit!memcpy+0x2ef [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm @ 405]
01 00007ff7`aeef02d7 : 00000040`a1bff5e9 00000040`a1bff480 00000215`be521ff0 00000215`c9e56c10 : fluent_bit!es_bulk_append+0x105 [C:\projects\fluent-bit-2e87g\plugins\out_es\es_bulk.c @ 105]
02 00007ff7`aeef05f1 : 00000215`bedbedd0 00000215`c4ce2df0 00000215`bedbedd0 00000215`bae6efd0 : fluent_bit!elasticsearch_format+0x17a7 [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 577]
03 00007ff7`aeea7ac1 : 00000215`c4e9aad0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!cb_es_flush+0x81 [C:\projects\fluent-bit-2e87g\plugins\out_es\es.c @ 816]
04 00007ff7`af308b8e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xc1 [C:\projects\fluent-bit-2e87g\include\fluent-bit\flb_output.h @ 522]
05 00007ffc`a024e951 : 00007ff7`aeea7a00 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe [C:\projects\fluent-bit-2e87g\lib\monkey\deps\flb_libco\fiber.c @ 23]
06 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
0:006> .frame 0n1;dv /t /v
01 00000040`a1bff340 00007ff7`aeef02d7 fluent_bit!es_bulk_append+0x105 [C:\projects\fluent-bit-2e87g\plugins\out_es\es_bulk.c @ 105]
@rbx struct es_bulk * bulk = 0x00000215`be521ff0
@r12 char * index = 0x00000215`ca1fcf50 "{"create":{"_index":"logstash-2022.07.08","_type":"_doc"}}."
@r14 int i_len = 0n59
@r15 char * json = 0x00000215`c9e56c10 "{"@timestamp":"2022-07-08T13:20:09.945Z","Payload":{"httpMethod":"GET","clientIP":"","methodName":"","activityIdentifier":"3af913c1-83e3-4efa-9c94-8c17f1750e1b","error":"Pharos.Mps.Library.Exceptions.MpsServerException: Difference in date between Client and Server outside allowable limit of 900 seconds\r\n at Pharos.Mps.Library.ServiceHost.Web.Filters.RequestExpiredFilter.OnActionExecuting(HttpActionContext actionContext)\r\n at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutingAsync(HttpActionContext actionContext, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Pharos.Mps.Library.ServiceHost.Web.MessageHandlers.LoggingMessageHandler.<SendAsync>d__0.MoveNext()","customerGuid":"00000000-0000-0000-0000-000000000000","processName":"Pharos.Mps.PrintScoutApi.Service","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/f8bee939-bc42-4566-adef-f4f65851cf1a/agents/41c7e704-6db4-98b4-4d57-f7929cd7b6d2/settings","threadId":23,"jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Service":"Pharos.Mps.PrintScoutApi.Service","Task":"None","Version":0,"EventName":"Error","Timestamp":"2022-07-08T13:20:09.0424971Z","Keywords":"0000000000000010","EventSource":"Pharos-Mps-System","Message":"GET http://printscout.insights.hpondemand.com/printAgentCommunication/customers/f8bee939-bc42-4566-adef-f4f65851cf1a/agents/41c7e704-6db4-98b4-4d57-f7929cd7b6d2/settings failed.","EventId":332,"Level":"Error","Opcode":"50"}"
@rbp unsigned int64 j_len = 0x1ac9
00000040`a1bff3a8 unsigned int64 whole_size = 0x9e54
00000040`a1bff3b0 unsigned int64 converted_size = 0x8476
@esi int available = 0n266
<unavailable> char * ptr = <value unavailable>
<unavailable> int append_size = <value unavailable>
0:006> dx -r1 ((fluent_bit!es_bulk *)0x215be521ff0)
((fluent_bit!es_bulk *)0x215be521ff0) : 0x215be521ff0 [Type: es_bulk *]
[+0x000] ptr : 0x215c9e5b7c0 : "{"create":{"_index":"logstash-2022.07.08","_type":"_doc"}}.{"@timestamp":"2022-07-08T13:20:09.940Z","Payload":{"messageIdentifier":"62419c3b-5b1d-4597-b43d-15cdf8debabc","queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"ee3dfbfb-542d-459d-9fdc-648416cac703","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"6fa89ebe-965a-425f-acfb-9117aa7630ec","customerGuid":"00000000-0000-0000-0000-000000000000","threadId":35,"packetTy... [Type: char *]
[+0x008] len : 0x9d85 [Type: unsigned int]
[+0x00c] size : 0xb832 [Type: unsigned int]
0:006> db 0x215c9e5b7c0 + 0xb832
00000215`c9e66ff2 65 76 65 6c 22 3a 22 45-72 72 6f 72 22 2c ?? ?? evel":"Error",??
00000215`c9e67002 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67012 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67022 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67032 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67042 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67052 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
00000215`c9e67062 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
@edsiper this reveals a buffer overrun:
fluent-bit/plugins/out_es/es_bulk.c
Line 104
9ec4344
It definitely looks like i_len should be added there.
As for the division, I'm not familiar with that plugin, what I interpret from that code is that :
whole_size is the total size of the chunk that's being flushedconverted_size is the length of the data that has already been converted up to the last msgpack record)bulk->size is the current size of the buffer where the converted entry is being appendedFrom that I infer that they are establishing the size relationship between "raw msgpack record size" and "bulk encoded record size". I wonder it that's as predictable as it sounds. Also, I think you're right about the rounding error.
To my fresh eyes, I can only deduce the seemingly redundant code complexity is trying to predict subsequent buffer requirements to avoid future reallocs. However, the current buffer requirements are clear based on the immediately following writes, so this prediction should at least make sure it's not less than this minimum requirement:
#include <math.h>
required = i_len + j_len + ES_BULK_HEADER + 1;
available = (bulk->size - bulk->len);
if (available < required) {
append_size = required - available;
if (converted_size > 0) {
append_size = max(append_size,
ceil((whole_size - converted_size) /* rest of size to convert */
* ((double)bulk->size / converted_size))); /* = json size / msgpack size */
I made another custom build using 1.9...gitfool:fluent-bit:gh5680-ods with extra logging to Windows OutputDebugString, which I can see inline during a debug session with WinDbg. Then I captured another crash with a debug build:
fluent-bit_0f18_2022-07-25_18-11-08-224_06a8.zip
fluent-bit-ods.log.zip
WinDbg.log:
Microsoft (R) Windows Debugger Version 10.0.25136.1001 AMD64
Loading Dump File [D:\Work\Support\FluentBit\fluent-bit_0f18_2022-07-25_18-11-08-224_06a8\fluent-bit_0f18_2022-07-25_18-11-08-224_06a8.mdmp]
User Mini Dump File with Full Memory: Only application data is available
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK D:\SymbolsTemp
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\SymbolsTemp
Executable search path is:
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 14393.5192.amd64fre.rs1_release.220610-1622
Machine Name:
Debug session time: Tue Jul 26 06:11:08.000 2022 (UTC + 12:00)
System Uptime: 18 days 15:34:50.119
Process Uptime: 0 days 16:09:30.000
...........................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(6a8.1654): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for fluent-bit.exe
fluent_bit!memcpy+0x2ef:
00007ff7`d09d9a6f c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:000001e2`dbb15fe1=2d
0:007> .kframes 0n100
Default stack trace depth is 0n100 frames
0:007> .prefer_dml 1
DML versions of commands on by default
0:007> .reload /f
.*** WARNING: Unable to verify checksum for fluent-bit.exe
..........................................
0:007> lm
start end module name
00007ff7`d05c0000 00007ff7`d0cab000 fluent_bit C (private pdb symbols) D:\Devel\gitfool\fluent-bit\build\bin\Debug\fluent-bit.pdb
00007ffb`2a8d0000 00007ffb`2a93e000 verifier (pdb symbols) d:\symbols\verifier.pdb\DCFC37DA2F6E4567B06FC84BCF9BEDAF1\verifier.pdb
00007ffb`3b930000 00007ffb`3b946000 NapiNSP (pdb symbols) d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffb`3b950000 00007ffb`3b95e000 winrnr (pdb symbols) d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffb`40870000 00007ffb`4087a000 rasadhlp (pdb symbols) d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffb`417f0000 00007ffb`4180a000 dhcpcsvc (pdb symbols) d:\symbols\dhcpcsvc.pdb\896F47CEEF60427FBE41C39E04C01D9E1\dhcpcsvc.pdb
00007ffb`41830000 00007ffb`41846000 dhcpcsvc6 (pdb symbols) d:\symbols\dhcpcsvc6.pdb\BC7D98B165E241B7856EA8A03648BD371\dhcpcsvc6.pdb
00007ffb`41b70000 00007ffb`41bd1000 wevtapi (pdb symbols) d:\symbols\wevtapi.pdb\2F5835A7F06F43EE86ADF99297D785921\wevtapi.pdb
00007ffb`42cd0000 00007ffb`42ce8000 nlaapi (pdb symbols) d:\symbols\nlaapi.pdb\55CC67AE0E7A423BB758846D361972001\nlaapi.pdb
00007ffb`43f90000 00007ffb`43fc3000 rsaenh (pdb symbols) d:\symbols\rsaenh.pdb\95B56C8731B44E9AA2858411CD4281E21\rsaenh.pdb
00007ffb`44110000 00007ffb`44148000 IPHLPAPI (pdb symbols) d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffb`44150000 00007ffb`441f2000 dnsapi (pdb symbols) d:\symbols\dnsapi.pdb\5C6526F84E6C4878B0CCB26BA6E564A41\dnsapi.pdb
00007ffb`443e0000 00007ffb`4443d000 mswsock (pdb symbols) d:\symbols\mswsock.pdb\DAA810367BC04CDD8EBFE12DA2EB501C1\mswsock.pdb
00007ffb`44590000 00007ffb`445a7000 cryptsp (pdb symbols) d:\symbols\cryptsp.pdb\677162669D6740539787653CD28E727F1\cryptsp.pdb
00007ffb`445b0000 00007ffb`445bb000 CRYPTBASE (pdb symbols) d:\symbols\cryptbase.pdb\A4945E336F8347DDA43A2E9E5FC0ED541\cryptbase.pdb
00007ffb`44a70000 00007ffb`44a9b000 bcrypt (pdb symbols) d:\symbols\bcrypt.pdb\C66626E3ED344020A87327CB85BC26D01\bcrypt.pdb
00007ffb`44b20000 00007ffb`44b34000 profapi (pdb symbols) d:\symbols\profapi.pdb\CAC4E3CC2E9B4973B17D131D0EFC8ACE1\profapi.pdb
00007ffb`44b40000 00007ffb`44b4f000 kernel_appcore (pdb symbols) d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffb`44b50000 00007ffb`44b60000 msasn1 (pdb symbols) d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffb`44b60000 00007ffb`44bac000 powrprof (pdb symbols) d:\symbols\powrprof.pdb\74B38765BC2D4D1B9E53834273CE9ABF1\powrprof.pdb
00007ffb`44bb0000 00007ffb`44bf2000 cfgmgr32 (pdb symbols) d:\symbols\cfgmgr32.pdb\0C85585E81E54222A3065E20B6C913D41\cfgmgr32.pdb
00007ffb`44c00000 00007ffb`44c6c000 bcryptPrimitives (pdb symbols) d:\symbols\bcryptprimitives.pdb\66FD402F0E2C4120A9CE02CE189B48E91\bcryptprimitives.pdb
00007ffb`44d20000 00007ffb`44e14000 ucrtbase (pdb symbols) d:\symbols\ucrtbase.pdb\FB16A8A30FFD4D868EA1F6402B3143F81\ucrtbase.pdb
00007ffb`44e20000 00007ffb`45013000 crypt32 (pdb symbols) d:\symbols\crypt32.pdb\37224C862FEF48A78732D55DA43319AF1\crypt32.pdb
00007ffb`450c0000 00007ffb`45244000 gdi32full (pdb symbols) d:\symbols\gdi32full.pdb\747C73435FBD4F1784BB0FB4F88D54961\gdi32full.pdb
00007ffb`45250000 00007ffb`4546e000 KERNELBASE (pdb symbols) d:\symbols\kernelbase.pdb\162A8BD56C304693A8AB2468E50C5F6A1\kernelbase.pdb
00007ffb`45470000 00007ffb`4548e000 win32u (pdb symbols) d:\symbols\win32u.pdb\CE777C7E22FC4E43A47A1D6EF5E1AF031\win32u.pdb
00007ffb`45490000 00007ffb`45538000 SHCore (pdb symbols) d:\symbols\shcore.pdb\DD5B4BDC74714727A71E4F62D53DB41B1\shcore.pdb
00007ffb`45540000 00007ffb`45c13000 windows_storage (pdb symbols) d:\symbols\Windows.Storage.pdb\815F10C0A2804CF49A9BEAA4152EBB5F1\Windows.Storage.pdb
00007ffb`45d60000 00007ffb`47266000 shell32 (pdb symbols) d:\symbols\shell32.pdb\C1EE097F73C64A229FFE4097566613111\shell32.pdb
00007ffb`47270000 00007ffb`47278000 nsi (pdb symbols) d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffb`47280000 00007ffb`472b4000 gdi32 (pdb symbols) d:\symbols\gdi32.pdb\44AE0735CFE84F5D847786D34C02BC401\gdi32.pdb
00007ffb`47430000 00007ffb`476f6000 combase (private pdb symbols) d:\symbols\combase.pdb\7F9766E264B5460DB6B97F055B10AB241\combase.pdb
00007ffb`47700000 00007ffb`47866000 user32 (pdb symbols) d:\symbols\user32.pdb\BB7C068C642C4ED2AA5E9DD14A009B311\user32.pdb
00007ffb`47870000 00007ffb`4798d000 rpcrt4 (pdb symbols) d:\symbols\rpcrt4.pdb\C4E0FD0447934EC38FCB2D46A2A4BED21\rpcrt4.pdb
00007ffb`47990000 00007ffb`47a2e000 msvcrt (pdb symbols) d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffb`47a40000 00007ffb`47a9b000 sechost (pdb symbols) d:\symbols\sechost.pdb\F78401B62F914F38AEF989B6261E214B1\sechost.pdb
00007ffb`47aa0000 00007ffb`47b0a000 ws2_32 (pdb symbols) d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffb`47b40000 00007ffb`47be6000 advapi32 (pdb symbols) d:\symbols\advapi32.pdb\0EC476B28EA04617B741EDF5C605AEBC1\advapi32.pdb
00007ffb`47c60000 00007ffb`47d99000 ole32 (private pdb symbols) d:\symbols\ole32.pdb\0E805AFF195F4448B05AE219EAB08F071\ole32.pdb
00007ffb`48410000 00007ffb`48462000 shlwapi (pdb symbols) d:\symbols\shlwapi.pdb\BE622E17968E4B1CBFF14120667BC4C61\shlwapi.pdb
00007ffb`484d0000 00007ffb`4857d000 kernel32 (pdb symbols) d:\symbols\kernel32.pdb\5098B486BCBA4FDFBE1B74CE5C48DC341\kernel32.pdb
00007ffb`486e0000 00007ffb`488af000 ntdll (pdb symbols) d:\symbols\ntdll.pdb\13B5E8DAE6F540E49137EE219A7ED12D1\ntdll.pdb
0:007> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : AV.Fault
Value: Write
Key : Analysis.CPU.mSec
Value: 421
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 2988
Key : Analysis.Init.CPU.mSec
Value: 562
Key : Analysis.Init.Elapsed.mSec
Value: 7649
Key : Analysis.Memory.CommitPeak.Mb
Value: 181
Key : Timeline.OS.Boot.DeltaSec
Value: 1611290
Key : Timeline.Process.Start.DeltaSec
Value: 58170
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Timestamp
Value: 2022-06-10T16:22:00Z
Key : WER.OS.Version
Value: 10.0.14393.5192
Key : WER.Process.Version
Value: 1.9.7.0
FILE_IN_CAB: fluent-bit_0f18_2022-07-25_18-11-08-224_06a8.mdmp
NTGLOBALFLAG: 2000000
APPLICATION_VERIFIER_FLAGS: 0
APPLICATION_VERIFIER_LOADED: 1
CONTEXT: (.ecxr)
rax=000001e2dbb14539 rbx=0000000000000000 rcx=000001e2dbb15f40
rdx=000001e2ddc28617 rsi=00000098d11ffa50 rdi=00000098d11ffbf8
rip=00007ff7d09d9a6f rsp=00000098d11ff4e8 rbp=0000000000000000
r8=00000000000000c1 r9=00000000000000e0 r10=00007ff7d05c0000
r11=00007ff7d09d9a09 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
fluent_bit!memcpy+0x2ef:
00007ff7`d09d9a6f c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:000001e2`dbb15fe1=2d
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ff7d09d9a6f (fluent_bit!memcpy+0x00000000000002ef)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 000001e2dbb16000
Attempt to write to address 000001e2dbb16000
PROCESS_NAME: fluent-bit.exe
WRITE_ADDRESS: 000001e2dbb16000
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000001
EXCEPTION_PARAMETER2: 000001e2dbb16000
STACK_TEXT:
00000098`d11ff4e8 00007ff7`d069b150 : 00007ff7`d0bf3728 000001e2`dbb0a720 00000000`0000b8dc 000001e2`dbb15ffb : fluent_bit!memcpy+0x2ef
00000098`d11ff4f0 00007ff7`d069808a : 000001e2`de300ff0 000001e2`d5c05f50 000001e2`0000003b 000001e2`ddc26c10 : fluent_bit!es_bulk_append+0x240
00000098`d11ff550 00007ff7`d0698ea2 : 000001e2`d266e7c0 000001e2`d655cdd0 000001e2`d8034df0 00000000`00000000 : fluent_bit!elasticsearch_format+0x104a
00000098`d11ffc40 00007ff7`d063345b : 000001e2`dbdc1fd0 000001e2`dc076fc0 000001e2`d655cdd0 000001e2`d8034df0 : fluent_bit!cb_es_flush+0x112
00000098`d11ffd60 00007ff7`d09d41be : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xbb
00000098`d11ffe20 00007ffb`452be951 : 00007ff7`d06333a0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe
00000098`d11ffe50 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
STACK_COMMAND: ~7s; .ecxr ; kb
FAULTING_SOURCE_LINE: D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm
FAULTING_SOURCE_FILE: D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm
FAULTING_SOURCE_LINE_NUMBER: 405
FAULTING_SOURCE_CODE:
No source found for 'D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm'
SYMBOL_NAME: fluent_bit!memcpy+2ef
MODULE_NAME: fluent_bit
IMAGE_NAME: fluent-bit.exe
FAILURE_BUCKET_ID: INVALID_POINTER_WRITE_AVRF_c0000005_fluent-bit.exe!memcpy
OS_VERSION: 10.0.14393.5192
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 1.9.7.0
FAILURE_ID_HASH: {58f7f8fb-0ec2-b253-306f-243a10916113}
Followup: MachineOwner
---------
0:007> ~7s; .ecxr ; kb
fluent_bit!memcpy+0x2ef:
00007ff7`d09d9a6f c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:000001e2`dbb15fe1=2d
rax=000001e2dbb14539 rbx=0000000000000000 rcx=000001e2dbb15f40
rdx=000001e2ddc28617 rsi=00000098d11ffa50 rdi=00000098d11ffbf8
rip=00007ff7d09d9a6f rsp=00000098d11ff4e8 rbp=0000000000000000
r8=00000000000000c1 r9=00000000000000e0 r10=00007ff7d05c0000
r11=00007ff7d09d9a09 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
fluent_bit!memcpy+0x2ef:
00007ff7`d09d9a6f c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:000001e2`dbb15fe1=2d
*** Stack trace for last set context - .thread/.cxr resets it
# RetAddr : Args to Child : Call Site
00 00007ff7`d069b150 : 00007ff7`d0bf3728 000001e2`dbb0a720 00000000`0000b8dc 000001e2`dbb15ffb : fluent_bit!memcpy+0x2ef [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm @ 405]
01 00007ff7`d069808a : 000001e2`de300ff0 000001e2`d5c05f50 000001e2`0000003b 000001e2`ddc26c10 : fluent_bit!es_bulk_append+0x240 [D:\Devel\gitfool\fluent-bit\plugins\out_es\es_bulk.c @ 110]
02 00007ff7`d0698ea2 : 000001e2`d266e7c0 000001e2`d655cdd0 000001e2`d8034df0 00000000`00000000 : fluent_bit!elasticsearch_format+0x104a [D:\Devel\gitfool\fluent-bit\plugins\out_es\es.c @ 584]
03 00007ff7`d063345b : 000001e2`dbdc1fd0 000001e2`dc076fc0 000001e2`d655cdd0 000001e2`d8034df0 : fluent_bit!cb_es_flush+0x112 [D:\Devel\gitfool\fluent-bit\plugins\out_es\es.c @ 823]
04 00007ff7`d09d41be : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!output_pre_cb_flush+0xbb [D:\Devel\gitfool\fluent-bit\include\fluent-bit\flb_output.h @ 522]
05 00007ffb`452be951 : 00007ff7`d06333a0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0xe [D:\Devel\gitfool\fluent-bit\lib\monkey\deps\flb_libco\fiber.c @ 23]
06 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21
0:007> .frame 0n1;dv /t /v
01 00000098`d11ff4f0 00007ff7`d069808a fluent_bit!es_bulk_append+0x240 [D:\Devel\gitfool\fluent-bit\plugins\out_es\es_bulk.c @ 110]
00000098`d11ff550 struct es_bulk * bulk = 0x000001e2`de300ff0
00000098`d11ff558 char * index = 0x000001e2`d5c05f50 "{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}."
00000098`d11ff560 int i_len = 0n59
00000098`d11ff568 char * json = 0x000001e2`ddc26c10 "{"@timestamp":"2022-07-25T12:21:08.863Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":332,"Opcode":"50","Timestamp":"2022-07-25T12:21:05.0528675Z","Level":"Error","EventName":"Error","Payload":{"activityIdentifier":"f865817d-68bc-41d6-91a4-3b3f3e7b00cd","httpMethod":"GET","threadId":4,"error":"Pharos.Mps.Library.Exceptions.MpsServerException: Difference in date between Client and Server outside allowable limit of 900 seconds\r\n at Pharos.Mps.Library.ServiceHost.Web.Filters.RequestExpiredFilter.OnActionExecuting(HttpActionContext actionContext)\r\n at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutingAsync(HttpActionContext actionContext, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Pharos.Mps.Library.ServiceHost.Web.MessageHandlers.LoggingMessageHandler.<SendAsync>d__0.MoveNext()","customerGuid":"00000000-0000-0000-0000-000000000000","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/agents/4027030c-9dab-a0ca-ce58-72e6ee154952/settings","methodName":"","clientIP":"","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"None","Message":"GET http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/agents/4027030c-9dab-a0ca-ce58-72e6ee154952/settings failed."}"
00000098`d11ff570 unsigned int64 j_len = 0x1ac8
00000098`d11ff578 unsigned int64 whole_size = 0x9efe
00000098`d11ff580 unsigned int64 converted_size = 0x8520
00000098`d11ff520 int available = 0n288
00000098`d11ff530 char * ptr = 0x000001e2`dbb0a720 "{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.859Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":333,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.4787479Z","Level":"Debug","EventName":"WebMethodCompletedInfo","Payload":{"activityIdentifier":"ab6a18ba-cbb7-40e3-a579-934af2bbbfa1","httpMethod":"GET","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","clientIP":"","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/042de839-91b9-4e28-9662-6d2663d17bc7/agents/32d0f833-5449-b870-dd59-b44ffde6ed0c/settings","methodName":"","duration":"00:00:00.047","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65201","Message":"<< GET http://printscout.insights.hpondemand.com/printAgentCommunication/customers/042de839-91b9-4e28-9662-6d2663d17bc7/agents/32d0f833-5449-b870-dd59-b44ffde6ed0c/settings completed in 00:00:00.047."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":331,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.5270509Z","Level":"Debug","EventName":"WebMethodStartingInfo","Payload":{"activityIdentifier":"6f58a1ba-7ef3-42aa-94cb-4d216735cdde","httpMethod":"PUT","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2","methodName":"","clientIP":"","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65203","Message":">> PUT http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2 starting."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":100,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.5273454Z","Level":"Debug","EventName":"ConfigurationValidateInfo","Payload":{"threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","configurationType":"AmazonSQSConfiguration","activityIdentifier":"6f58a1ba-7ef3-42aa-94cb-4d216735cdde","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000001","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65434","Message":"Validating 'AmazonSQSConfiguration' configuration for correctness."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":318,"Opcode":"50","Timestamp":"2022-07-25T12:21:03.5277038Z","Level":"Debug","EventName":"Error","Payload":{"threadId":4,"sourceIP":"","activityIdentifier":"6f58a1ba-7ef3-42aa-94cb-4d216735cdde","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"00000000-0000-0000-0000-000000000000","processName":"Pharos.Mps.PrintScoutApi.Service","httpMethod":"PUT"},"Keywords":"0000000000000012","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"None","Message":"The payload of the request to 'http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2' is not valid."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":333,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.5277927Z","Level":"Debug","EventName":"WebMethodCompletedInfo","Payload":{"activityIdentifier":"6f58a1ba-7ef3-42aa-94cb-4d216735cdde","httpMethod":"PUT","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","clientIP":"","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2","methodName":"","duration":"00:00:00.000","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65201","Message":"<< PUT http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/agents/accf8baa-f9f8-9580-fb54-eb64f96805c2 completed in 00:00:00.000."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":331,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6233126Z","Level":"Debug","EventName":"WebMethodStartingInfo","Payload":{"activityIdentifier":"ae41cb77-d751-47e0-be69-f5cd19fccef9","httpMethod":"POST","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/jobs","methodName":"","clientIP":"","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65203","Message":">> POST http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/jobs starting."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":100,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6235981Z","Level":"Debug","EventName":"ConfigurationValidateInfo","Payload":{"threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","configurationType":"AmazonSQSConfiguration","activityIdentifier":"ae41cb77-d751-47e0-be69-f5cd19fccef9","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000001","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65434","Message":"Validating 'AmazonSQSConfiguration' configuration for correctness."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-Library","EventId":4707,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6469139Z","Level":"Debug","EventName":"QueuePublishPacketInfo","Payload":{"threadId":11,"queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"ae41cb77-d751-47e0-be69-f5cd19fccef9","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"88d0b322-304f-4a77-ab1c-8aab7d58285c","processName":"Pharos.Mps.PrintScoutApi.Service","packetType":"JobWrapper"},"Keywords":"0000000000000100","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"60827","Message":"Publishing a 'JobWrapper' packet to the https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd queue."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-Library","EventId":4708,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6570392Z","Level":"Debug","EventName":"QueuePublishedPacketInfo","Payload":{"threadId":11,"queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"ae41cb77-d751-47e0-be69-f5cd19fccef9","messageIdentifier":"6699e126-d3c9-491c-9042-72a9a10b1b9d","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"88d0b322-304f-4a77-ab1c-8aab7d58285c","pro...cessName":"Pharos.Mps.PrintScoutApi.Service","packetType":"JobWrapper"},"Keywords":"0000000000000100","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"60826","Message":"Published a 'JobWrapper' packet to the https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd queue."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":333,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6572756Z","Level":"Debug","EventName":"WebMethodCompletedInfo","Payload":{"activityIdentifier":"ae41cb77-d751-47e0-be69-f5cd19fccef9","httpMethod":"POST","threadId":11,"customerGuid":"00000000-0000-0000-0000-000000000000","clientIP":"","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/jobs","methodName":"","duration":"00:00:00.033","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65201","Message":"<< POST http://printscout.insights.hpondemand.com/printAgentCommunication/customers/78cbef39-e6fb-47df-aaa3-d94f015c7bc4/jobs completed in 00:00:00.033."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":331,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6779158Z","Level":"Debug","EventName":"WebMethodStartingInfo","Payload":{"activityIdentifier":"70877671-63a3-4904-9155-6a75d23f262b","httpMethod":"POST","threadId":11,"customerGuid":"00000000-0000-0000-0000-000000000000","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs","methodName":"","clientIP":"","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65203","Message":">> POST http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs starting."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":100,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.6782615Z","Level":"Debug","EventName":"ConfigurationValidateInfo","Payload":{"threadId":11,"customerGuid":"00000000-0000-0000-0000-000000000000","configurationType":"AmazonSQSConfiguration","activityIdentifier":"70877671-63a3-4904-9155-6a75d23f262b","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000001","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65434","Message":"Validating 'AmazonSQSConfiguration' configuration for correctness."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-Library","EventId":4707,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.7021691Z","Level":"Debug","EventName":"QueuePublishPacketInfo","Payload":{"threadId":4,"queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"70877671-63a3-4904-9155-6a75d23f262b","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"15dbc196-b1de-49f7-9d13-0e65da1598be","processName":"Pharos.Mps.PrintScoutApi.Service","packetType":"JobWrapper"},"Keywords":"0000000000000100","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"60827","Message":"Publishing a 'JobWrapper' packet to the https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd queue."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.860Z","Version":0,"EventSource":"Pharos-Mps-Library","EventId":4708,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.7121388Z","Level":"Debug","EventName":"QueuePublishedPacketInfo","Payload":{"threadId":4,"queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"70877671-63a3-4904-9155-6a75d23f262b","messageIdentifier":"b9dcb7ed-4aaf-4f09-aa3b-fc50f8f3045b","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"15dbc196-b1de-49f7-9d13-0e65da1598be","processName":"Pharos.Mps.PrintScoutApi.Service","packetType":"JobWrapper"},"Keywords":"0000000000000100","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"60826","Message":"Published a 'JobWrapper' packet to the https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd queue."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.861Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":333,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.7123666Z","Level":"Debug","EventName":"WebMethodCompletedInfo","Payload":{"activityIdentifier":"70877671-63a3-4904-9155-6a75d23f262b","httpMethod":"POST","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","clientIP":"","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs","methodName":"","duration":"00:00:00.034","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65201","Message":"<< POST http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs completed in 00:00:00.034."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.861Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":331,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.9915674Z","Level":"Debug","EventName":"WebMethodStartingInfo","Payload":{"activityIdentifier":"4f2b569d-62ce-4f7e-b793-bc47659fad83","httpMethod":"POST","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","requestUrl":"http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs","methodName":"","clientIP":"","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000010","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65203","Message":">> POST http://printscout.insights.hpondemand.com/printAgentCommunication/customers/dde5f939-263f-49ea-9408-c7ad1ca959f7/jobs starting."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.861Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":100,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.9918627Z","Level":"Debug","EventName":"ConfigurationValidateInfo","Payload":{"threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","configurationType":"AmazonSQSConfiguration","activityIdentifier":"4f2b569d-62ce-4f7e-b793-bc47659fad83","processName":"Pharos.Mps.PrintScoutApi.Service","jobIdentifier":"00000000-0000-0000-0000-000000000000"},"Keywords":"0000000000000001","Service":"Pharos.Mps.PrintScoutApi.Service","Task":"65434","Message":"Validating 'AmazonSQSConfiguration' configuration for correctness."}.{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.861Z","Version":0,"EventSource":"Pharos-Mps-Library","EventId":4707,"Opcode":"Info","Timestamp":"2022-07-25T12:21:04.0185028Z","Level":"Debug","EventName":"QueuePublishPacketInfo","Payload":{"threadId":11,"queueIdentifier":"https://sqs.us-west-2.amazonaws.com/992335167891/Jobs-HPProd","activityIdentifier":"4f2b569d-62ce-4f7e-b793-bc47659fad83","customerGuid":"00000000-0000-0000-0000-000000000000","jobIdentifier":"1356845a-d415-45f6-b8ec-12362ea79478","processName":"Pharos.Mps.PrintScoutApi.Service","packetType":"JobWrapper"},"Keywords":"0000000000000100","Service":"Pharos.Mps.PrintScout..."
00000098`d11ff524 int append_size = 0n6622
00000098`d11ff528 int required = 0n7022
0:007> dx -r1 ((fluent_bit!es_bulk *)0x1e2de300ff0)
((fluent_bit!es_bulk *)0x1e2de300ff0) : 0x1e2de300ff0 [Type: es_bulk *]
[+0x000] ptr : 0x1e2dbb0a720 : "{"create":{"_index":"logstash-2022.07.25","_type":"_doc"}}.{"@timestamp":"2022-07-25T12:21:08.859Z","Version":0,"EventSource":"Pharos-Mps-System","EventId":333,"Opcode":"Info","Timestamp":"2022-07-25T12:21:03.4787479Z","Level":"Debug","EventName":"WebMethodCompletedInfo","Payload":{"activityIdentifier":"ab6a18ba-cbb7-40e3-a579-934af2bbbfa1","httpMethod":"GET","threadId":4,"customerGuid":"00000000-0000-0000-0000-000000000000","clientIP":"","requestUrl":"http://printscout.insights.hpondema... [Type: char *]
[+0x008] len : 0x9e19 [Type: unsigned int]
[+0x00c] size : 0xb8dc [Type: unsigned int]
0:007> db 0x1e2dbb0a720 + 0xb8dc
000001e2`dbb15ffc 65 64 2e 22 ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ed."????????????
000001e2`dbb1600c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1601c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1602c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1603c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1604c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1605c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
000001e2`dbb1606c ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
fluent-bit-ods.log:
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: available=4096 required=942 required+i_len=1001
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: available=3260 required=1002 required+i_len=1061
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: available=2364 required=1087 required+i_len=1146
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: available=1383 required=1043 required+i_len=1102
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: available=446 required=840 required+i_len=899
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: whole_size=3585 converted_size=3005 bulk->size=4096 append_size=580
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: append_size=4096
[2022-07-25 11:51:33] [6a8.1654] [out_es] es_bulk_append: bulk->ptr=0x000001E2DCB79000 bulk->size=8192 (0x000001E2DCB7AFFF)
[2022-07-25 11:51:33] [6a8.1654] [out_es] cb_es_flush: done (okay)
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=40702 required=1180 required+i_len=1239
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=39628 required=1118 required+i_len=1177
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=38616 required=839 required+i_len=898
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=37883 required=1112 required+i_len=1171
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=36877 required=1162 required+i_len=1221
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=35821 required=1042 required+i_len=1101
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=34885 required=839 required+i_len=898
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=34152 required=942 required+i_len=1001
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=33316 required=1002 required+i_len=1061
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=32420 required=1087 required+i_len=1146
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=31439 required=1043 required+i_len=1102
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=30502 required=840 required+i_len=899
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=29768 required=941 required+i_len=1000
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=28933 required=1001 required+i_len=1060
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=28038 required=1086 required+i_len=1145
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=27058 required=1042 required+i_len=1101
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=26122 required=839 required+i_len=898
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=25389 required=942 required+i_len=1001
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=24553 required=1002 required+i_len=1061
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=23657 required=1087 required+i_len=1146
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=22676 required=1043 required+i_len=1102
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=21739 required=840 required+i_len=899
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=21005 required=941 required+i_len=1000
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=20170 required=1001 required+i_len=1060
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=19275 required=1086 required+i_len=1145
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=18295 required=1042 required+i_len=1101
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=17359 required=839 required+i_len=898
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=16626 required=942 required+i_len=1001
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=15790 required=1002 required+i_len=1061
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=14894 required=1087 required+i_len=1146
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=13913 required=1043 required+i_len=1102
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=12976 required=840 required+i_len=899
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=12242 required=1137 required+i_len=1196
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=11211 required=840 required+i_len=899
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=10477 required=941 required+i_len=1000
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=9642 required=1001 required+i_len=1060
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=8747 required=1086 required+i_len=1145
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=7767 required=1203 required+i_len=1262
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=6670 required=6488 required+i_len=6547
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: available=288 required=7022 required+i_len=7081
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: whole_size=40702 converted_size=34080 bulk->size=40702 append_size=6622
[2022-07-25 12:21:09] [6a8.1654] [out_es] es_bulk_append: bulk->ptr=0x000001E2DBB0A720 bulk->size=47324 (0x000001E2DBB15FFB)
(6a8.1654): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\PharosSystems\Tools\FluentBit\bin\fluent-bit.exe
fluent_bit!memcpy+0x2ef:
00007ff7`d09d9a6f c4a17e7f6c01e0 vmovdqu ymmword ptr [rcx+r8-20h],ymm5 ds:000001e2`dbb15fe1=2d
The output debug string (ods) log shows the sequence of reallocs leading up to the buffer overrun. This highlights the issues I try to address in #5770:
i_len should be added to required sizeappend_size should not be less than required sizeappend_size should convert to double before scaling
推荐文章
|
|
潇洒的课本 · Error 400 - Rejected by OpenSearch [error type]: mapper_parsing_exception - OpenSearch - OpenSearch 1 月前 |
|
|
谦和的蘑菇 · 中国科学院生物与化学交叉研究中心 7 月前 |
|
|
暴躁的四季豆 · [Wireless Router] How to access my ASUS router’s web GUI setting page via HTTPS? | Official Support 8 月前 |
|
|
英姿勃勃的鸡蛋面 · warning: Foam::dimensioned<Type>::dimensioned(Foam::Istrea m&) [with Type = double] -- CFD Online 11 月前 |
|
|
打篮球的荒野 · Search Results – Runners Shop 11 月前 |