By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Original thread here: https://discuss.elastic.co/t/server-2012-r2-some-metrics-not-send/51403

Version: TopBeat 1.2.3

Operating System: Windows Server 2012 R2 Standard (VMware host)

Steps to Reproduce:

Installed TopBeat via Powershell with Administrator Rights (right click, run as admin).

In processes, it is seen that TopBeat is owned by SYSTEM. Topbeat is also executed as SYSTEM.

The error in topbeat.log with debug level:

2016-05-31T11:00:13+02:00 DBG Skip process pid=0: error getting process state for pid=0: OpenProcess fails with The parameter is incorrect.
2016-05-31T11:00:13+02:00 DBG Skip process pid=4: error getting process state for pid=4: OpenProcess fails with Access is denied.
2016-05-31T11:00:13+02:00 DBG Windows is interactive: false

For convience, the full image:

It would be useful to know what processes are not captured. The low PIDs numbers make me think they are system processes.

@asednev To clarify, you get the error for some of the processes but most of the processes are captured?

Topbeat 1.3.1 I ran as windows service running under "Local System" account and also from command prompt "Run As Administrator". Neither worked.

Topbeat 1.2.3 mysteriously works.

To help debug the issue, could you use SysInternals - Process Explorer to examine the permissions of the running process.

I suspect the interesting privilege might be the SeDebugPrivilege.

This security policy could be related. https://technet.microsoft.com/en-us/library/cc976527.aspx?f=255&MSPPError=-2147217396

There is one place in gosigar where it requests PROCESS_ALL_ACCESS permission but the operation that it performs, GetProcessMemoryInfo , requires only PROCESS_QUERY_INFORMATION and PROCESS_VM_READ. gosigar should only request the permissions that it requires. Not sure if this would address the problem at hand, but in any case it's something that should be changed.

In other parts of gosigar it requests PROCESS_QUERY_INFORMATION, but in Windows Vista and newer there is a more limited privilege we can request called PROCESS_QUERY_LIMITED_INFORMATION. This can be used for GetProcessTimes , GetProcessImageFileName , and GetExitCodeProcess . It cannot be used for OpenProcessToken .

I haven't figured out what OS settings are causing the issue for these users. Some feedback is required (see my earlier request for SysInternals data).

I have made the changes mentioned in my previous comment. See elastic/gosigar#50 . We can put together a build of Topbeat/Metricbeat with these changes and see if the issue persists.

@csuka @asednev Would you be able to test the snapshot build of Metricbeat containing my changes on a machine that was having the "Access is denied" problem for all processes. BTW Metricbeat is replacing Topbeat in 5.0.

Access is denied is expected for some system processes, specifically the csrss.exe processes. And "incorrect parameter" is expected for PID 0. From the OpenProcess docs:

If the specified process is the System Process (0x00000000), the function fails and the last error code is ERROR_INVALID_PARAMETER. If the specified process is the Idle process or one of the CSRSS processes, this function fails and the last error code is ERROR_ACCESS_DENIED because their access restrictions prevent user-level code from opening them.

@andrewkroh , I tried metricbeat-6.0.0-alpha1-SNAPSHOT-windows-x86_64 and it still threw same errors:

2016/10/26 16:44:33.521779 helper.go:276: ERR Error getting process details pid=91076: error getting process mem for pid=91076: OpenProcess
fails with Access is denied.

2016/10/26 16:44:33.527793 helper.go:276: ERR Error getting process details pid=702776: error getting process mem for pid=702776: OpenProces
s fails with Access is denied.

I see system metrics logged to Elasticsearch but none of the process metrics.

I don't see any events with metricset.name:process in Elasticsearch. Only cpu, memory, filesystem, and network.

Actually the list of PID in the output was rather long. I only copied 2 as an example. I went through the list of PID, these are the processes that I'm looking to monitor - our application services.

Sounds like the changes I made didn't help. In the metricbeat download you used, can you tell me what the commit has is inside the .build_hash.txt file. Just want to confirm that the version has my changes.

Are you aware of any local or group security policies that were tightened on this machine? The reason I ask is that it would be nice to be able to reproduce this locally so I don't have to take up your valuable time.

Could you run Sysinternals - ProcExp (like I did in #1897 (comment) ) and check the privileges of Metricbeat?

There is one other change that I think could help. It basically involves escalating the Metricbeat processes access token to include the SeDebugPrivilege , like what is described in https://stackoverflow.com/questions/2932461/windows-vista-win7-privilege-problem-sedebugprivilege-openprocess .

@asednev I added code locally to metricbeat that tries to add SeDebugPrivilege to the process' access token. It's just test code at the moment and should print "AdjustTokenPrivileges success" to stdout at start. This version also has a bit more info in the error messages that will help identify the call that failed due to "Access is denied".

https://s3.amazonaws.com/beats-files/metricbeat.exe

If you are willing to try this binary and report the output this would be very helpful. Thanks

- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

- Fixes elastic#53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes elastic#6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

- Fixes #53 (ProcStatus.PPID value is wrong on Windows). The code was incorrectly using `CreateToolhelp32Snapshot` + `Process32First`. This could have been fixed, but it would have required iterating over each process until finding the PPID of process we were interested it. Instead the code has been changed to use `NtQueryInformationProcess` to get the PPID given a process token.
- Fixes #6 (Get the cpu usage per core in Windows). I used `NtQuerySystemInformation` to collect the timing information on a per CPU basis.
- Adds OS version checks to the functions that make certain WMI calls. The `Win32_Process` data is only available on Vista and newer. On XP and Win2003, these methods will return `ErrNotImplemented`. This will help address elastic/beats#1704.
- Implements `Uptime.Get` for Windows.
- Implements `Swap.Get` for Windows based on page file metrics.
- Removes cgo usage for Windows.
- Adds support to `github.com/gosigar/sys/windows` for querying and enabling privileges in a process token. This will help in addressing elastic/beats#1897.

- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes #2860 (PPID is zero on Windows).
- Fixes #1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes #1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes #2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.

The snapshot build now has the fix for this if you'd like to test it out. https://beats-nightlies.s3.amazonaws.com/index.html?prefix=winlogbeat/

If you look in the log output there will be some lines containing "SeDebugPrivilege" that provide some feedback. In your case, @asednev , you should see "SeDebugPrivilege is now enabled." .

- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.
(cherry picked from commit eb88c2c)

- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes #2860 (PPID is zero on Windows).
- Fixes #1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes #1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes #2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.
(cherry picked from commit eb88c2c)

- Added system core metricset for Windows. Per core metrics were implemented
  in elastic/gosigar.
- Added logging of process/system details on Windows to aid in debugging (user,
  arch, cores, sid, privs).
- Fixes elastic#2860 (PPID is zero on Windows).
- Fixes elastic#1704 (Server 2003 - PID’s not recognized). The command line arguments
  for the process will not be reported on XP and 2003.
- Fixes elastic#1897 (OpenProcess access denied on Windows). Added code to enable the
  SeDebugPrivilege when it is available.
- Fixes elastic#2885 (diskio metricset fails on XP and 2003).
- Enabled fsstats by default in Metricbeat config.
(cherry picked from commit eb88c2c)