Solved: R80.40 snmpv3 - Check Point CheckMates

We have just upgraded our MDS servers and VSX clusters from R80.30 to R80.40 with JHF take48. We monitored the devices and virtual servers using SNMPv3. After upgrading to R80.40, our monitoring server, which is Solarwinds, cannot connect to the VSX clusters and virtual servers. However, it can connect to the MDS servers fine.

If I enabled SNMP v2 on the VSX gateways, our Solarwinds server can connect to them. They just cannot connect using v3. Has anyone seen this? Is it likely to be a bug?

Thanks
Roy

I had an similar issue with snmpd on an R80.40-JHF Take 48 VSX-Cluster.
The snmpd settings in GAIA are "mode vs" and "vs-direct-access off".

When i add an snmpv3 USM-User in GAIA the access with this User fails with "unknown user" after a some seconds.
I think the reason for this is that the multiple snpmd processes for VS0 and the virtual systems are overwriting the current engineID from the net-snmp persistent file /var/lib/net-snmp/snmpd.conf and then it does not match the engineID of the USM-User.
(The engineID ist random.)

I could only solve this with

1.) Stopping th snmpd-agent and all snmpd processes.
2.) Deleting /var/lib/net-snmp/snmpd.conf
3.) Deleting the USM-User in GAIA
4.) Setting the following entries in /etc/snmp/userDefinedSettings.conf.
engineIDType 3
engineIDNic Mgmt
5.) Adding the USM-User in GAIA
6.) Starting the agent
7.) Checking the file /var/lib/net-snmp/snmpd.conf (matching engineID)

Hi Martin

The SNMP settings are below. Other than using mode vs and vs-direct-access, all other settings are the same as on the MDS servers:

set snmp mode vs
set snmp vs-direct-access on
set snmp agent on
set snmp agent-version any
add snmp traps receiver ***** version v3
add snmp usm user ***** security-level authPriv auth-pass-phrase-hashed ***** privacy-pass-phrase-hashed ***** privacy-protocol AES authentication-protocol SHA1
set snmp traps trap authorizationError disable
set snmp traps trap biosFailure enable
set snmp traps trap clusterXLFailover disable
set snmp traps trap coldStart enable
set snmp traps trap configurationChange disable
set snmp traps trap configurationSave disable
set snmp traps trap fanFailure enable
set snmp traps trap highVoltage enable
set snmp traps trap linkUpLinkDown enable
set snmp traps trap lowDiskSpace enable
set snmp traps trap lowVoltage enable
set snmp traps trap overTemperature enable
set snmp traps trap powerSupplyFailure enable
set snmp traps trap raidVolumeState disable
set snmp traps trap vrrpv2AuthFailure disable
set snmp traps trap vrrpv2NewMaster disable
set snmp traps trap vrrpv3NewMaster disable
set snmp traps trap vrrpv3ProtoError disable
set snmp contact *****
set snmp location *****
set snmp traps advanced coldStart reboot-only off

Jan

You are correct, the line was missing, probably due to me deleting and recreating the snmp user as part of my troubleshooting. We have 2 VSX clusters that this happened on, and the line was still there on the other cluster.

Anyway, when I put the line

set snmp usm user ***** vsid 0-13

I get the message:

NMSSNM9999 Timeout waiting for response from database server.

And when I check the config, I see the line as:

set snmp usm user ***** vsid 0-4

If I run the command again, I get the same error but then the config says vsid 0-9. And I'm still unable to connect from the Solarwinds server.

I followed the instructions in sk168180. This seems to work until I switch back to snmp mode VS. In default mode, I can successfully run snmpwalk and get a connection from our monitoring server to the VSX chassis, i.e. VS 0. But when I run "set snmp mode vs", snmpwalk fails with the same "snmpwalk: Unknown user name" message and the monitoring server loses connection.

If I leave it in default mode, I cannot monitor the virtual systems, which is what we need to do.

I had an similar issue with snmpd on an R80.40-JHF Take 48 VSX-Cluster.
The snmpd settings in GAIA are "mode vs" and "vs-direct-access off".

I could only solve this with

**bleep**, got the same issue after our upgrade too, take 78. Tried @Friedrich_Recht method but did not work for me, still different engineIDs

@assafav - sk168180 is not available I'm afraid

This is pretty bad as we will be blind about box performance when morning comes!

Bingo you saved my day (night!) @Friedrich_Recht !

Had to add one additional step - when creating user, snmp should be in default mode, not vs. So:

5.) set snmp mode default

6.) Adding the USM-User in GAIA (don't forget to grant access to VSes)

7.) Starting the agent

8.) set snmp mode vs

9.) Checking the file /var/lib/net-snmp/snmpd.conf (matching engineID)

Guys

I'm afraid these steps have not resolved my issue. I followed them and now I have "snmp mode vs" I can monitor VS0, which I could not before. However, I still cannot connect to the other virtual systems. When I look at the snmpd.conf file EngineID for the user and the oldEngineID are the same.

I'm still on JHF Take 48, but interesting to know the issue happens with Take 78 as well. I'm still waiting for this to get fixed.

Thanks
Roy

Yapp, just created new test users and all is working good on T78 after implementing initial fix described above

Setting the following entries in /etc/snmp/userDefinedSettings.conf.
engineIDType 3
engineIDNic Mgmt

These are steps that worked for us (the order is fundamental):

# clish
> set snmp agent off
> exit
# mv /var/lib/net-snmp/snmpd.conf /home/admin
# clish
> delete snmp usm user youruser
> save config
> exit
# vim /etc/snmp/userDefinedSettings.conf
engineIDType 3
engineIDNic Mgmt
# clish
> set snmp mode default
> set snmp agent on
> set snmp mode vs
> add snmp usm user youruser security-level authNoPriv auth-pass-phrase yourpassword authentication-protocol MD5
> set snmp usm user youruser vsid all
> save config
> exit

Hi Johan,

A few questions to better understand the problem:

Are you getting the 'unknown user' error?

Does the error happen after deleting and re-adding the snmp user?

Does the file /var/lib/net-snmp/snmpd.conf get updated when deleting and re-adding the snmp user?

Can you please share the content of the file /var/lib/net-snmp/snmpd.conf after deleting and re-adding the snmp user?

Finally got around to installing JHF89 but I still cannot get snmp monitoring of my virtual systems. I've now gone through the deletion/addition of snmp user many times with no luck. I can monitor VS0 and not the virtual systems.

I noticed sk168180 is not available any more, which is curious.

I'm not too familiar with using snmpwalk but when I try the following:

snmpwalk -v3 -l authPriv -u vsxsnmp -a SHA1 -A *** -x AES -X *** <IP or localhost> 1.3.6.1.2.1

I get "Timeout: No Response from localhost"

Any additional help would be appreciated

Thanks
Roy

The fix for the 'unknown user' error was integrated in R80.40 JHF as of take 91.

sk168180 was not effective, therefore was deleted.

As for the timeout issue, does it happen in snmp default mode as well as vs mode?