Too many open files in web.log - jetty problem - 🏅 Community Support

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

闯红灯的冰棍 · Untouch推移动端3D手势交互 ...· 2 月前 ·

空虚的南瓜 · 请用python ...· 4 月前 ·

高大的电影票 · ChristineB - 影视搜索 - 好看网· 8 月前 ·

从容的绿豆 · qt里面实现自定义标题栏 ...· 9 月前 ·

宽容的野马 · 用于检测小儿创伤患者颈椎损伤的分诊工具 - ...· 11 月前 ·

Problem:
After migration to new hardware platform and new system (Centos 7 to Centos 9) and set new system limits Jetty server doesn’t start or crash with too many open files alarm.
New system params:
Hardware:
Two Intel Xeon Gold 6336Y CPU @ 2.40GHz
256GB RAM
Storage: RAID10 on NVMe Samsung drives

Software:
Centos 9
Limits for file descriptors: hard 100000, soft 50000
Java: OpenJDK 64-Bit Server VM version 11.0.18+10-LTS

On startup I set in opennms.conf:
JAVA_HEAP_SIZE=150000
MAXIMUM_FILE_DESCRIPTORS=50000
RUNAS=root

OpenNMS daemons started and worked, but Jetty didn’t start - too many open files.

I changed settings in opennms.conf
JAVA_HEAP_SIZE=150000
MAXIMUM_FILE_DESCRIPTORS=30000
RUNAS=root

OpenNMS daemons started, Jetty started, but after some time Jetty crashed with too many open files.

Now, I changed settings to:
JAVA_HEAP_SIZE=150000
MAXIMUM_FILE_DESCRIPTORS=20000
RUNAS=root

All deamons are worked, but how long??? I don’t know…

I heve about 12000 nodes, 70000 interfaces and 32000 services in OpenNMS monitoring environment.

Expected outcome:
I thint that is only Jetty problem. What can I do to set higher value of MAXIMUM_FILE_DESCRIPTORS and get Jetty working?

OpenNMS version:
OpenNMS 33.0.3

Other relevant data:

2024-04-19 08:00:53,067 WARN  [qtp150522667-23695-acceptor-1@79c54f26-ServerConnector@480500dd{HTTP/1.1, (http/1.1)}{0.0.0.0:8980}] o.e.j.s.AbstractConnector:
java.io.IOException: Too many open files
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704) [jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
        at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-04-19 08:00:53,067 WARN  [qtp150522667-23693-acceptor-0@499de329-ServerConnector@480500dd{HTTP/1.1, (http/1.1)}{0.0.0.0:8980}] o.e.j.s.AbstractConnector:
java.io.IOException: Too many open files
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704) [jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
        at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
2024-04-19 08:00:53,067 WARN  [qtp150522667-23694-acceptor-2@525b8290-ServerConnector@480500dd{HTTP/1.1, (http/1.1)}{0.0.0.0:8980}] o.e.j.s.AbstractConnector:
              Can you double-check what limits are applied to the Java process when OpenNMS is running?
We have instructions on the page

    How to investigate file descriptor issues 🧞‍♀️ Knowledge Base
    If you run a centralized monitoring system in a large environment you can run in some issues regarding file descriptor limits. Linux gives you the detailed information in the kernel control and information center in /proc. The soft and hard limits have an effect on file and network sockets, which can end up in too many files open exceptions in OpenNMS. This might also take into account when you run Minions or Sentinels in large environments. 
What is the difference between the soft- and hard lim…
Are there any reasons for running OpenNMS as root? As you have 33.0.3 it was changed to running as a non-privileged user. Can you share also the content of your executed systemd unit with:
systemctl show opennms.service
              I checked limits in proc:
# cat /proc/2696831/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              8388608              bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             1027363              1027363              processes
Max open files            20000                20000                files
Max locked memory         8388608              8388608              bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       1027363              1027363              signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
[root@opennms etc]# cat /etc/security/limits.conf
*         hard    nofile      100000
*         soft    nofile      50000
root      hard    nofile      100000
root      soft    nofile      50000
opennms   hard    nofile      100000
opennms   soft    nofile      50000
systemctl show opennms.service
[root@opennms etc]# systemctl show opennms.service
Type=forking
ExitType=main
Restart=no
PIDFile=/run/opennms/opennms.pid
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
TimeoutAbortUSec=1min 30s
TimeoutStartFailureMode=terminate
TimeoutStopFailureMode=terminate
RuntimeMaxUSec=infinity
RuntimeRandomizedExtraUSec=0
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=2696831
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=0
GID=0
NRestarts=0
OOMPolicy=stop
ReloadSignal=1
ExecMainStartTimestamp=Fri 2024-04-19 08:08:51 CEST
ExecMainStartTimestampMonotonic=254566154546
ExecMainExitTimestampMonotonic=0
ExecMainPID=2696831
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/opt/opennms/bin/opennms ; argv[]=/opt/opennms/bin/opennms -s start ; ignore_errors=no ; start_time=[Fri 2024-04-19 08:08:46 CEST] ; stop_time=[Fri 2024-04-19 08:08:48 CEST] ; pid=2695613 ; code=exited ; status=0 }
ExecStartEx={ path=/opt/opennms/bin/opennms ; argv[]=/opt/opennms/bin/opennms -s start ; flags= ; start_time=[Fri 2024-04-19 08:08:46 CEST] ; stop_time=[Fri 2024-04-19 08:08:48 CEST] ; pid=2695613 ; code=exited ; status=0 }
ExecStartPost={ path=/bin/sleep ; argv[]=/bin/sleep 3 ; ignore_errors=no ; start_time=[Fri 2024-04-19 08:08:48 CEST] ; stop_time=[Fri 2024-04-19 08:08:51 CEST] ; pid=2696832 ; code=exited ; status=0 }
ExecStartPostEx={ path=/bin/sleep ; argv[]=/bin/sleep 3 ; flags= ; start_time=[Fri 2024-04-19 08:08:48 CEST] ; stop_time=[Fri 2024-04-19 08:08:51 CEST] ; pid=2696832 ; code=exited ; status=0 }
ExecStop={ path=/opt/opennms/bin/opennms ; argv[]=/opt/opennms/bin/opennms -s stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecStopEx={ path=/opt/opennms/bin/opennms ; argv[]=/opt/opennms/bin/opennms -s stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/opennms.service
ControlGroupId=9829
MemoryCurrent=72393314304
MemoryAvailable=infinity
CPUUsageNSec=30689879584000
TasksCurrent=2499
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=no
CPUAccounting=yes
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=1643780
IPAccounting=no
ManagedOOMSwap=auto
ManagedOOMMemoryPressure=auto
ManagedOOMMemoryPressureLimit=0
ManagedOOMPreference=none
Environment=OPENNMS_HOME=/opt/opennms
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=524288
LimitNOFILESoft=1024
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=1027363
LimitNPROCSoft=1027363
LimitMEMLOCK=8388608
LimitMEMLOCKSoft=8388608
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=1027363
LimitSIGPENDINGSoft=1027363
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
CoredumpFilter=0x33
Nice=0
IOSchedulingClass=2
IOSchedulingPriority=4
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_>
AmbientCapabilities=cap_net_bind_service cap_net_raw
User=root
DynamicUser=no
RemoveIPC=no
PrivateTmp=no
PrivateDevices=no
ProtectClock=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectKernelLogs=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
PrivateMounts=no
PrivateIPC=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=2147483646
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0775
RuntimeDirectory=opennms
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
TimeoutCleanUSec=infinity
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictSUIDSGID=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
ProtectProc=default
ProcSubset=all
ProtectHostname=no
KillMode=control-group
KillSignal=15
RestartKillSignal=15
FinalKillSignal=9
SendSIGKILL=yes
SendSIGHUP=no
WatchdogSignal=6
Id=opennms.service
Names=opennms.service
Requires=network-online.target sysinit.target system.slice network.target -.mount
Wants=postgresql.service
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=network-online.target network.target postgresql-13.service systemd-journald.socket postgresql-12.service basic.target system.slice sysinit.target postgresql.service postgresql-11.service postgresql-10.service -.mount
RequiresMountsFor=/run/opennms
Description=OpenNMS server
LoadState=loaded
ActiveState=active
FreezerState=running
SubState=running
FragmentPath=/usr/lib/systemd/system/opennms.service
UnitFileState=enabled
UnitFilePreset=disabled
StateChangeTimestamp=Fri 2024-04-19 08:08:51 CEST
StateChangeTimestampMonotonic=254566154604
InactiveExitTimestamp=Fri 2024-04-19 08:08:46 CEST
InactiveExitTimestampMonotonic=254560658975
ActiveEnterTimestamp=Fri 2024-04-19 08:08:51 CEST
ActiveEnterTimestampMonotonic=254566154604
ActiveExitTimestamp=Fri 2024-04-19 08:03:56 CEST
ActiveExitTimestampMonotonic=254271014457
InactiveEnterTimestamp=Fri 2024-04-19 08:04:36 CEST
InactiveEnterTimestampMonotonic=254310871205
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
CanClean=runtime
CanFreeze=yes
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnSuccessJobMode=fail
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2024-04-19 08:08:46 CEST
ConditionTimestampMonotonic=254560658203
AssertTimestamp=Fri 2024-04-19 08:08:46 CEST
AssertTimestampMonotonic=254560658203
Transient=no
Perpetual=no
StartLimitIntervalUSec=10s
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=d18bc90d8c914900b19107b825504f2e
CollectMode=inactive
              It seems like the MAXIMUM_FILE_DESCRIPTORS=20000 is applied regarding the limits shown from the process ID cat /proc/2696831/limits.
Can you change the MAXIMUM_FILE_DESCRIPTORS=65000 and double-check if the process gets these new limits?
Can you disable Enlinkd and reduce the JAVA_HEAP_SIZE to 64GB. We have collected some recommednations for larger deployments in  OpenNMS in large environments
              I tested many settings and I got running configuration.

I had to set new system limits:
[root@opennms etc]# cat /etc/security/limits.conf
*         hard    nofile      400000
*         soft    nofile      300000
root      hard    nofile      400000
root      soft    nofile      300000
opennms   hard    nofile      400000
opennms   soft    nofile      300000
I set new value of MAXIMUM_FILE_DESCRIPTORS=200000 in opennms.conf
OpenNMS works without to many open files error, but in Centos 9 file descriptors consumption is higher than in Centos 7.
[root@opennms etc]# ls /proc/296797/fd | wc -l
116989
              As I have seen you are using 33.0.3, you can use OpenJDK 17 which should help you with dealing with large JVM heap. Have you changed the OpenJDK from CentOS 7 to CentOS 9 as well? Just switching the underlying OS and seeing an increase in file handles using the same OpenNMS version and configuration makes this an interesting observation.
Thank you for sharing your experience on this topic.
              With java 17 OpenNMS works better then with java 11 under Centos 9.

With java 17 I see lower file descriptors consumption ( about 77000).

Garbage collector works better too. It works automatic heap memory cleaning. CPU usage is lower about 20%.