We are seeing very high memory and CPU usage with the version of DHIS2 2.37. I have increased instance size to have 8 Processing cores with 40GB RAM; 32Gs available to Apache Tomcat for DHIS2. (Xmx 32G, Xms 32G)
Following is the screenshot of VisualVM monitoring. You can see the memory usage spiked to near 25GBs before a run of garbage collection brought it down to moderate 10GBs. Once the application tops the maximum value memory available to it it completely crashes and apache needs to be restarted.
DHIS2-Load
1329×609 86.6 KB
I am also attaching the output of Catalina from crash this morning along with DHIS2 logs. If someone can analyze and help us it would be greatly appreciated.
CATALINA
05-Mar-2022 10:39:18.626 SEVERE [https-jsse-nio-443-exec-111] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.642 SEVERE [https-jsse-nio-443-exec-99] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.751 SEVERE [https-jsse-nio-443-exec-107] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 11:37:53.338 INFO [Thread-17] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:54.400 INFO [Thread-17] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[email protected]/java.lang.Object.wait(Native Method)
[email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
[email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[email protected]/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[email protected]/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.385 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [Thread-75 (ActiveMQ-client-global-threads)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
[email protected]/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
[email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
05-Mar-2022 11:37:55.385 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@69810994]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5e16581e]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [SecurityContextImpl [Null authentication]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ab75fd4]) and a value of type [org.hibernate.boot.internal.MetadataImpl] (value [org.hibernate.boot.internal.MetadataImpl@31f1ba65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7377071f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1daf6b26]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4e1affc0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@54d7355f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fd39f25]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@794321a2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@603e25b5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@d2e9920]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@e070517]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6ea97b44]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7f2bf8a9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6b589f5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@30ac1b89]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@157a3583]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@35b47935]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fe4b457]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4fc49471]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@b566627]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1d3dcba9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@27418255]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.39
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Oct 6 2020 14:11:46 UTC
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.39.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 6.3
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: C:\Program Files\Java\jdk-11.0.13
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.13+10-LTS-370
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 9.0\temp
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 9.0\conf\logging.properties
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF8
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dlog4j2.formatMsgNoLookups=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9090
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.rmi.server.hostname=10.10.1.235
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [C:\Program Files\Apache Software Foundation\Tomcat 9.0\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\jdk-11.0.13\bin;C:\Program Files\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Java\jdk1.8.0_271\bin;C:\Program Files\PostgreSQL\12\bin;C:\Users\Administrator\AppData\Local\Programs\Python\Python39;;.]
05-Mar-2022 11:39:06.585 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.944 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1139] milliseconds
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.39]
05-Mar-2022 11:39:07.069 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war]
05-Mar-2022 11:39:29.272 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war] has finished in [63,765] ms
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager]
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager] has finished in [47] ms
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT]
05-Mar-2022 11:40:10.912 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT] has finished in [31] ms
05-Mar-2022 11:40:10.912 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:40:10.928 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [63989] milliseconds
DHIS2 Logs
* INFO 2022-03-05T09:32:03,269 Validation done for data set: 'GeQcXbMeHzo', period: '202202', org unit: 'f2i45ymtOdM', validation rule count: 17, violations found: 0 (ValidationAction.java [https-jsse-nio-443-exec-7])
* INFO 2022-03-05T09:32:03,363 Starting complete data set registration import, options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false}: 00:00:00.000 (Clock.java [https-jsse-nio-443-exec-51])
* INFO 2022-03-05T09:32:03,363 Import options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false} (DefaultCompleteDataSetRegistrationExchangeService.java [https-jsse-nio-443-exec-51])
* INFO 2022-03-05T09:32:03,363 Data set scheme: IdScheme{identifiableProperty=UID, attribute=null}, org unit scheme: IdScheme{identifiableProperty=UID, attribute=null}, attribute option combo scheme: IdScheme{identifiableProperty=UID, attribute=null} (ImportConfig.java [https-jsse-nio-443-exec-51])
* INFO 2022-03-05T09:32:14,878 Authentication event: AuthenticationSuccessEvent; username: ml-southwaziristan-de; ip: 39.43.67.102; sessionId: 888d19669421ba6f41987f113a4d9acc9b2f1d0e0d358ef23ec3654c49aa0cc6; (AuthenticationLoggerListener.java [https-jsse-nio-443-exec-43])
* INFO 2022-03-05T09:32:25,378 No template found (DefaultDataSetNotificationService.java [https-jsse-nio-443-exec-51])
* WARN 2022-03-05T09:32:56,674 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! (Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* WARN 2022-03-05T10:39:18,626 Java heap space (OrderedExecutor.java [Thread-618 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* WARN 2022-03-05T10:39:18,783 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@2068d50d<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@1ca32dcb [wrapping: null]> (LoadContexts.java [taskScheduler-11])
* WARN 2022-03-05T10:39:18,783 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-11])
* WARN 2022-03-05T10:39:18,829 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a7880a0<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@604c0338 [wrapping: null]> (LoadContexts.java [taskScheduler-1])
* WARN 2022-03-05T10:39:18,829 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-1])
* WARN 2022-03-05T10:39:18,845 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@508f7e0c<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@5a0471e3 [wrapping: null]> (LoadContexts.java [taskScheduler-13])
* WARN 2022-03-05T10:39:18,845 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-13])
* WARN 2022-03-05T10:39:18,861 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5ca0bc37<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@31778609 [wrapping: null]> (LoadContexts.java [taskScheduler-25])
* WARN 2022-03-05T10:39:18,861 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-25])
* ERROR 2022-03-05T10:39:18,861 AMQ214013: Failed to decode packet (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:18,861 AMQ224016: Caught exception (ServerSessionPacketHandler.java [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1b8d9495)])
java.lang.OutOfMemoryError: Java heap space
* WARN 2022-03-05T10:39:18,892 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5841c2d3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@63a8b06d [wrapping: null]> (LoadContexts.java [taskScheduler-10])
* WARN 2022-03-05T10:39:18,892 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-10])
* WARN 2022-03-05T10:39:18,908 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a850f80<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@27a91d20 [wrapping: null]> (LoadContexts.java [taskScheduler-21])
* WARN 2022-03-05T10:39:18,908 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-21])
* WARN 2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@756f116<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@3d6e51a3 [wrapping: null]> (LoadContexts.java [taskScheduler-23])
* WARN 2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-23])
* WARN 2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@7b8010e3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@7baaee4 [wrapping: null]> (LoadContexts.java [taskScheduler-24])
* WARN 2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-24])
* WARN 2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@59a9f946<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@712d3dfb [wrapping: null]> (LoadContexts.java [taskScheduler-2])
* WARN 2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@4bfab067<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@51f4daf4 [wrapping: null]> (LoadContexts.java [taskScheduler-20])
* WARN 2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-2])
* WARN 2022-03-05T10:39:25,754 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@39faa0ad<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@225a0ff6 [wrapping: null]> (LoadContexts.java [taskScheduler-14])
* WARN 2022-03-05T10:39:25,754 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-14])
* WARN 2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-20])
* WARN 2022-03-05T10:39:26,364 AMQ222061: Client connection failed, clearing up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN 2022-03-05T10:39:26,364 AMQ222107: Cleared up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN 2022-03-05T10:39:26,379 AMQ222218: Server disconnecting: Error decoding buffer (RemotingServiceImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.IllegalStateException: java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:390) ~[artemis-core-client-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:683) ~[artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:209) ~[artemis-server-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.17.0.jar:2.17.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) ~[artemis-commons-2.17.0.jar:2.17.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
Caused by: java.lang.OutOfMemoryError: Java heap space
* WARN 2022-03-05T10:39:26,379 AMQ212037: Connection failure to invm:0 has been detected: java.lang.OutOfMemoryError: Java heap space [code=GENERIC_EXCEPTION] (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN 2022-03-05T10:39:26,395 Setup of JMS message listener invoker failed for destination 'dhis2.jobs.tracker.notifications' - trying to recover. Cause: Session is closed (DefaultMessageListenerContainer.java [org.springframework.jms.JmsListenerEndpointContainer#4-1])
* WARN 2022-03-05T10:39:26,801 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@2789a43d
on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@75afb5ff
on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1398af81
on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b0c4707
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@314eb817
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@39c6f38c
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ff4a792
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5679e114
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@12b375cc
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1c67ae85
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@566552ac
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4f0d7835
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4619690d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@672be4cd
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1b273ab6
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@3f4f2537
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@321362d2
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@aa6f5d1
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@28768cb9
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ed6721d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@47fa98e7
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7acff5b4
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
(Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-20])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-14])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-23])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-24])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-2])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-21])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-25])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-10])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-13])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-1])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-11])
java.lang.OutOfMemoryError: Java heap space
* INFO 2022-03-05T10:39:27,333 Complete data set registration import done, total: 1, imported: 1, updated: 0, deleted: 0: 01:07:24.207 (Clock.java [https-jsse-nio-443-exec-51])
* ERROR 2022-03-05T10:39:27,333 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-5])
java.lang.OutOfMemoryError: Java heap space
at org.hibernate.property.access.spi.GetterMethodImpl.get(GetterMethodImpl.java:42) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:574) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:154) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:5174) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:207) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:164) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener$$Lambda$1147/0x00000017da9d7440.accept(Unknown Source) ~[?:?]
at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:230) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.internal.SessionImpl$$Lambda$1143/0x00000017da9d1040.accept(Unknown Source) ~[?:?]
at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1362) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:453) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3212) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2380) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:447) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:183) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300(JdbcResourceLocalTransactionCoordinatorImpl.java:40) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:281) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:643) ~[spring-orm-5.3.6.jar:5.3.6]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
at com.sun.proxy.$Proxy314.evaluateEventAndRunEffects(Unknown Source) ~[?:?]
at org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(ProgramRuleEngineListener.java:61) ~[dhis-service-program-rule-2.37.3.jar:?]
@janagha I tried both but still no data shows up in event reports, I tried to export the programs so that I can just reinstall but am getting the attached error. Not sure what could be causing this. image1576×706 66.9 KB
@WaluQ
Please try again in a different/new browser/profile (preferably Guest mode) Do you still see the issues? If not then it’s a browser cache issue, but if you still see this issue, and still see it even after clearing the server cache using the Administration app then would you please share the Catalina.out log (without the sensitive info)?
Are you using a superuser account? If not does this also appear to all the main admin accounts? Please mention the dhis2 version number if you share the log.
Thanks!
I am still getting the same response and behavior. Attached in the catalina log. I can’t seem to find the better way to attached the log here. DHIS2 Version 2.38.1.1 catalina log.docx (18.7 KB)
I am moving the posts to the post linked above because this new issue you posted is relevant to the memory leak issue.
Thank you!
@WaluQ,
I’m asking for support and here are some suggestions I received from @hendrik:
“To establish a definite cause of the app crashing, it’d be good to check the response to the GET request to /api/37/dataSets?paging=false”
Please let us know what response are you getting. This could also happen by checking the Network tab in your DevTools.
