I am attempting to integrate a third-party application with an existing log4j implementation into Splunk.  I have what I beleive should be a working appender configuration in my log4j.properties file.  However, when my Tomcat server starts I receive the below index out of bounds error.  I am using logging library version 1.9.0. I'm looking for advice on where to look in order to resolve this.  I have included the appender config for reference.

APPENDER CONFIG:

appender.splunkHEC=com.splunk.logging.HttpEventCollectorLog4jAppender
appender.splunkHEC.name=splunkHEC
appender.splunkHEC.layout=org.apache.log4j.PatternLayout
appender.splunkHEC.layout.ConversionPattern=%d{ISO8601} [%t] %p %c %x - %m%n
appender.splunkHEC.url=<redacted>
appender.splunkHEC.token=<redacted>
appender.splunkHEC.index=ioeng
appender.splunkHEC.source=IIQ_Tomcat
appender.splunkHEC.sourceType=log4j
appender.splunkHEC.batch_size_count=100
appender.splunkHEC.disableCertificateValidation=true

RELEVANT JAVA STACK:

Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 9
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
at java.base/java.lang.String.substring(String.java:1874)
at org.apache.logging.log4j.util.PropertiesUtil.partitionOnCommonPrefixes(PropertiesUtil.java:555)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.build(PropertiesConfigurationBuilder.java:156)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:56)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:35)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:557)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:481)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:716)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:270)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)
at org.apache.logging.log4j.jcl.LogAdapter.getContext(LogAdapter.java:40)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
at org.apache.logging.log4j.jcl.LogFactoryImpl.getInstance(LogFactoryImpl.java:40)
at org.apache.logging.log4j.jcl.LogFactoryImpl.getInstance(LogFactoryImpl.java:55)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
at sailpoint.web.StartupContextListener.<clinit>(StartupContextListener.java:59)

SERVER DETAILS:

20-Mar-2024 11:52:03.882 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.64
20-Mar-2024 11:52:03.883 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jun 2 2022 19:08:46 UTC
20-Mar-2024 11:52:03.884 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.64.0
20-Mar-2024 11:52:03.884 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
20-Mar-2024 11:52:03.885 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.10.0-1160.108.1.el7.x86_64
20-Mar-2024 11:52:03.886 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
20-Mar-2024 11:52:03.886 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/java/jdk-11.0.22
20-Mar-2024 11:52:03.887 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.22+9-LTS-219
20-Mar-2024 11:52:03.887 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation

@kidderjc - I'm no Java expert based on my past experience with log4j to Splunk HEC. If Splunk fails for some reason your solution will encounter a memory issue and may crash.

My Recommendation : Store logs to log files on the server and use Splunk UF to forward the logs to Splunk indexers.

I hope this helps!!!

Splunk, Splunk>, Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.