We encountered one issue in the last production change and getting an exception on the UI while submitting enrollments to downstream system.
Please find the steps which we performed.
-
Deleted the Database instance as this is duplicate and having database connection string, username / Password hard coded in this instance and we have already PEGA_RULES with JNDI connection.
-
After that we bounced all Pega servers.
-
As part of testing killed one database instance to check the application availability and database server automatically restarting or not? (we have a RAC node solution in the Database)
-
Oracle server restarted successfully and traffic routed to the second instance successfully.
-
We were able to login into application, and able to create enrollments & at final submission process to downstream system getting an exception on the UI <<Implementation class name>> Java.lang.NoClasssDefFoundError:Com/Pega/apache/log4J/spi/ThrowableInformation
Question:-
-
We need to bounce Pega Serves bounce or not ? after the oracle instance restarted.
Note:-
Observed this error in the log files while testing.
<< Security Access Error: Fail: No credentials supplied >>
Thanks,
Veera Illuri.
***Edited by Moderator Marissa to update platform capability tags***
Hi Veera,
If you could provide the complete stack trace/PegaRULES_log for the above error will be helpful. The exception could be as a result of failure to load the engine classes(ThrowableInformation and StackTrace are common classes to preload). It is recomended to restart the a prpc instance on restarting the database instance.
The PDN article might help here:
https://community.pega.com/support/support-articles/exception-while-processing-application-and-agent-failure
Regards,
Basavaraj
Thanks for your response Basavarj. We have a RAC node solution from the Database side, as part of this work we are testing the PROD application availability. Incase if the first instance went down, it should automatically connect to the second database instance (This step is successfully tested)
After killing the 1
st
database instance the overall application is okay to connect (The application relies on database connections) only issue when submitting this application at final step to downstream systems.
Questions here, even the database is available to connect, Pega servers bounce is required?
Technically what happened was that another exception occurred right before the
NoClasssDefFoundError was experienced hence the engine tried to log the stack trace of such initial exception.
In order to do so the logging framework tried to load the definition of the
Com/Pega/apache/log4J/spi/ThrowableInformation class which lead to the error observed.
Please note that quite frequently such error is preceded by a java.lang.ClassNotFoundException and another exception which is the issue root cause. E.g.:
java.lang.NoClassDefFoundError: SomeClass
at somepackage.SomeOtherClass.aMethod(SomeOtherClass.java:10)
Caused by: java.lang.ClassNotFoundException: SomeClass
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
So it's always useful to scroll the logs backwards to search for the root cause of the issue.
In this case given the DB outage and considered that Pega loads engine classes from the database you should be able to identify a database related root cause.
Beside that please follow Basavaraj recommendations and restart Pega nodes after database outages/restarts.
Hi Veera,
Are you saying that your DB failover appears to have worked fine, in the sense that the connection was re-established, however, when you tested the application end to end, you saw some failure with submitting your applications.
And you are asking, if you should also restart the app server?
If so, are you asking because you notice the issue goes away after restart?
Hi Rett,
Yes connection was re-established to second database instance successfully and application is available to create enrollments, issue is while submitting to downstream system getting an error on the UI Java.lang.NoClassDefFound:Com/Pega/apache/log4j/spi/ThrowableInformation
After Pega servers restarted we are able to submit to application successfully without any issues.
now question is even the applications is available and database connection is established why servers bounce is required? and also normally in which scenario's this exception is generating.
@VeeraRaghavaI
,
If you end up opening an SR, please reply here with the SR ID so that we may track for you.
Thank you!
About Pegasystems
Pegasystems is the leader in cloud software for customer engagement and operational excellence. If you’ve driven a car, used a credit card, called a company for service, opened an account, flown on a plane, submitted a claim, or performed countless other everyday tasks, chances are you’ve interacted with Pega. For the past 30 years, our technology – CRM, digital process automation, robotics, AI, and more – has empowered the world’s leading companies to achieve breakthrough results.
Hi Veera,
If you could provide the complete stack trace/PegaRULES_log for the above error will be helpful. The exception could be as a result of failure to load the engine classes(ThrowableInformation and StackTrace are common classes to preload). It is recomended to restart the a prpc instance on restarting the database instance.
The PDN article might help here: https://community.pega.com/support/support-articles/exception-while-processing-application-and-agent-failure
Regards,
Basavaraj
Thanks for your response Basavarj. We have a RAC node solution from the Database side, as part of this work we are testing the PROD application availability. Incase if the first instance went down, it should automatically connect to the second database instance (This step is successfully tested)
After killing the 1 st database instance the overall application is okay to connect (The application relies on database connections) only issue when submitting this application at final step to downstream systems.
Questions here, even the database is available to connect, Pega servers bounce is required?
Technically what happened was that another exception occurred right before the NoClasssDefFoundError was experienced hence the engine tried to log the stack trace of such initial exception.
In order to do so the logging framework tried to load the definition of the Com/Pega/apache/log4J/spi/ThrowableInformation class which lead to the error observed.
Please note that quite frequently such error is preceded by a java.lang.ClassNotFoundException and another exception which is the issue root cause. E.g.:
java.lang.NoClassDefFoundError: SomeClass at somepackage.SomeOtherClass.aMethod(SomeOtherClass.java:10) Caused by: java.lang.ClassNotFoundException: SomeClass at java.net.URLClassLoader$1.run(URLClassLoader.java:372)So it's always useful to scroll the logs backwards to search for the root cause of the issue.In this case given the DB outage and considered that Pega loads engine classes from the database you should be able to identify a database related root cause.
Beside that please follow Basavaraj recommendations and restart Pega nodes after database outages/restarts.
Hi Veera,
Are you saying that your DB failover appears to have worked fine, in the sense that the connection was re-established, however, when you tested the application end to end, you saw some failure with submitting your applications.
And you are asking, if you should also restart the app server?
If so, are you asking because you notice the issue goes away after restart?
Hi Rett,
Yes connection was re-established to second database instance successfully and application is available to create enrollments, issue is while submitting to downstream system getting an error on the UI Java.lang.NoClassDefFound:Com/Pega/apache/log4j/spi/ThrowableInformation
After Pega servers restarted we are able to submit to application successfully without any issues.
now question is even the applications is available and database connection is established why servers bounce is required? and also normally in which scenario's this exception is generating.
@VeeraRaghavaI ,
If you end up opening an SR, please reply here with the SR ID so that we may track for you.
Thank you!
About Pegasystems
Pegasystems is the leader in cloud software for customer engagement and operational excellence. If you’ve driven a car, used a credit card, called a company for service, opened an account, flown on a plane, submitted a claim, or performed countless other everyday tasks, chances are you’ve interacted with Pega. For the past 30 years, our technology – CRM, digital process automation, robotics, AI, and more – has empowered the world’s leading companies to achieve breakthrough results.