Hi All,
We have a web services application which is built on
java
and Spring WS and hosted on two nodes on the production environment. The application is running under TC server version "vfabric-tc-server-standard-2.6.3.RELEASE" and we are using java version "jdk1.6.0_24".
We have configured the
JDBC
data source connection pool using the factory "org.apache.tomcat.jdbc.pool.DataSourceFactory". The application was running fine for few days and suddenly on one of the nodes we started encountering "Protocol Violation" exception for some of the requests and it is the same case on the QA environment as well. The issue is only one of the nodes and on the other node the applicaiton is running fine with the same configurations, same software and hardware. BTW, the application is running fine on development without any issues and there we do not have any RAC configuration.
Mainly the underlying message of the exception says
1. SQL state [99999]; error code [17412]; Bigger type length than Maximum; nested exception is java.sql.SQLException: Bigger type length than Maximum
2. SQL state [99999]; error code [17402]; OALL8 is in an inconsistent state
We have checked many forums and tried lot of options but still could not resolve the issue. We are using the ojdbc driver version "11.2.0.3.0" and the back-end database is Oracle which is 11g 11.2.0.3.0.
Here are the details of the environment and also on the database side we do have RAC configured and there are two physical instances connected through 3 scan IPs.
1- Unix OS 64 bit
2- Oracle 11.2.0.3.0
3- JRE 1.6.24
4- Ojdbc1.6 jar (compatible with oracle 11.2.0.3.0)
5- Spring source tc server 2.6.3
Here is the configuration of the data source in server.xml
<Resource defaultAutoCommit="false" defaultReadOnly="false"
driverClassName="oracle.jdbc.OracleDriver"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
fairQueue="false" initialSize="10"
jdbcInterceptors="ConnectionState;StatementFinalizer"
jmxEnabled="true" logAbandoned="false" maxActive="200"
maxIdle="100" maxWait="30000"
minEvictableIdleTimeMillis="5000" minIdle="10"
name="jdbc/test" password="XXXXXXXXXXXXXXXXXXXXXX"
removeAbandoned="false" removeAbandonedTimeout="60"
testOnBorrow="true" testOnReturn="false"
testWhileIdle="false" timeBetweenEvictionRunsMillis="5000"
type="javax.sql.DataSource"
url="jdbc: oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SCANIP)(PORT=1521))(FAILOVER=ON)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=HIDDEN)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=10)(DELAY=1))))"
useEquals="false" username="HIDDEN"
validationInterval="30000" validationQuery="SELECT 1 FROM DUAL"/>
The exception details are as below.
2013-02-02 17:48:41,010 - ERROR [tomcat-http--18] (JDBCExceptionReporter.java:78) - logExceptions()
Bigger type length than Maximum
2013-02-02 17:48:41,107 - ERROR [tomcat-http--18] (JDBCTransaction.java:168) - rollback()
JDBC rollback failed
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:271)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:445)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:130)
at oracle.jdbc.driver.T4CConnection.doRollback(T4CConnection.java:626)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3470)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
2013-02-02 17:49:40,607 - ERROR [tomcat-http--14] (JDBCExceptionReporter.java:78) - logExceptions()
OALL8 is in an inconsistent state
2013-02-02 17:49:40,610 - ERROR [tomcat-http--14] (ApplicationLogger.java:208) - logError()
[Method: logResponseTime][org.springframework.dao.DataAccessResourceFailureException: Hibernate operation: could not get next sequence value; SQL [select SEQIPERFDASHBOARDID.nextval from dual]; OALL8 is in an inconsistent state; nested exception is java.sql.SQLException: OALL8 is in an inconsistent state
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:253)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:424)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:410)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:737)
at com.adidas.scv.framework.datalayer.HibernateDataAccessObject.saveOrUpdate(HibernateDataAccessObject.java:83)
We already tried following options
1. Recreated the TC server instance
2. Upgraded ojdbc version from 11.2.0.2.0 to 11.2.0.3.0
3. Firewall monitoring to check if any packets are lost
4. On QA, directly connected to one of the database instance instead of using SCANIP
5. Upgraded java to 1.7 update 13
Please let me know if anyone has encountered similar issue and has found any solution.
Regards,
Sri
Do you want ants? Because that's how you get ants. And a tiny ads:
We need your help - Coderanch server fundraiser
https://coderanch.com/wiki/782867/Coderanch-server-fundraiser
