We are seeing below SEVERE Message on Daily basis on our four node Forked Molecule(Cluster of JVM's).

We are using CA APM to monitor the Atom JVM's (where the CA APM Agent is Installed on each node and added properties of CA APM in atom.vmoptions(JVM Properties file where we add CA APM configuration Properties).

Could you please brief us what is causing this error and Is there any timeout value which is causing to throw Connection Refused to Host

<Month Date>, <Year> <Time> <Time Zone> SEVERE [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff
java.rmi.ConnectException: Connection refused to host: <IP Addresss> of Head Node; nested exception is:
java.net.ConnectException: Connection refused

Stack Trace
<Month Date>, <Year> <Time> <Time Zone> SEVERE [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff
java.rmi.ConnectException: Connection refused to host: <IP Addresss> of Head Node; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:906)
at com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:122)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)
at com.sun.proxy.$Proxy50.getId(Unknown Source)
at com.boomi.execution.forker.SunForkedProcessHandler$3.executeAction(SunForkedProcessHandler.java:372)
at com.boomi.execution.forker.SunForkedProcessHandler$3.executeAction(SunForkedProcessHandler.java:368)
at com.boomi.execution.forker.SunForkedProcessHandler$ForkedProcRetryable.execute(SunForkedProcessHandler.java:572)
at com.boomi.util.retry.Retryable.call(Retryable.java:92)
at com.boomi.execution.forker.SunForkedProcessHandler.executeRunnerAction(SunForkedProcessHandler.java:389)
at com.boomi.execution.forker.SunForkedProcessHandler.interrogateVm(SunForkedProcessHandler.java:368)
at com.boomi.execution.forker.SunForkedProcessHandler.getKnownRunners(SunForkedProcessHandler.java:296)
at com.boomi.execution.forker.SunForkedProcessHandler.getExecution(SunForkedProcessHandler.java:263)
at com.boomi.execution.forker.SunForkedProcessHandler.isExecuting(SunForkedProcessHandler.java:148)
at com.boomi.execution.forker.SunForkedProcessHandler.isExecuting(SunForkedProcessHandler.java:142)
at com.boomi.execution.forker.MultiProcessExecutionForker.reapAbortedProcesses(MultiProcessExecutionForker.java:770)
at com.boomi.execution.forker.MultiProcessExecutionForker.isLocalIdle(MultiProcessExecutionForker.java:668)
at com.boomi.container.cloudlet.execution.ClusterExecutionForker.isLocalIdle(ClusterExecutionForker.java:489)
at com.boomi.execution.ExecutionManager.isLocalIdle(ExecutionManager.java:736)
at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at javax.management.StandardMBean.getAttribute(StandardMBean.java:372)
at com.boomi.util.management.CustomMBean.getAttribute(CustomMBean.java:138)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at com.wily.introscope.agent.jmx.AJMXDataView.IJMXDataView_update(AJMXDataView.java:115)
at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.pollViews(IntroscopeProtocolAdaptor.java:659)
at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.findServersAndPollViews(IntroscopeProtocolAdaptor.java:549)
at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.setContextAndPoll(IntroscopeProtocolAdaptor.java:434)
at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.access$8(IntroscopeProtocolAdaptor.java:419)
at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor$IntroscopeProtocolAdaptorHeartbeatBehavior.ITimestampedRunnable_execute(IntroscopeProtocolAdaptor.java:1410)
at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)
at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)
at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)
at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 54 more

It has to do more with connectivity than a timeout value.  Basically this excpetion means that there is either no service listening on the port or something is blocking the traffic.  Are there any firewalls enabled and blocking the traffic?  Is the port set up correctly?  Are you using any JMX authentication method?

we have a four node cluster of JVM's ..For all Four Nodes (5 to 6 times only we are seeing this connection refused errors.

If the port is not listening at all or wrong firewall rules the issue should exist all the time in a day, but we are seeing issue 04 to 05 times a day with different timings (no specific pattern of time).

When CA APM lost communication  does it post any errors in APM log why it is not able to access?

The intermittent nature of the problem may point to Linux/Unix file descriptors being occasionally exhausted for the  jvm/java process and "ulimit -n" may need to be increased for the java process owner id to allow for the extra requirements of the agent.  However as suggested by Matt musma03 a related error should be visible in the agent debug log e.g. "java.io.IOException: Too many open files"

I have Just checked from APM for  past 7 days OpenFileDescriptorCount on all four nodes. No host crossed 1200 open files also. we have a limit of 6000

JMX - java.lang - type= OperatingSystem - OpenFileDescriptorCount

First Node:

JMX|
java.lang|
type=OperatingSystem:
OpenFileDescriptorCount
Value = 1,138.00
Min = 292.00
Max = 1,138.00
Count = 672

2016 Oct 19 4:00:00 PM

Second Node:

JMX|
java.lang|
type=OperatingSystem:
OpenFileDescriptorCount
Value = 261.00
Min = 244.00
Max = 261.00
Count = 672

2016 Oct 19 6:48:00 PM

Third Node:

JMX|
java.lang|
type=OperatingSystem:
OpenFileDescriptorCount
Value = 264.00
Min = 252.00
Max = 264.00
Count = 672

2016 Oct 19 7:36:00 AM

Fourth Node:

Molecule_Prod_EMB_Molecule|
JMX|
java.lang|
type=OperatingSystem:
OpenFileDescriptorCount
Value = 273.00
Min = 262.00
Max = 273.00
Count = 672

2016 Oct 21 1:36:00 AM

Oct 23, 2016 7:01:26 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host:  (Third Node)
Oct 23, 2016 8:00:51 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)
Oct 23, 2016 9:21:54 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)
Oct 23, 2016 10:00:51 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)

Same issue I have posted to our product vendor Dell boomi, I got the below update

https://community.boomi.com/message/11089

most likely exacerbated by your use of the "isLocalIdle" method for monitoring.  As i mentioned in my previous comment, i wouldn't recommend this method for monitoring.  If you can tell me why you are calling it, maybe there is a better metric to monitor instead.

OOTB the agent should not be explicitly monitoring method com.boomi.execution.ExecutionManager.isLocalIdle aso that must be indirect.

You could try to disable socket tracing to see if that helps i.e. comment the line "TurnOn: SocketTracing" in file toggles-typical.pbd or toggles-full.pbd whichever is being used. If no help please raise a support case for more detailed advice as suggested above.

Hiko_Davis for other suggestions.

Thanks

Lynn