[2017-03-23 17:09:35.432] clan WARN [DUBBO] An exception was thrown by a user handler while handling an exception event ([id: 0xa9483a99, /10.160.37.164:36657 => /10.25.0.15:31100] EXCEPTION: com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process received event .), dubbo version: 2.8.4, current host: 10.25.0.15 [New I/O worker #8] org.jboss.netty.channel.DefaultChannelPipeline.warn(62) com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process caught event . at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.caught(AllChannelHandler.java:67) at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44) at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44) at com.alibaba.dubbo.remoting.transport.AbstractPeer.caught(AbstractPeer.java:127) at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.exceptionCaught(NettyHandler.java:112) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.exceptionCaught(NettyCodecAdapter.java:165) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:148) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.RejectedExecutionException: Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.25.0.15:31100, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 4783 (completed: 4583), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.25.0.15:31100! at com.alibaba.dubbo.common.threadpool.support.AbortPolicyWithReport.rejectedExecution(AbortPolicyWithReport.java:53) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.caught(AllChannelHandler.java:65) ... 19 common frames omitted
线程池耗尽,内存溢出,错误也很明显。
1 2 3 4
23-Mar-2017 18:56:31.121 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [clan] appears to have started a thread named [DubboResponseTimeoutScanTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan.run(DefaultFuture.java:300) java.lang.Thread.run(Thread.java:745)
[2017-03-28 21:59:54.451] clan INFO [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 10.25.0.15 [DubboShutdownHook] com.alibaba.dubbo.config.AbstractConfig.info(42) [2017-03-28 21:59:54.452] clan INFO [DUBBO] Close all registries [zookeeper://10.175.206.177:2182/com.alibaba.dubbo.registry.RegistryService?application=clan&backup=10.117.18.169:2182,10.160.4.48:2182&client=curator&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&organization=dongjia&owner=dongjia&pid=14956×tamp=1490703425648], dubbo version: 2.8.4, current host: 10.25.0.15 [DubboShutdownHook] com.alibaba.dubbo.registry.support.AbstractRegistryFactory.info(42)
② shutdown worker【最终关闭dubbo进程】
1 2 3 4 5 6 7 8 9 10 11
[2017-03-28 21:59:54.853] clan WARN [DUBBO] An exception was thrown by an exception handler., dubbo version: 2.8.4, current host: 10.25.0.15 [DubboClientReconnectTimer-thread-1] org.jboss.netty.channel.DefaultChannelPipeline.warn(62) java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:115) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:73) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:57) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54)
[附录2]
① 先挂掉的服务
1
[2017-03-28 21:59:54.451] clan INFO [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 10.25.0.15 [DubboShutdownHook] com.alibaba.dubbo.config.AbstractConfig.info(42)
○ shutdown hook的时间是 2017-03-28 21:59:54.451。
②后挂掉的服务
1
[2017-03-28 22:00:21.146] clan INFO [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 10.252.111.151 [DubboShutdownHook] com.alibaba.dubbo.config.AbstractConfig.info(42)