问题情况

当访问服务端口时mina无法分配可用线程到新的连接中.

线程状态

通过arthas检查线程状态确认到mina线程池将所有线程分配了,mina初始化时设置的线程池参数为100个线程,全部进入到TIMED_WAITING中

 1"pool-13-thread-22" prio=10 tid=0x00007f0c1410c000 nid=0x62ad in Object.wait() [0x00007f0cf6cf9000]
 2   java.lang.Thread.State: TIMED_WAITING (on object monitor)
 3	at java.lang.Object.wait(Native Method)
 4	- waiting on <0x00000007842d8c50> (a org.apache.mina.core.future.DefaultWriteFuture)
 5	at org.apache.mina.core.future.DefaultIoFuture.await0(DefaultIoFuture.java:192)
 6	- locked <0x00000007842d8c50> (a org.apache.mina.core.future.DefaultWriteFuture)
 7	at org.apache.mina.core.future.DefaultIoFuture.awaitUninterruptibly(DefaultIoFuture.java:133)
 8	at org.apache.mina.core.future.DefaultWriteFuture.awaitUninterruptibly(DefaultWriteFuture.java:112)
 9	at org.apache.mina.handler.stream.IoSessionOutputStream.flush(IoSessionOutputStream.java:84)
10	- locked <0x00000007842d8c78> (a org.apache.mina.handler.stream.IoSessionOutputStream)
11	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
12	- locked <0x00000007842d8c90> (a java.io.BufferedOutputStream)
13	......

追究根本原因

mina的IoSessionOutputStream.flush方法,实际为等待无限长时间,直到mina将IoBuffer中的数据全部发送完毕后对等待进行唤醒

但是存在一个检查服务存活的程序,负责每隔几秒钟对服务进行连接测试,连接上就断开,程序发送欢迎信息还没有结束就断开了连接,造成mina无法将存储在IoBuffer中的信息发送,也就无法唤醒阻塞等待,造成线程卡死

IoSessionOutputStream.flush

 1@Override
 2    public synchronized void flush() throws IOException {
 3        if (lastWriteFuture == null) {
 4            return;
 5        }
 6
 7        lastWriteFuture.awaitUninterruptibly();
 8        if (!lastWriteFuture.isWritten()) {
 9            throw new IOException("The bytes could not be written to the session");
10        }
11    }

DefaultIoFuture.await0

 1/**
 2     * Wait for the Future to be ready. If the requested delay is 0 or
 3     * negative, this method immediately returns the value of the
 4     * 'ready' flag.
 5     * Every 5 second, the wait will be suspended to be able to check if
 6     * there is a deadlock or not.
 7     * 
 8     * @param timeoutMillis The delay we will wait for the Future to be ready
 9     * @param interruptable Tells if the wait can be interrupted or not
10     * @return <tt>true</tt> if the Future is ready
11     * @throws InterruptedException If the thread has been interrupted
12     * when it's not allowed.
13     */
14    private boolean await0(long timeoutMillis, boolean interruptable) throws InterruptedException {
15        long endTime = System.currentTimeMillis() + timeoutMillis;
16
17        if (endTime < 0) {
18            endTime = Long.MAX_VALUE;
19        }
20
21        synchronized (lock) {
22            // We can quit if the ready flag is set to true, or if
23            // the timeout is set to 0 or below : we don't wait in this case.
24            if (ready||(timeoutMillis <= 0)) {
25                return ready;
26            }
27
28            // The operation is not completed : we have to wait
29            waiters++;
30
31            try {
32                for (;;) {
33                    try {
34                        long timeOut = Math.min(timeoutMillis, DEAD_LOCK_CHECK_INTERVAL);
35                        
36                        // Wait for the requested period of time,
37                        // but every DEAD_LOCK_CHECK_INTERVAL seconds, we will
38                        // check that we aren't blocked.
39                        lock.wait(timeOut);
40                    } catch (InterruptedException e) {
41                        if (interruptable) {
42                            throw e;
43                        }
44                    }
45
46                    if (ready || (endTime < System.currentTimeMillis())) {
47                        return ready;
48                    } else {
49                        // Take a chance, detect a potential deadlock
50                        checkDeadLock();
51                    }
52                }
53            } finally {
54                // We get here for 3 possible reasons :
55                // 1) We have been notified (the operation has completed a way or another)
56                // 2) We have reached the timeout
57                // 3) The thread has been interrupted
58                // In any case, we decrement the number of waiters, and we get out.
59                waiters--;
60                
61                if (!ready) {
62                    checkDeadLock();
63                }
64            }
65        }
66    }