C3P0 APPARENT DEADLOCK!!!

服务运行一段时间后，突然就不可能一直超时，
查看日志发现有大量的数据库线程池死锁APPARENT DEADLOCK
报错的主要类是ThreadPoolAsynchronousRunner 这个类是C3P0包下的
在 C3P0PooledConnectionPoolManager 类在初始化时中创建了一个taskRunner，其类型为 ThreadPoolAsynchronousRunner ， 是一个 C3P0 自定义实现的线程池

int matt = this.getMaxAdministrativeTaskTime( null );
        if ( matt > 0 )
        {
            int matt_ms = matt * 1000;
            this.taskRunner = new ThreadPoolAsynchronousRunner( num_task_threads, 

                            true,        // daemon thread

                            matt_ms,     // wait before interrupt()

                            matt_ms * 3, // wait before deadlock declared if no tasks clear

                            matt_ms * 6, // wait before deadlock tasks are interrupted (again)
                            // after the hung thread has been cleared and replaced
                            // (in hopes of getting the thread to terminate for
                            // garbage collection)

                            timer );
        }
        else
            this.taskRunner = new ThreadPoolAsynchronousRunner( num_task_threads, true, timer );

ThreadPoolAsynchronousRunner 线程池被作为成员注入到多个对象之中，最终用于执行 BasicResourcePool 类中定义的 ScatteredAcquireTask， AcquireTask， RemoveTask， DestroyResourceTask， RefurbishCheckinResourceTask， AsyncTestIdleResourceTask 等任务
ThreadPoolAsynchronousRunner 的代码

this.num_threads = num_threads;
        this.daemon = daemon;
        this.max_individual_task_time = max_individual_task_time;
        this.deadlock_detector_interval = deadlock_detector_interval;
        this.interrupt_delay_after_apparent_deadlock = interrupt_delay_after_apparent_deadlock;
        this.myTimer = myTimer;
        this.should_cancel_timer = should_cancel_timer;

        recreateThreadsAndTasks();

        myTimer.schedule( deadlockDetector, deadlock_detector_interval, deadlock_detector_interval );

可以看到， ThreadPoolAsynchronousRunner 线程池默认创建3个执行线程，并且创建了一个 死锁检测（DeadLockDetector） 的定时任务，默认情况下每10秒执行一次
DeadLockDetector内部机制

class DeadlockDetector extends TimerTask
    {
        LinkedList last = null;
        LinkedList current = null;

        public void run()
        {
            boolean run_stray_tasks = false;
            synchronized ( ThreadPoolAsynchronousRunner.this )
            { 
                if (pendingTasks.size() == 0)
                {
                    last = null;
                    return;
                }

                current = (LinkedList) pendingTasks.clone();
                if ( current.equals( last ) )
                { 
if ( logger.isLoggable( MLevel.WARNING ) )
                    {
                        logger.warning(this + " -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!");
                        StringWriter sw = new StringWriter( 4096 );
                        PrintWriter pw = new PrintWriter( sw );

DeadLockDetector是
ThreadPoolAsynchronousRunner 的内部类，是一个默认每10秒执行一次的定时任务。其工作原理是定时检查任务队列（pendingTasks）中的任务，如果前后两次检查发现待执行的任务没有变化，就认为可能产生了死锁，并额外创建线程执行等待任务

从代码中可以看出，APPARENT DEADLOCK!!!报错原因主要是ThreadPoolAsynchronousRunner线程池的中的等待任务队列在经过一定时长（默认10秒）前后没有变化

所以修改数据库配置文件的空闲检测周期和连接最大空闲时间，默认是下面的都是一分钟
60
60
修改为空闲检测周期5分钟，连接最大空闲时间5小时
300
18000