1 背景

线上Hive任务偶尔出现hang住的现象.
经排查确认是触发了Hive的bug, 该bug在Hive-10569( https://issues.apache.org/jira/browse/HIVE-10569 中已修复.

2 现象

用户beeline任务运行至Ended Job之后,一直未结束,查看yarn日志,该job已经运行成功.

INFO  : 2020-03-20 20:25:21,401 Stage-36 map = 100%,  reduce = 0%, Cumulative CPU 3.96 sec
INFO  : 2020-03-20 20:25:30,224 Stage-36 map = 100%,  reduce = 100%, Cumulative CPU 9.44 sec
INFO  : MapReduce Total cumulative CPU time: 9 seconds 440 msec
INFO  : Ended Job = job_1582793079899_13676666INFO  : 2020-03-20 20:25:21,401 Stage-36 map = 100%,  reduce = 0%, Cumulative CPU 3.96 sec
INFO  : 2020-03-20 20:25:30,224 Stage-36 map = 100%,  reduce = 100%, Cumulative CPU 9.44 sec
INFO  : MapReduce Total cumulative CPU time: 9 seconds 440 msec
INFO  : Ended Job = job_1582793079899_13676666

3 排查步骤

3.1 查看hiveserver2日志

找到该job在hive server端的日志,没有发现任何异常.

3.2 抓取现场

分别通过 jstack -l <pid> 和 jmap -dump:[live,]format=b,file=<filename> <pid> 获取server端的堆栈信息.

3.3 定位排查

首先, 确定session ID .

 根据 job id 定位到该MR对应的hive session id(session id会在hive conf中记录);
然后, 确定线程ID.  通过MAT分析工具打开之前获取到的heap文件, 根据session id查找到对应的HiveSessionImpl对象, 进而追踪到对应的线程ID;
最后, 确定线程栈. 在jstack文件中找到对应的线程,从而定位到具体的线程栈.
本案例中线程栈如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
"HiveServer2-Background-Pool: Thread-10663039" #10663039 prio=5 os_prio=0 tid=0x00007fcd94c87000 nid=0x213ed in Object.wait() [0x00007fcbec2fb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.hive.ql.DriverContext.pollFinished(DriverContext.java:118)
        - locked <0x00007fd29eff91d0> (a org.apache.hadoop.hive.ql.DriverContext)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1574)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:157)
        at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:209)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1923)
        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

3.4 源码剖析
查看源码位置, 分析可知runner.isRunning状态未更新会导致hang住不结束.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/**
 * Polls running tasks to see if a task has ended.
 *
 * @return The result object for any completed/failed task
 */
public synchronized TaskRunner pollFinished() throws InterruptedException {
  while (!shutdown) {
    Iterator<TaskRunner> it = running.iterator();
    while (it.hasNext()) {
      TaskRunner runner = it.next();
      if (runner != null && !runner.isRunning()) {
        it.remove();
        return runner;
      }
    }
    wait(SLEEP_TIME);
  }
  return null;
}

# 4 解决方案
查找社区issue发现, 该bug已经在1.3版本中进行了修复.(https://issues.apache.org/jira/browse/HIVE-10569
后续 backport 该patch即可.
            缺失模块。
1、请确保node版本大于6.2
2、在博客根目录（注意不是yilia根目录）执行以下命令：
 npm i hexo-generator-json-content --save


            3、在根目录_config.yml里添加配置：
  jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true