java.lang.IllegalStateException: unread block data
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |
Spark 通过HIVE ON HBASE表读取数据源报错：unread block data
具体错误信息示例如下：
18/03/19 20:46:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
18/03/19 20:46:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)问题原因分析：该问题查了好长时间(开始以为是数据源有问题；后来各种查、各种试发现是缺少jar包因为是CDH环境通过增加spark/jars下面的jar即可：
cd /opt/...../CDH/spark/jars/
ln -s /opt/...../CDH/jars/hive-hbase-handler-***.jar hive-hbase-handler***.jar总结：引起该问题的原因有很多最好由简入繁、从最简单直接的原因查起逐一排除提高排查问题效率。
相关的解决方案:
方法一：spark-submit --jars
根据spark官网在提交任务的时候指定--jars用逗号分开。这样做的缺点是每次都要指定jar包如果jar包少的话可以这么做但是如果多的话会很麻烦。可以封装一个sh脚本做目录/引用文件扫描也可以。
spark-submit --master yarn-client --jars ***.jar,***.jar(你的jar包用逗号分隔) myjar.jar方法二：extraClassPath
提交时在spark-default中设定参数将所有需要的jar包考到一个文件里然后在参数中指定该目录就可以了较上一个方便很多：
spark.executor.extraClassPath=/extlib/*
spark.driver.extraClassPath=/extlib/*
# 修改为自己的目录方法二：fat-jar
如果你还是觉得第二种麻烦这种方法是将所有依赖的jar,包括你写的代码全部打包在一起（fat-jar就是提交的时候比较慢毕竟大啊。还有可能和系统jar冲突。自己注意解决把。
参考链接：