Chinaunix首页 | 论坛 | 博客
  • 博客访问: 760662
  • 博文数量: 259
  • 博客积分: 10
  • 博客等级: 民兵
  • 技术积分: 2518
  • 用 户 组: 普通用户
  • 注册时间: 2012-10-13 16:12
个人简介

科技改变世界,技术改变人生。

文章分类

全部博文(259)

分类: 大数据

2016-03-30 14:52:08

启动spark-sql
spark-sql --master yarn-client  --executor-memory 1G

报错:
16/03/30 14:45:32 WARN BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

使用命令查看databases
spark-sql> show databases;

16/03/30 14:46:01 ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.hadoop.compression.lzo.GPLNativeCodeLoader.(GPLNativeCodeLoader.java:32)
        at com.hadoop.compression.lzo.LzoCodec.(LzoCodec.java:71)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1834)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1799)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:175)
        at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

16/03/30 14:46:01 ERROR LzoCodec: Cannot load native-lzo without native-hadoop

解决方法:
spark-env.sh中增加
export SPARK_SUBMIT_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_SUBMIT_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_LIBRARY_PATH
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

后来尝试了下,保留以下两个即可:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

参考:
http://find.searchhub.org/document/a128707a98fe4ec6
http://lotso.blog.51cto.com/3681673/1441737
http://guoyunsky.iteye.com/blog/1237327
阅读(3636) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册