原创

Hadoop Hbase 问题记录

Performance

  1. 为rawlog表,修改data model,使得raw key的设计,对时间部分取模(Mod(timestamp), TTL_IN_DAYS),并且设置一个巨大的MAX_FILESIZE。固定每个region的范围,防止做split,防止split带来的隐患。 (参考http://blog.linezing.com/2012/11/hbase-how-to-resolve-not-serving-region-exception

  2. 设置hbase.hregion.majorcompaction=0禁掉majorcompact,手动在空闲时做majorcompact (参考http://blogread.cn/it/wap/article.php?id=5635)

  3. 设置hbase.hstore.compactionThreshold=Integer.MAX关闭minorcompact (参考http://blogread.cn/it/wap/article.php?id=5635)

  4. 设置合适的hfile.block.cache.size for读性能 (参考http://blogread.cn/it/wap/article.php?id=5635)


Plan

1.production关闭major compact,添加cronjob手动compact

2.production环境将hadoop hbase的log导入logging系统

3.在uat测试rawlog表增加snappy压缩

 

Development

1.查看最热的region

2.查看最大的region


Exception

1、2015-11-27 14:05:08,274 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater

java.lang.reflect.UndeclaredThrowableException
 at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
 at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:72)
 at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:352)
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "SVR2364HP360/192.168.83.96"; destination host is: "SVR2364HP360":28025; 
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
 at com.sun.proxy.$Proxy23.nodeHeartbeat(Unknown Source)
 at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:70)
 ... 1 more
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "SVR2364HP360/192.168.83.96"; destination host is: "SVR2364HP360":28025; 
 at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
 at org.apache.hadoop.ipc.Client.call(Client.java:1242)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
 ... 3 more
Caused by: java.io.IOException: Response is null.
 at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:952)
 at org.apache.hadoop.ipc.Client$Connection.run(Client.java:847)

  

2、2015-11-27 13:59:18,010 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/hadoop/hadoop/name is in an inconsistent state: storage directory does not exist or is not accessible.
       at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:302)
       at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:207)
       at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:728)
       at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:521)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:445)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:621)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:606)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1177)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241)

 

 3、2015-11-27 14:05:08,274 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater

java.lang.reflect.UndeclaredThrowableException
       at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
       at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:72)
       at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:352)
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "SVR2364HP360/192.168.83.96"; destination host is: "SVR2364HP360":28025;
       at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
       at com.sun.proxy.$Proxy23.nodeHeartbeat(Unknown Source)
       at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:70)
       ... 1 more
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "SVR2364HP360/192.168.83.96"; destination host is: "SVR2364HP360":28025;
       at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
       at org.apache.hadoop.ipc.Client.call(Client.java:1242)
       at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
       ... 3 more
Caused by: java.io.IOException: Response is null.
       at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:952)
       at org.apache.hadoop.ipc.Client$Connection.run(Client.java:847)

 

 

4、2015-11-27 14:01:48,056 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint

java.net.ConnectException: Call From SVR2364HP360/192.168.83.96 to SVR2364HP360:9200 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
       at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
       at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
       at org.apache.hadoop.ipc.Client.call(Client.java:1242)
       at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
       at com.sun.proxy.$Proxy7.getTransactionId(Unknown Source)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:606)
       at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
       at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
       at com.sun.proxy.$Proxy7.getTransactionId(Unknown Source)
       at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:117)
       at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:597)
       at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:605)
       at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:353)
       at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:321)
       at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:452)
       at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:317)
       at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
       at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
       at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
       at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
       at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:510)
       at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:604)
       at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
       at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
       at org.apache.hadoop.ipc.Client.call(Client.java:1209)
       ... 17 more

 

 


关注下方微信公众号“Java精选”(w_z90110),回复关键字领取资料:如HadoopDubboCAS源码等等,免费领取资料视频和项目。 

涵盖:程序人生、搞笑视频、算法与数据结构、黑客技术与网络安全、前端开发、Java、Python、Redis缓存、Spring源码、各大主流框架、Web开发、大数据技术、Storm、Hadoop、MapReduce、Spark、elasticsearch、单点登录统一认证、分布式框架、集群、安卓开发、iOS开发、C/C++、.NET、Linux、Mysql、Oracle、NoSQL非关系型数据库、运维等。

评论

分享:

支付宝

微信