Hadoop的distcp命令出現(xiàn)Permission denied錯(cuò)誤
來(lái)源：程序員人生 發(fā)布時(shí)間：2015-06-08 08:25:51 閱讀次數(shù)：8868次
Hadoop的distcp命令可以實(shí)現(xiàn)將文件從1個(gè)hdfs文件系統(tǒng)中拷貝到另外1個(gè)文件系統(tǒng)中，以下所示：
$ bin/hadoop distcp -overwrite hdfs://123.123.23.111:9000/hsd/t_url hdfs://123.123.23.156:9000/data/t_url
正常情況下應(yīng)當(dāng)出現(xiàn)以下運(yùn)行結(jié)果：
Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
/tmp/hsperfdata_hugetable/16744
Try using the -Djava.io.tmpdir= option to select an alternate temp location.
15/04/29 20:35:07 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs://192.168.34.135:9000/zyx/t_url], targetPath=hdfs://192.168.34.156:9000/data/t_url, targetPathExists=false, preserveRawXattrs=false}
15/04/29 20:35:07 INFO client.RMProxy: Connecting to ResourceManager at compute⑵3-06.local/192.168.34.135:8032
15/04/29 20:35:08 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
15/04/29 20:35:08 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
15/04/29 20:35:08 WARN conf.Configuration: bad conf file: element not <property>
15/04/29 20:35:08 WARN conf.Configuration: bad conf file: element not <property>
15/04/29 20:35:08 INFO client.RMProxy: Connecting to ResourceManager at compute⑵3-06.local/192.168.34.135:8032
15/04/29 20:35:09 INFO mapreduce.JobSubmitter: number of splits:21
15/04/29 20:35:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1429262156603_0032
15/04/29 20:35:10 INFO impl.YarnClientImpl: Submitted application application_1429262156603_0032
15/04/29 20:35:10 INFO mapreduce.Job: The url to track the job: http://compute⑵3-06.local:8088/proxy/application_1429262156603_0032/
15/04/29 20:35:10 INFO tools.DistCp: DistCp job-id: job_1429262156603_0032
15/04/29 20:35:10 INFO mapreduce.Job: Running job: job_1429262156603_0032
15/04/29 20:35:21 INFO mapreduce.Job: Job job_1429262156603_0032 running in uber mode : false
15/04/29 20:35:21 INFO mapreduce.Job: map 0% reduce 0%
15/04/29 20:35:32 INFO mapreduce.Job: map 10% reduce 0%
15/04/29 20:35:33 INFO mapreduce.Job: map 18% reduce 0%
15/04/29 20:35:34 INFO mapreduce.Job: map 25% reduce 0%
……
但是我在運(yùn)行的進(jìn)程中出現(xiàn)了Permission denied毛病，具體以下：
Error: java.io.IOException: File copy failed: hdfs://192.168.34.135:9000/zyx/t_url/000031_0 --> hdfs://192.168.34.156:9000/data/000031_0
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://192.168.34.135:9000/zyx/t_url/000031_0 to hdfs://192.168.34.156:9000/data/000031_0
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280)
... 10 more
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=hugetable,
access=WRITE, inode="/data":root:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5584)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5566)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5540)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2282)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2235)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2188)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:505)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
從命令中可以很明顯的看出來(lái)，是權(quán)限不夠，“supergroup:drwxr-x-r-x”，通過(guò)hadoop fs -ls /來(lái)查看，可見(jiàn)對(duì)組外的其他用戶(hù)的確沒(méi)有寫(xiě)權(quán)限（drwxr-xr-x的最后3個(gè)是組外的其他用戶(hù)權(quán)限，目前只有r:read和x:executable兩個(gè)權(quán)限）。可見(jiàn)用戶(hù)對(duì)hdfs文件沒(méi)有寫(xiě)權(quán)限。
可以通過(guò)chmod 來(lái)修改權(quán)限：
具體以下：
$ hadoop fs -chmod 777 /lgh
$ hadoop fs -chmod 777 /data
上述代碼是給hdfs文件/lgh和/data兩個(gè)文件夾賦予777權(quán)限（777權(quán)限即所有用戶(hù)的所有權(quán)限），運(yùn)行完了再次履行hadoop fs /可見(jiàn)權(quán)限已全部賦予上去了。
全文完，轉(zhuǎn)載請(qǐng)注明出處：http://blog.csdn.net/ghuil/article/details/45372469
生活不易，碼農(nóng)辛苦
如果您覺(jué)得本網(wǎng)站對(duì)您的學(xué)習(xí)有所幫助,可以手機(jī)掃描二維碼進(jìn)行捐贈(zèng)
