HDFS基础
1. hdfs shell基础
[root@master hadoop]# hadoop fs
usage: hadoop fs [generic options]
[-appendtofile <localsrc> ... <dst>]
[-cat [-ignorecrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-r] group path...]
[-chmod [-r] <mode[,mode]... | octalmode> path...]
[-chown [-r] [owner][:[group]] path...]
[-copyfromlocal [-f] [-p] [-l] <localsrc> ... <dst>]
[-copytolocal [-p] [-ignorecrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createsnapshot <snapshotdir> [<snapshotname>]]
[-deletesnapshot <snapshotdir> <snapshotname>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignorecrc] [-crc] <src> ... <localdst>]
[-getfacl [-r] <path>]
[-getfattr [-r] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-r] [<path> ...]]
[-mkdir [-p] <path> ...]
[-movefromlocal <localsrc> ... <dst>]
[-movetolocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renamesnapshot <snapshotdir> <oldname> <newname>]
[-rm [-f] [-r|-r] [-skiptrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-r] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-r] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignorecrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]
查看某个命令的具体帮助信息:
[root@master hadoop]# hadoop fs -help test
-test -[defsz] <path> :
answer various questions about <path>, with result via exit status.
-d return 0 if <path> is a directory.
-e return 0 if <path> exists.
-f return 0 if <path> is a file.
-s return 0 if file <path> is greater than zero bytes in size.
-z return 0 if file <path> is zero bytes in size, else return 1.
这里尤其要注意,如果文件存在,返回结果是0
我们上传一个文件,然后用test命令测试:
[root@master hadoop]# hadoop fs -put /root/test test.txt
[root@master hadoop]# hadoop fs -ls .
found 4 items
drwxrwxrwx - hdfs hdfs 0 2018-02-10 22:22 quasimontecarlo_1518319340789_698036166
drwxrwxrwx - hdfs hdfs 0 2018-02-10 23:21 quasimontecarlo_1518322909671_1083050937
-rw-r--r-- 3 root hdfs 5 2019-02-08 21:52 test
-rw-r--r-- 3 root hdfs 5 2019-02-09 02:12 test.txt
[root@master hadoop]# hadoop fs -test -e test.txt
[root@master hadoop]# echo $?
0
[root@master hadoop]# hadoop fs -test -e /user/root/test.txt
[root@master hadoop]# echo $?
0
[root@master hadoop]# hadoop fs -test -e /user/root/test.txtt
[root@master hadoop]# echo $?
1
2. hdfs 编程基础
2.1 判断文件是否存在
import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; public class hdfsfileifexist { public static void main(string[] args){ try{ //hdfs路径 string filename = "/user/root/test.txt"; configuration conf = new configuration(); conf.set("fs.defaultfs", "hdfs://master:8020"); conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.distributedfilesystem"); filesystem fs = filesystem.get(conf); if(fs.exists(new path(filename))){ system.out.println("文件存在"); }else{ system.out.println("文件不存在"); } }catch (exception e){ e.printstacktrace(); } } }
上面的代码中fs.defaultfs的值,请查看hdfs配置文件 core-site.xml ,该文件位于hadoop安装目录的etc/hadoop目录下。
如果在windows系统中运行该程序,请在c:\windows\system32\drivers\etc\hosts文件中加上主机名master和ip地址的对应关系。
上一篇: 好结实的塑料椅子
下一篇: SQL面试题:求时间差之和(有重复不计)