欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

HDFS基础

程序员文章站 2023-10-31 13:19:10
1. HDFS Shell基础 [root@master hadoop]# hadoop fsUsage: hadoop fs [generic options] [-appendToFile ... ] [-cat [-ignoreCrc] ...] [ ......

1. hdfs shell基础

[root@master hadoop]# hadoop fs
usage: hadoop fs [generic options]
[-appendtofile <localsrc> ... <dst>]
[-cat [-ignorecrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-r] group path...]
[-chmod [-r] <mode[,mode]... | octalmode> path...]
[-chown [-r] [owner][:[group]] path...]
[-copyfromlocal [-f] [-p] [-l] <localsrc> ... <dst>]
[-copytolocal [-p] [-ignorecrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createsnapshot <snapshotdir> [<snapshotname>]]
[-deletesnapshot <snapshotdir> <snapshotname>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignorecrc] [-crc] <src> ... <localdst>]
[-getfacl [-r] <path>]
[-getfattr [-r] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-r] [<path> ...]]
[-mkdir [-p] <path> ...]
[-movefromlocal <localsrc> ... <dst>]
[-movetolocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renamesnapshot <snapshotdir> <oldname> <newname>]
[-rm [-f] [-r|-r] [-skiptrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-r] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-r] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignorecrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]

查看某个命令的具体帮助信息:

[root@master hadoop]# hadoop fs -help test
-test -[defsz] <path> :
answer various questions about <path>, with result via exit status.
-d return 0 if <path> is a directory.
-e return 0 if <path> exists.
-f return 0 if <path> is a file.
-s return 0 if file <path> is greater than zero bytes in size.
-z return 0 if file <path> is zero bytes in size, else return 1.

这里尤其要注意,如果文件存在,返回结果是0

我们上传一个文件,然后用test命令测试:

[root@master hadoop]# hadoop fs -put /root/test test.txt

[root@master hadoop]# hadoop fs -ls .
found 4 items
drwxrwxrwx - hdfs hdfs 0 2018-02-10 22:22 quasimontecarlo_1518319340789_698036166
drwxrwxrwx - hdfs hdfs 0 2018-02-10 23:21 quasimontecarlo_1518322909671_1083050937
-rw-r--r-- 3 root hdfs 5 2019-02-08 21:52 test
-rw-r--r-- 3 root hdfs 5 2019-02-09 02:12 test.txt

[root@master hadoop]# hadoop fs -test -e test.txt
[root@master hadoop]# echo $?
0
[root@master hadoop]# hadoop fs -test -e /user/root/test.txt
[root@master hadoop]# echo $?
0

[root@master hadoop]# hadoop fs -test -e /user/root/test.txtt
[root@master hadoop]# echo $?
1

2. hdfs 编程基础

2.1 判断文件是否存在

import org.apache.hadoop.conf.configuration;
import org.apache.hadoop.fs.filesystem;
import org.apache.hadoop.fs.path;

public class hdfsfileifexist {
    public static void main(string[] args){
        try{
            //hdfs路径
            string filename = "/user/root/test.txt";
            configuration conf = new configuration();
            conf.set("fs.defaultfs", "hdfs://master:8020");
            conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.distributedfilesystem");
            filesystem fs = filesystem.get(conf);
            if(fs.exists(new path(filename))){
                system.out.println("文件存在");
            }else{
                system.out.println("文件不存在");
            }
            
        }catch (exception e){
            e.printstacktrace();
        }
    }

}

上面的代码中fs.defaultfs的值,请查看hdfs配置文件 core-site.xml ,该文件位于hadoop安装目录的etc/hadoop目录下。

如果在windows系统中运行该程序,请在c:\windows\system32\drivers\etc\hosts文件中加上主机名master和ip地址的对应关系。