CDH hadoop-hdfs-fuse的安装使用
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |
参考文档： http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_hdfs_mountable.html
挂载HDFS
CDH5包含一个HDFS的接口FUSE（用户空间文件系统），FUSE能让你写一个正常的用户空间应用程序作为一个传统文件系统接口。hadoop-hdfs-fuse包使您能够使用HDFS集群就好像它是一个传统的文件系统在Linux上。假设你有一个工作HDFS集群和知道你的NameNode暴露的主机名和端口。(翻译) 简单总结：意思就是你可以把hdfs作为一个本次挂载的磁盘，进行交换。
下面是具体的安装步骤。
如果你的CDH集群不是rpm安装的，你得先去安装 hadoop-hdfs-fuse所需要依赖的包。具体如下：
找到你所需要的rpm包先进下载：(都是国外的网站，速度比较慢，可以放后台下载)
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/
# 构建本地yum源
mkdir -p /data/hadoop_install/cdh5
curl http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/ |awk -F'href= "' '{print $2}'|awk -F'">' '{print $1}' |grep rpm>noarch.list
curl http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/|awk -F’href'= "' '{print $2}'|awk -F'">' '{print $1}' |grep rpm>rpm.list
然后开启后台 screen -S 下载。
for i in `cat noarch.list` ; do wget http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/$i ;done
for i in `cat rpm.list` ; do wget http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/$i ; done
下载完成后记得构建依赖关系：
createrepo .
# 安装httpd服务并配置，并配置虚拟主机（如果有nginx的话，可以略过，nginx配置虚拟主机时记得开启autoindex参数）
#yum -y install httpd
#vim /etc/httpd/conf.d/cdh_yum.conf
<VirtualHost *:80>
ServerAdmin webmaster@gongchang.com
DocumentRoot "/data/hadoop_install/"
ServerName 192.168.8.222
ErrorLog "logs/cm-error.log"
CustomLog "logs/cm-access.log" common
<Directory "/data/hadoop_install/cm/5.0.2">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
<Directory "/data/hadoop_install/cdh/5.0.2">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
#/etc/init.d/httpd restart //重启httpd服务
然后可以通过浏览器查看是否能够访问http://192.168.8.222/cm/5.0.2
如果是Nginx的话，配置如下：
server {
listen 80;
server_name 192.168.8.222;
charset utf-8;
index index.php index.html index.htm;
root /data/hadoop_install;
location /cm/ {
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
}
location /cdh/ {
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
}
access_log off;
}
#/etc/init.d/nginx restart //重启nginx服务
#####在需要安装的服务上添加repo文件。例如8.94上。
[root@hadoop94 ~]# cat /etc/yum.repos.d/c-m.repo
[cloud-manager-5]
name=Cloud Manager,Version 5.7.1
baseurl=http://192.168.8.183:8800/cdh5
gpgcheck=0
################ 接下来就可以 yum安装了。
[root@hadoop94 ~]# yum install hadoop-hdfs-fuse
参考：
To set up and test your mount point in a non-HA installation:
$ mkdir -p <mount_point>
$ hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>
namenode_port is the NameNode's RPC port, dfs.namenode.servicerpc-address.
To set up and test your mount point in an HA installation:
$ mkdir -p <mount_point>
$ hadoop-fuse-dfs dfs://<nameservice_id> <mount_point>
nameservice_id is the value of fs.defaultFS. In this case the port defined for dfs.namenode.rpc-address.[nameservice ID].[name node ID] is used automatically. See Configuring Software for HDFS HA for more information about these properties.
fuse-dfs program, and
umount the partition if it is still mounted.
Note:
hadoop-fuse-dfs uses the HADOOP_CONF_DIR configured at the time the mount
To clean up your test:
$ umount <mount_point>
To add a system mount:
1. /etc/fstab and add lines to the bottom similar to these: hadoop-fuse-dfs#dfs://<name_node_hostname>:<namenode_port> <mount_point> fuse allow_other,usetrash,rw 2 0 For example: hadoop-fuse-dfs#dfs://localhost:8020 /mnt/hdfs fuse allow_other,usetrash,rw 2 0
2. $ mount <mount_point>
ls
/etc/default/hadoop-fuse
To change it:
export LIBHDFS_OPTS="-Xms64m -Xmx256m"
Be careful not to set the minimum to a higher value than the maximum.
hadoop-fuse-dfs:
$ hadoop-fuse-dfs --help
测试：
建立和测试您的挂载点non-HA安装:
[root@hadoop94 ~]# hadoop fs -ls /tmp //查看Hdfs文件系统/tmp目录下文件
[root@hadoop94 ~]# mkdir -p /tmp/fusetest // 建立一个本地文件
[root@hadoop94 tmp]# hadoop-fuse-dfs dfs://192.168.8.94:8022 /tmp/fusetest
INFO /data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.6.0-cdh5.5.1/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164 Adding FUSE arg /tmp/fusetest
[root@hadoop94 tmp]# df -h //查看本地系统的硬盘
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
165G 61G 97G 39% /
tmpfs 3.9G 8.0K 3.9G 1% /dev/shm
/dev/xvda1 477M 51M 401M 12% /boot
cm_processes 3.9G 3.9M 3.9G 1% /var/run/cloudera-scm-agent/process
fuse_dfs 209G 2.4G 206G 2% /tmp/fuse
fuse_dfs 209G 2.4G 206G 2% /tmp/fusetest
[root@hadoop94 tmp]# cd /tmp/fusetest/
[root@hadoop94 tmp]# touch bbb.txt //创建一个文件，编辑如下 aaaaaaa
[root@hadoop94 tmp]# vim bbb.txt
aaaaaaa
## 在hdfs下挂载后查看：
[root@hadoop94 ~]# hadoop fs -mkdir /tmp/fusetest
[root@hadoop94 ~]# hadoop fs -ls /tmp
[root@hadoop94 ~]# hadoop fs -cat /tmp/bbb.txt
aaaaaaa
[root@hadoop94 ~]#
针对hdfs ha的挂载没有测试。这个主要是nameserver_id 找正确就可以。
下面是卸载：
[root@hadoop94 ~]# umount /tmp/fusetest
[root@hadoop94 ~]#
#####可以通过一些参数，添加一个永久HDFS持续通过重新启动。添加一个系统安装:
参考上文。
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |