架构

HDFS和YARN是两个概念，搭建YARN和HDFS无冲突。

HOSTNNNNJNNDNZKFCZKRMNM
node01***
node02******
node03*****
node04****

发展进程

hdoop1.x2.x3.x
hdfsno haha(向前兼容，没有过多的改NN，而是通过新增角色zkfc)
yarnno yarnyarn（已经有了ha的概念，不是新增角色，而是直接在RM进程中增加了HA的模块）

通过官网

https://hadoop.apache.org/docs/r2.10.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

伪分布式

etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

HA

etc/hadoop/yarn-site.xml

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-address</name>
  <value>node01:2181,node02:2181,node03:2181</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>mashibing</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>node03</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>node04</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>node03:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>node04:8088</value>
</property>

配置

分发

scp mapred-site.xml yarn-site.xml node02:`pwd`
scp mapred-site.xml yarn-site.xml node03:`pwd`
scp mapred-site.xml yarn-site.xml node04:`pwd`

vi slaves //可以不用管，搭建hdfs时候已经改过了

启动

# 启动 NodeManager
start-yarn.sh
# 启动 ResourceManager
yarn-daemon.sh start resourcemanager

访问验证

http://node03:8088
http://node04:8088

使用

hdfs dfs -mkdir -p /data/wc/input

hdfs dfs -D dfs.blocksize=1048576 -put data.txt /data/wc/input

计算

hadoop jar hadoop-mapreduce-examples-2.10.1.jar wordcount /data/wc/input /data/wc/output

结果

代码

Hadoop

maven打包

  1. clean
  2. package

拷贝到服务器执行

结果

Scroll to Top