首页 > hadoop > hdp离线安装:1.基础环境准备

hdp离线安装:1.基础环境准备

2017年10月11日 发表评论 阅读评论

1.系统设置

修改主机名未fqdn格式

修改/etc/sysconfig/network
最后HOSTNAME=xx.xx.xx 这种域名形式

selinux关闭,更换阿里云源,sudo权限

修改阿里云源:

yum -y install wget
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo

安装基本软件 yum install -y telnet vim sysstat bind-utils mlocate lsof libaio
关闭ipv6

selinux

查看selinux是否关闭:
/usr/sbin/sestatus -v|grep ‘SELinux status’

临时关闭无需重启:
setenforce 0
检查:
getenforce

修改配置文件需要重启永久生效
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/selinux/config

设置sudo权限

修改/etc/sudoers文件,增加:
注意:各版本ambari需要的权限有差别,注意看对应版本的sudo权限

  1. # Ambari Customizable Users
  2. Cmnd_Alias AMBARI_USERS_COMMAND = /bin/su hdfs *, /bin/su ambari-qa *, /bin/su ranger *, \
  3. /bin/su zookeeper *, /bin/su knox *, /bin/su falcon *, \
  4. /bin/su ams *, /bin/su flume *, /bin/su hbase *, \
  5. /bin/su spark *, /bin/su accumulo *, /bin/su hive *, \
  6. /bin/su hcat *, /bin/su kafka *, /bin/su mapred *, \
  7. /bin/su oozie *, /bin/su sqoop *, /bin/su storm *, \
  8. /bin/su tez *, /bin/su atlas *, /bin/su yarn *, \
  9. /bin/su kms *, \
  10. /bin/ln -s /data/app/hdp /usr,\
  11. /sbin/service rpcbind *,\
  12. /sbin/service portmap *
  13. # Ambari Commands
  14. Cmnd_Alias AMBARI_COMMAND = /usr/bin/yum, /usr/bin/zypper, /usr/bin/apt-get, \
  15. /bin/mkdir, /usr/bin/test, /bin/ln, /bin/chown, \
  16. /bin/chmod, /bin/chgrp, /usr/sbin/groupadd, /usr/sbin/groupmod, \
  17. /usr/sbin/useradd, /usr/sbin/usermod, /bin/cp, \
  18. /usr/sbin/setenforce, /usr/bin/test, /usr/bin/stat, \
  19. /bin/mv, /bin/sed, /bin/rm, /bin/kill, /bin/readlink, \
  20. /usr/bin/pgrep, /bin/cat, /usr/bin/unzip, /bin/tar, \
  21. /usr/bin/tee, /bin/touch, /usr/bin/hdp-select, /usr/bin/conf-select, \
  22. /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh, \
  23. /usr/lib/hadoop/bin/hadoop-daemon.sh, /usr/lib/hadoop/sbin/hadoop-daemon.sh, \
  24. /sbin/chkconfig gmond off, /sbin/chkconfig gmetad off, \
  25. /etc/init.d/httpd *, /sbin/service hdp-gmetad start, \
  26. /sbin/service hdp-gmond start, /usr/sbin/gmond, \
  27. /usr/sbin/update-rc.d ganglia-monitor *, /usr/sbin/update-rc.d gmetad *, \
  28. /etc/init.d/apache2 *, /usr/sbin/service hdp-gmond *, \
  29. /usr/sbin/service hdp-gmetad *, /sbin/service mysqld *, \
  30. /usr/bin/python2.6 /var/lib/ambari-agent/data/tmp/validateKnoxStatus.py *, \
  31. /usr/hdp/current/knox-server/bin/knoxcli.sh *
  32. # Ambari Ranger Commands
  33. Cmnd_Alias AMBARI_RANGER_COMMAND = /usr/hdp/*/ranger-usersync/setup.sh, \
  34. /usr/bin/ranger-usersync-stop, /usr/bin/ranger-usersync-start, \
  35. /usr/hdp/*/ranger-admin/setup.sh *, /usr/hdp/*/ranger-knox-plugin/disable-knox-plugin.sh *, \
  36. /usr/hdp/*/ranger-storm-plugin/disable-storm-plugin.sh *, \
  37. /usr/hdp/*/ranger-hbase-plugin/disable-hbase-plugin.sh *, \
  38. /usr/hdp/*/ranger-hdfs-plugin/disable-hdfs-plugin.sh *, \
  39. /usr/hdp/current/ranger-admin/ranger_credential_helper.py, \
  40. /usr/hdp/current/ranger-kms/ranger_credential_helper.py
  41. Cmnd_Alias AMBARI_SUDO_COMMAND = /bin/su - ambari,/bin/su - hive
  42. ambari ALL=(ALL) NOPASSWD:SETENV: AMBARI_USERS_COMMAND, AMBARI_COMMAND, AMBARI_RANGER_COMMAND
  43. %dba ALL=(ALL) NOPASSWD:SETENV: AMBARI_SUDO_COMMAND

2.服务

web服务、ntp、域名服务,MySQL数据库

web服务(任意web服务均可,只在主节点上安装,用来做离线本地yum源):

yum install -y httpd
chkconfig httpd on

相关源文件下载:
使用迅雷下载文件:
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.3.0/HDP-2.5.3.0-centos6-rpm.tar.gz
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari-2.4.2.0-centos6.tar.gz

最终的http服务商的目录结构:
注意,baseurl一定是要到repodata 这个目录同一级

Ambari Base URL

 

 

 

 

 

 

 

 

 

 

HDP Base URL

 

 

 

 

 

 

 

 

 

 

 

HDP-UTILS Base URL

 

 

 

 

dns服务(只在主节点上安装)

这里为了简单实用dnsmasq做dns服务
yum install -y dnsmasq
chkconfig dnsmasq on

vim /etc/dnsmasq.conf
最后一行 conf-dir=/etc/dnsmasq.d
去掉注释

然后
vim /etc/dnsmasq.d/hadoop.conf 加入以下各节点的解析内容

  1. server=/hadoop/192.168.1.xx
  2. server=/test.hadoop/192.168.1.47
  3. address=/hadoop1.test.hadoop/192.168.1.47
  4. address=/hadoop2.test.hadoop/192.168.1.48
  5. address=/hadoop3.test.hadoop/192.168.1.49
  6. address=/hadoop4.test.hadoop/192.168.1.50
  7. address=/hadoop5.test.hadoop/192.168.1.51
  8. address=/master.test.hadoop/192.168.1.47
  9. address=/repo.test.hadoop/192.168.1.47
  10. address=/time.test.hadoop/192.168.1.47
  11. #反向解析记录
  12. ptr-record=47.1.168.192.in-addr.arpa,hadoop1.test.hadoop
  13. ptr-record=48.1.168.192.in-addr.arpa,hadoop2.test.hadoop
  14. ptr-record=49.1.168.192.in-addr.arpa,hadoop3.test.hadoop
  15. ptr-record=50.1.168.192.in-addr.arpa,hadoop4.test.hadoop
  16. ptr-record=51.1.168.192.in-addr.arpa,hadoop5.test.hadoop

这样需要解析的节点就在dns服务器上加好了

注意
因为需要访问外部源,所以要增加其他域名访问外部公共dns的设置
/etc/dnsmasq.conf 中增加:
server=192.168.111.1 那么没有增加解析记录的就会通过这个dns来解析了,dnsmasq做了dns缓存了

重启dns服务生效:
sudo /etc/init.d/dnsmasq restart

各节点dns解析设置

dns都修改成主节点的IP
cat  < < EOF > /etc/resolv.conf
nameserver xxxx
EOF
注:这样修改只是临时生效
永久生效修改
vim /etc/sysconfig/network-scripts/ifcfg-em1
最后dns1=192.168.1.xx
这样重启dns解析依然是生效的

ntp: (主要是hbase对各节点时间一致性要求高所有节点都安装)

主节点:
yum install -y ntp
ntpdate asia.pool.ntp.org # 注意开启时间服务器前一定要手动对时一次
hwclock -w
chkconfig –list ntpd
chkconfig ntpd on
chkconfig –list ntpd
service ntpd start

其他各节点上:
yum install -y ntp
ntpdate asia.pool.ntp.org # 注意开启时间服务器前一定要手动对时一次
hwclock -w
chkconfig –list ntpd
chkconfig ntpd on

然后各从节点修改ntp时间同步
vim /etc/ntp.conf

  1. server 0.centos.pool.ntp.org iburst
  2. server 1.centos.pool.ntp.org iburst
  3. server 2.centos.pool.ntp.org iburst
  4. server 3.centos.pool.ntp.org iburst

修改为

  1. server time.test.yunwei.puppet.dh iburst

可以直接在shell中执行:
sudo sed -i ‘s/^server /#server /g’ /etc/ntp.conf
sudo sed -i ’21aserver time.hadoop.yunwei.puppet.dh’ /etc/ntp.conf
修改后注意检查
架设的ntp时间服务器要能与上游的ntp连接上同步自己的时间并且在开启约20分钟之后才能被其他服务器用来当时间服务器同步

同时在shell中执行:
sudo bash -c “echo ‘logfile /var/log/ntpd.log’ >> /etc/ntp.conf”
重启ntp服务使之生效
service ntpd restart

3.安装MySQL数据库(只在主节点上安装,用来存储ambari的数据和hive的元数据等)

4. 系统层面性能优化:

关闭transparent_hugepage
使用root权限执行:
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
要开机生效需要将以上内容追加到 /etc/rc.local 里面

swap使用调整
cat /proc/sys/vm/swappiness
sysctl -w vm.swappiness=10
编辑 /etc/sysctl.conf 在最后一行增加:
vm.swappiness=10
echo vm.swappiness=10 >> /etc/sysctl.conf

参考:
http://bguncle.blog.51cto.com/3184079/957315
http://mirrors.aliyun.com/help/centos
https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/obtaining_the_repositories.html
https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/setting_up_a_local_repository_with_no_internet_access.html.
http://sndapk.blog.51cto.com/5385144/1336546
https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_admin_performance.html
http://tech.meituan.com/cpu_sys_high_in_hadoop_cluster.html
http://blog.sina.com.cn/s/blog_60aa9ccd0100h433.html
http://lsscto.blog.51cto.com/779396/762240
https://blog.rathena.cn/zai-centos-shang-shi-yong-dnsmasq-shi-xian-dns-zhuan-fa/
http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/commands_agent.html
http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/customizable_users_agent.html

 

移动设备快速阅读本文:
            请扫描二维码  -->
分类: hadoop 标签:
  1. 本文目前尚无任何评论.
  1. 本文目前尚无任何 trackbacks 和 pingbacks.