Ambari
发表于：2021-09-28 | 分类： 平台

Ambari

1.什么是Ambari

Apache Ambari项目旨在通过开发用于配置，管理和监视Apache Hadoop集群的软件来简化Hadoop管理。Ambari通过其RESTful API提供了直观，易于使用的Hadoop管理Web UI。

Ambari使系统管理员可以：

  • 设置Hadoop集群

    • Ambari提供了用于在任意数量的主机上安装Hadoop服务的分步向导。
    • Ambari处理群集的Hadoop服务的配置。

  • 管理Hadoop集群

    • Ambari提供了用于在整个集群中启动，停止和重新配置Hadoop服务的集中管理。

  • 监控Hadoop集群

    • Ambari提供了一个仪表板，用于监视Hadoop集群的运行状况和状态。
    • Ambari利用Ambari Metrics System收集指标。
    • Ambari利用Ambari Alert Framework进行系统警报，并在需要您关注时（例如，节点故障，剩余磁盘空间不足等）通知您。

Ambari使应用程序开发人员和系统集成商能够：

官方网站：http://ambari.apache.org/

2.Ambari架构

ambari-server

  • 提供外部访问的API
  • 接受Ambari-agent的心跳信息和管理Ambari-agent
  • 元数据的管理和数据库的访问

ambari-agent

  • 采集所在节点的信息并且汇总发心跳汇报给ambari-server
  • 处理ambari-server的执行请求,安装、启动、停止服务等

ambari-web

  • 提供可视化的操作界面

3.下载

Ambari所在的公司已经被Cloudera公司收购所以相关的文档都在clodera公司的官方网站：https://docs.cloudera.com/HDPDocuments/index.html

我们选择的安装方式是:Ambari+HDP+HDP-UTILS,因为Ambari本身只是一个大数据平台自动化部署和管理的工具，所以需要配合HDP(大数据软件安装包集合)一起使用。

image-20211009131251511

选择合适的版本，我们这里选择的是2.7.0.0版本，然后就会有一个安装指南：https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-installation/content/ambari_repositories.html

其实可以直接配置官网提供的yum源就可以安装，但是由于是国外的网站，速度比较慢，所以就使用离线本地yum源的安装方式

4.安装环境说明

使用VMWare/VirtualBox，虚拟四台虚拟机，(也可以将server配置在一台节点中)其中ambari是服务器，其他为HDP安装节点，配置如下:

主机 系统 网络 内存 磁盘
ambari CentOS7 172.18.74.160(NAT/birdge) 4G 50G
hdp-1 CentOS7 172.18.74.161(NAT/birdge) 4G 50G
hdp-2 CentOS7 172.18.74.162(NAT/birdge) 4G 50G
hdp-3 CentOS7 172.18.74.163(NAT/birdge) 4G 50G

配置好虚拟机就可以安装了,安装的方式如下

在这里插入图片描述

5.Server端环境配置(ambari)

说明：如果服务器已经配置好，则直接进行第6节客户端的配置。

5.1.安装http服务

1
2
3
4
yum install -y httpd
systemctl start httpd
systemctl enable httpd

5.2.配置yum源

虚拟机的网络模式是nat或者桥接的时候就可以连通外网，这样我们就可以不用配置CentOS的yum源了，但是Ambari和HDP的还需要我们配置，需要将Ambari和HDP以及HDP-UTILS的安装包通过SFTP工具上传到ambari上。

1
2
3
4
5
6
7
#创建文件夹
mkdir /var/www/html/{ambari,hdp,hdp-utils}
# 解压安装包到HTTP的网页根目录
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp-utils/
tar -zxvf ambari-2.7.0.0-centos7.tar.gz -C /var/www/html/
tar -zxvf HDP-3.0.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp

这样就可以通过http的方式获取到相应的rpm文件和repodata文件来配置yum源了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 配置ambari的yum源
vi /etc/yum.repos.d/ambari.repo
[ambari]
name=ambari
baseurl=http://172.18.74.160/centos7/2.7.0.0-897/
gpgcheck=0

#配置hdp和hdp-utils的yum源
vi /etc/yum.repos.d/hdp.repo
[HDP]
name=HDP
baseurl=http://172.18.74.160/hdp/HDP/centos7/3.0.0.0-1634/
gpgcheck=0
[HDP-UTILS]
name=HDP_UTILS
baseurl=http://172.18.74.160/hdp-utils/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
# 验证
yum repolist

将yum文件使用scp发送到客户端节点

1
2
scp /etc/yum.repos.d/ambari.repo hdp-1:/etc/yum.repos.d/
scp /etc/yum.repos.d/hdp.repo hdp-1:/etc/yum.repos.d/

5.3.数据库配置

数据库使用MariaDB作为元数据储存的库，存放ambari服务数据

1
2
3
4
5
6
# 安装
yum install -y mariadb-server
# 启动 开机自启
systemctl start mariadb && systemctl enable mariadb
# 初始化
mysql_secure_installation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
[root@ambari ~]# mysql_secure_installation 

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] 
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] 
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] 
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] 
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

5.3.1.创建数据库

1
2
3
4
5
6
7
8
9
mysql -uroot -p
# 创建amabri数据库
create database ambari;
#授权
grant all on ambari.* to ambari@'%' identified by 'bigdata';
grant all on ambari.* to ambari@localhost identified by 'bigdata';
# 使用ambari-server提供的sql脚本创建相关的表
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

5.3.2.将JDBC驱动包复制到指定目录（/usr/share/java）

将jdbc的驱动包使用sftp工具上传到ambari

1
2
mv mysql-connector-java-*.jar /usr/share/java/mysql-connector-java.jar
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

5.4.安装ambari-server

1
2
3
4
5
# 安装
yum install -y ambari-server

# 配置ambari-server
ambari-server setup
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
[root@hdp-1 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/jdk1.8.0_171/
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? 
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? 
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? 
Extracting system views...
ambari-admin-2.7.0.0.897.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

5.5启动ambari-server

1
ambari-server start

可以访问ambari的web界面: http://172.18.74.160:8080

6.Client端环境配置

说明：以下配置是在需要安装集群的各个节点中进行，此处为hdp-1,hdp-2,hdp-3。

6.1修改主机名和配置主机到IP的映射(所有节点)

1
2
3
4
5
6
7
8
9
10
11
12
13
# 分别修改三台机器的主机名
hostnamectl set-hostname hdp-1
hostnamectl set-hostname hdp-2
hostnamectl set-hostname hdp-3
#立即生效
bash

# 配置主机名到IP的映射
vi /etc/hosts
172.18.74.160 ambari
172.18.74.161 hdp-1
172.18.74.162 hdp-2
172.18.74.163 hdp-3

6.2关闭防火墙和关闭SELinux(所有节点)

1
2
3
4
5
6
# 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
# 关闭selinux
setenforce 0
vi /etc/selinux/config
SELINUX=disabled

6.3.配置免密登录(所有节点)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# hpd-1
ssh-keygen
ssh-copy-id hdp-1
ssh-copy-id hdp-2
ssh-copy-id hdp-3
# hdp-2
ssh-keygen
ssh-copy-id hdp-1
ssh-copy-id hdp-2
ssh-copy-id hdp-3
# hdp-3
ssh-keygen
ssh-copy-id hdp-1
ssh-copy-id hdp-2
ssh-copy-id hdp-3

6.4.安装时间同步服务ntp

1
2
3
4
5
6
7
8
9
10
11
12
13
#hdp-1
yum install -y ntp
vi /etc/ntp.conf
server 127.127.1.0
fudge 127.127.1.0 stratum 10
systemctl start ntpd
systemctl enable ntpd
#hdp-2
yum install -y ntpdate
ntpdate hdp-1
#hdp-3
yum install -y ntpdate
ntpdate hdp-1

6.5关闭大页面压缩(所有节点)

1
2
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

6.6.安装配置JDK

将jdk的安装包上传到hdp-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#解压
tar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/local/

# 配置环境变量
vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=${JAVA_HOME}/jre
export PATH=${JAVA_HOME}/bin:$PATH
# 刷新环境变量
source /ect/profile
java
# 将jdk和环境变量发送到其它机器
scp -r /usr/local/jdk  hdp-2:/usr/local/
scp -r /usr/local/jdk  hdp-3:/usr/local/
scp  /etc/profile hdp-2:/etc/
scp  /etc/profile hdp-3:/etc/
# hdp-2 hdp-3
source /ect/profile
java

6.7.安装必要软件(所有节点)

1
2
3
4
5
6
7
8
9
yum -y install libtirpc-devel
yum -y install psmisc
yum -y install redhat-lsb
yum -y install nc
yum -y install gcc
yum -y install python-devel
yum -y install python-kerberos-1.1-15.el7.x86_64
yum -y install rpcbind-0.2.0-49.el7.x86_64

6.8.安装MySQL数据库(hdp-2，安装Hive可选)

1
2
3
4
5
6
7
8
# 安装
yum install -y mariadb-server
# 启动 开机自启
systemctl start mariadb && systemctl enable mariadb
# 初始化
mysql_secure_installation
# 进入数据库
mysql -u root -p
1
2
3
4
create database hive;
create user "hive"@"%" identified by "hive";
grant all privileges on hive.* to 'hive'@'%' identified by 'hive';
flush privileges;

6.9.安装和配置ambari-agent(所有节点)

1
2
3
4
5
6
7
8
# 安装
yum install -y ambari-agent
# 配置
vi /etc/ambari-agent/conf/ambari-agent.ini 
[server]
hostname=ambari
#启动
ambari-agent start

7.HDP部署

7.1登录

在浏览器中访问 http://172.18.74.160:8080

默认的用户名和密码都是admin

image-20211009131454796

7.2配置集群

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

image-20211009132042277

上一篇:
Spark+RDD
下一篇:
HBNIS WriteUp
本文目录
本文目录
    •

    Copyright © 2019 - 2023 DIFSSR | Powered by Hexo | Theme Bamboo