Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/deployment/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
* Since Kyuubi 1.13, the support of Flink engine for Flink 1.17, 1.18 and 1.19 is removed.
* Since Kyuubi 1.13, the support of Flink engine for Flink 2.0 is deprecated, and will be removed in the future.
* Since Kyuubi 1.13, `kyuubi.server.redaction.regex` defaults to `(?i)secret|password|token|access[.]key` instead of being unset, so `kyuubi.server.conf.retrieveMode=REDACTED` (the default) redacts matching session-config keys/values out of the box; it also affects the command-line arguments Kyuubi logs for spawned engine processes. Set it to a different pattern to override.
* Since Kyuubi 1.13, the Spark authorization plugin is migrated from the Ranger plugin API to the Ranger 2.9 authorization API, and Apache Ranger 2.9.0 or above is required for both authorizer modes. See [Installing and Configuring Kyuubi Spark AuthZ Plugin](../security/authorization/spark/install.md) for how to set up the new API.

## Upgrading from Kyuubi 1.11 to 1.12

Expand Down
18 changes: 5 additions & 13 deletions docs/security/authorization/spark/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,24 +74,16 @@ The available `spark.version`s are shown in the following table.

The maven option `ranger.version` is used for specifying Ranger version to compile with and generate corresponding transitive dependencies.
By default, it is always built with the latest `ranger.version` defined in kyuubi project main pom file.
Sometimes, it may be incompatible with other Ranger Admins, then you may need to build the plugin on your own targeting the Ranger Admin version you connect with.

```shell
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -am -DskipTests -Dranger.version=2.4.0
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -am -DskipTests -Dranger.version=2.9.0
```

The available `ranger.version`s are shown in the following table.
The plugin is built on the Ranger 2.9 authorization API (`ranger-authz-api` and
`authz-remote`), which was introduced in Ranger 2.9.0, so only Ranger 2.9.0 and above
are supported.

| Ranger Version | Supported | Remark |
|:--------------:|:---------:|:------:|
| 2.6.x | √ | - |
| 2.5.x | √ | - |
| 2.4.x | √ | - |
| 2.3.x | √ | - |
| 2.2.x | √ | - |
| 2.1.x | √ | - |

Currently, all ranger releases are supported.
Please use branch-1.12 or prior to build against Ranger versions prior to 2.9.0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to specify the Ranger versions supported for branch-1.12 and prior?


## Test with ScalaTest Maven plugin

Expand Down
82 changes: 75 additions & 7 deletions docs/security/authorization/spark/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@

- [Apache Ranger](https://ranger.apache.org/)

This plugin works as a ranger rest client with Apache Ranger Admin server to do privilege check.
Thus, a ranger server need to be installed ahead and available to use.
The plugin talks to a Ranger PDP server (default) or Ranger Admin server to do privilege check.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do you want to specify what PDP is?

A Ranger 2.9.0 server or above needs to be installed ahead and available to use.

- Building(optional)

If your Ranger Admin or Spark distribution is not compatible with the official pre-built [artifact](https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-authz) in maven central.
If your Ranger or Spark distribution is not compatible with the official pre-built [artifact](https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-authz) in maven central.
You need to [build](build.md) the plugin targeting the spark/ranger you are using by yourself.

## Install
Expand All @@ -37,8 +37,75 @@ Use either the shaded jar `kyuubi-spark-authz-shaded_*.jar` or the `kyuubi-spark

## Configure

### Authorizer Modes

The plugin supports two authorizer modes, selected by `ranger.authorizer.impl.class`:

- `org.apache.ranger.authz.remote.RangerRemoteAuthorizer` (default) — Ranger PDP mode.
Authorization requests are sent to the Ranger PDP server via REST APIs. The plugin is a thin
client: policies are not downloaded to the client side, and access audits are recorded by
the PDP server.
- `org.apache.ranger.authz.embedded.RangerEmbeddedAuthorizer` — embedded mode, working as the
previous Ranger plugin did: policies are pulled from the Ranger admin server and access
requests are evaluated locally on the Spark driver side.

:::{warning}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is not rendering here, expected?

The security-critical configurations — the authorizer implementation
(`ranger.authorizer.impl.class`), the Ranger PDP server address
(`ranger.authz.remote.pdp.url`), the Ranger service name
(`ranger.plugin.spark.service.name`) and the Ranger PDP client authentication and
encryption settings (`ranger.authz.remote.authn.*`, `ranger.authz.remote.ssl.*`,
`ranger.authz.remote.header.*`) — are read from `ranger-spark-security.xml` only:
JVM system properties can neither set nor override them, and the plugin fails to
initialize when the Ranger PDP server address is missing for the Ranger PDP mode.
Other configurations with the `ranger.` or `xasecure.` prefix can still be overridden
by JVM system properties, so make sure tenant users cannot control the JVM options of
the engine process (e.g. `spark.driver.extraJavaOptions`).
:::

#### Ranger PDP mode (default)

- Create `ranger-spark-security.xml` in `$SPARK_HOME/conf` and add the following configurations
for pointing to the right Ranger PDP server.

```xml
<configuration>
<property>
<name>ranger.plugin.spark.service.name</name>
<value>a ranger service name, e.g. a ranger hive service name</value>
</property>

<property>
<name>ranger.authz.remote.pdp.url</name>
<value>ranger pdp server address like https://ranger-pdp.org:8585</value>
</property>

</configuration>
```

The PDP client supports authentication and encryption settings, configured with
`ranger.authz.remote.authn.type` (`header`, `jwt` or `kerberos`),
`ranger.authz.remote.authn.*`, `ranger.authz.remote.ssl.*` and
`ranger.authz.remote.header.*` properties. Refer to the
[Ranger client libraries](https://cwiki.apache.org/confluence/display/RANGER/Ranger+Client+Libraries)
for the full list.

#### Embedded mode

Set the authorizer implementation to the embedded one in `ranger-spark-security.xml`:

```xml
<property>
<name>ranger.authorizer.impl.class</name>
<value>org.apache.ranger.authz.embedded.RangerEmbeddedAuthorizer</value>
</property>
```

### Settings for Connecting Ranger Admin

The settings below apply to the embedded mode, where the plugin pulls policies from

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

below meaning everything below this line?

the Ranger admin server and evaluates access requests locally.

#### ranger-spark-security.xml

- Create `ranger-spark-security.xml` in `$SPARK_HOME/conf` and add the following configurations
Expand Down Expand Up @@ -76,7 +143,7 @@ Use either the shaded jar `kyuubi-spark-authz-shaded_*.jar` or the `kyuubi-spark

##### Using Macros in Row Level Filters

Macros are now supported for using user/group/tag in row filter expressions, introduced in [Ranger 2.3](https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.3.0+-+Release+Notes). This feature helps significantly simplify row filter expressions by using user/group/tag's attributes instead of explicit conditions. Considering a user with an attribute `born_city` of value `Guangzhou `, the row filter condition as `city='${{USER.born_city}}'` will be transformed to `city='Guangzhou'` in execution plan. More supported macros and usage refer to [RANGER-3605](https://issues.apache.org/jira/browse/RANGER-3605) and [RANGER-3550](https://issues.apache.org/jira/browse/RANGER-3550). Add the following configs to `ranger-spark-security.xml` to enable UserStore Enricher required by macros.
Macros are supported for using user/group/tag in row filter expressions (embedded mode only), introduced in [Ranger 2.3](https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.3.0+-+Release+Notes). This feature helps significantly simplify row filter expressions by using user/group/tag's attributes instead of explicit conditions. Considering a user with an attribute `born_city` of value `Guangzhou `, the row filter condition as `city='${{USER.born_city}}'` will be transformed to `city='Guangzhou'` in execution plan. More supported macros and usage refer to [RANGER-3605](https://issues.apache.org/jira/browse/RANGER-3605) and [RANGER-3550](https://issues.apache.org/jira/browse/RANGER-3550). Add the following configs to `ranger-spark-security.xml` to enable UserStore Enricher required by macros.

```xml
<property>
Expand All @@ -94,7 +161,7 @@ Macros are now supported for using user/group/tag in row filter expressions, int

##### Showing all disallowed privileges

By default, Authz plugin checks required privileges one by one and throw the first unsatisfied privilege in exception. By setting `ranger.plugin.spark.authorize.in.single.call` to `true`, Authz plugin executes access checks in single call and throws all disallowed privileges in exception message.
By default, Authz plugin checks required privileges one by one and throw the first unsatisfied privilege in exception. By setting `ranger.plugin.spark.authorize.in.single.call` to `true`, Authz plugin executes access checks in single call and throws all disallowed privileges in exception message. This setting also reduces the number of authorization requests in Ranger PDP mode.

```xml
<property>
Expand All @@ -106,8 +173,9 @@ By default, Authz plugin checks required privileges one by one and throw the fir

#### ranger-spark-audit.xml

Create `ranger-spark-audit.xml` in `$SPARK_HOME/conf` and add the following configurations
to enable/disable auditing.
In the embedded mode, create `ranger-spark-audit.xml` in `$SPARK_HOME/conf` and add the following
configurations to enable/disable auditing. In Ranger PDP mode, access audits are recorded by
the PDP server, and this file is not required.

```xml
<configuration>
Expand Down
38 changes: 10 additions & 28 deletions extensions/spark/kyuubi-spark-authz-shaded/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,13 @@
<artifactSet>
<includes>
<include>org.apache.kyuubi:*</include>
<include>org.apache.ranger:*</include>
<!-- RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x -->
<include>org.codehaus.jackson:*</include>
<include>org.apache.ranger:ranger-authz-api</include>
<include>org.apache.ranger:authz-remote</include>
<include>com.fasterxml.jackson.core:*</include>
<include>com.fasterxml.jackson.module:*</include>
<include>com.fasterxml.jackson.jaxrs:*</include>
<include>com.sun.jersey:*</include>
<include>javax.ws.rs:jsr311-api</include>
<include>commons-collections:commons-collections</include>
<include>com.fasterxml.jackson.module:jackson-module-scala_*</include>
<include>org.apache.httpcomponents:httpclient</include>
<include>org.apache.httpcomponents:httpcore</include>
<include>org.apache.commons:commons-lang3</include>
</includes>
</artifactSet>
<filters>
Expand All @@ -82,33 +80,17 @@
</filter>
</filters>
<relocations>
<relocation>
<pattern>org.codehaus.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.codehaus.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.fasterxml.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.jersey</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.jersey</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.ws.rs.ext</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.ws.rs.ext</shadedPattern>
</relocation>
<relocation>
<pattern>javax.ws.rs</pattern>
<shadedPattern>${kyuubi.shade.packageName}.javax.ws.rs</shadedPattern>
</relocation>
<relocation>
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
<pattern>org.apache.http</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.apache.http</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons.collections</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.apache.commons.collections</shadedPattern>
<pattern>org.apache.commons.lang3</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.apache.commons.lang3</shadedPattern>
</relocation>
</relocations>
<transformers>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -207,19 +207,12 @@ This project bundles some components that are licensed under the

Apache License Version 2.0
--------------------------
org.apache.ranger:ranger-plugins-common
org.apache.ranger:ranger-plugins-audit
org.codehaus.jackson:jackson-jaxrs
org.codehaus.jackson:jackson-core-asl
org.codehaus.jackson:jackson-mapper-asl
net.java.dev.jna:jna
net.java.dev.jna:jna-platform

Common Development and Distribution License (CDDL) 1.1
------------------------------------------------------
com.sun.jersey:jersey-client
com.sun.jersey:jersey-core

MIT license
-----------
com.kstruct:gethostname4j
com.fasterxml.jackson.core:jackson-annotations
com.fasterxml.jackson.core:jackson-core
com.fasterxml.jackson.core:jackson-databind
com.fasterxml.jackson.module:jackson-module-scala_*
org.apache.commons:commons-lang3
org.apache.httpcomponents:httpclient
org.apache.httpcomponents:httpcore
org.apache.ranger:authz-remote
org.apache.ranger:ranger-authz-api
17 changes: 8 additions & 9 deletions extensions/spark/kyuubi-spark-authz/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,15 @@
- [x] Row-level fine-grained authorization, a.k.a. Row-level filtering
- [x] Data masking

The plugin supports two authorizer modes: Ranger PDP mode (default) which sends
authorization requests to a Ranger PDP server via REST APIs with a thin client, and
the embedded mode which pulls policies from the Ranger admin server and evaluates
requests locally. See `docs/security/authorization/spark/install.md` for details.

## Build

```shell
build/mvn clean package -DskipTests -pl :kyuubi-spark-authz_2.12 -am -Dspark.version=3.5.6 -Dranger.version=2.6.0
build/mvn clean package -DskipTests -pl :kyuubi-spark-authz_2.12 -am -Dspark.version=3.5.6 -Dranger.version=2.9.0
```

### Supported Apache Spark Versions
Expand All @@ -44,11 +49,5 @@ build/mvn clean package -DskipTests -pl :kyuubi-spark-authz_2.12 -am -Dspark.ver

`-Dranger.version=`

- [ ] 2.7.x
- [x] 2.6.x (default)
- [x] 2.5.x
- [x] 2.4.x
- [x] 2.3.x
- [x] 2.2.x
- [x] 2.1.x
- [ ] 2.0.x
The plugin is built on the Ranger 2.9 authorization API, so Ranger 2.9.0 and above are
the only supported versions.
Loading
Loading