Skip to content

[AUTHZ] Upgrade Apache Ranger to 2.9.0 and migrate Spark authz plugin to Ranger PDP mode - #7437

Open
aajisaka wants to merge 2 commits into
apache:masterfrom
aajisaka:update-ranger
Open

[AUTHZ] Upgrade Apache Ranger to 2.9.0 and migrate Spark authz plugin to Ranger PDP mode#7437
aajisaka wants to merge 2 commits into
apache:masterfrom
aajisaka:update-ranger

Conversation

@aajisaka

@aajisaka aajisaka commented May 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Why are the changes needed?

Upgrade Apache Ranger from 2.6.0 to 2.9.0 in the Spark AuthZ plugin to fix
GHSA-c87w-642h-m97h, and migrate the plugin to the new Ranger authorization API
introduced in Ranger 2.9.0 (ranger-authz-api / authz-remote / authz-embedded):

  • SparkRangerAdminPlugin delegates to org.apache.ranger.authz.api.RangerAuthorizer.
    The default is the Ranger PDP mode (RangerRemoteAuthorizer, a thin client sending
    REST requests to the Ranger PDP server); the embedded mode (pull policies from the
    Ranger admin and evaluate locally, as the previous plugin did) is kept as an option
    via ranger.authorizer.impl.class. The plugin now requires Apache Ranger 2.9.0 or
    above; the migration guide and the docs are updated.
  • Security hardening found while reviewing the migration:
    • escape RRN metacharacters (backslash, slash) in resource names, and deny blank
      resource names instead of matching wildcard policies with them;
    • authorize a uri as alternative resource variants (exact path and trailing-slash
      path): the RRN request evaluates sub-resources as conjunctive requirements, which
      denied access to users granted exact-path url policies;
    • deny accesses when the Ranger authorizer returns no result, instead of skipping
      the checks or leaving the row filter / data mask unapplied;
    • read the security-critical configurations (the authorizer implementation, the
      Ranger PDP server address, the Ranger service name, and the Ranger PDP client
      authentication and encryption settings) from ranger-spark-security.xml only:
      JVM system properties can neither set nor override them.

How was this patch tested?

  • New and updated unit tests:
    • AccessResourceSuite: RRN metacharacter escaping (round-trip through the real
      RangerResourceNameParser), blank resource name denial, uri resource variants.
    • SparkRangerAdminPluginSuite: accesses denied when the authorizer returns no
      result (per-request and single-call paths), security-critical configurations
      cannot be overridden by system properties, fail fast on a missing Ranger PDP url.
    • RangerRemoteAuthorizerSuite (+ new MockPdpServer): the thin client against a
      mock PDP server.
  • Unit suites (all pass):
    build/mvn -nsu test -pl extensions/spark/kyuubi-spark-authz -am
    -Dtest=none
    -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.ranger.RangerRemoteAuthorizerSuite,org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPluginSuite,org.apache.kyuubi.plugin.spark.authz.ranger.AccessResourceSuite
  • Data masking and row filtering end-to-end suites (26 tests) pass.
  • Built Apache Ranger 2.9.0 server via docker-compose locally and validated against it with:
    build/mvn -nsu test -pl extensions/spark/kyuubi-spark-authz -am
    -DforkMode=never
    -DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.ranger.RangerRemoteAuthorizerSuite
    -Dranger.authz.remote.pdp.url=http://localhost:6500
    -Dranger.authz.remote.authn.type=header
    -Dranger.authz.remote.authn.header.X-Forwarded-User=ranger
    Also verified the uri variant fix on the real PDP with both an exact-path url
    policy and a trailing-slash url policy.

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: OpenCode (GLM-5.3-Flash)

@aajisaka aajisaka mentioned this pull request May 8, 2026
Open
@pan3793
pan3793 requested a review from wForget May 12, 2026 12:01
@wForget

wForget commented May 13, 2026

Copy link
Copy Markdown
Member

@aajisaka Thank you for this work. Did you verify kyuubi-spark-authz-shaded module? Do the maven-shade-plugin configurations need to be changed?

kyuubi/extensions/spark/kyuubi-spark-authz-shaded/pom.xml

Lines 46 to 112 in be35121

<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.kyuubi:*</include>
<include>org.apache.ranger:*</include>
<!-- RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x -->
<include>org.codehaus.jackson:*</include>
<include>com.fasterxml.jackson.core:*</include>
<include>com.fasterxml.jackson.module:*</include>
<include>com.fasterxml.jackson.jaxrs:*</include>
<include>com.sun.jersey:*</include>
<include>javax.ws.rs:jsr311-api</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/*.proto</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/LGPL2.1</exclude>
<exclude>META-INF/AL2.0</exclude>
<exclude>META-INF/ASL2.0</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/DEPENDENCIES</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>LICENSE.txt</exclude>
<exclude>NOTICE.txt</exclude>
<exclude>mozilla/**</exclude>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>org.codehaus.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.codehaus.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.fasterxml.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.jersey</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.jersey</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.ws.rs.ext</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.ws.rs.ext</shadedPattern>
</relocation>
<relocation>
<pattern>javax.ws.rs</pattern>
<shadedPattern>${kyuubi.shade.packageName}.javax.ws.rs</shadedPattern>
</relocation>
<relocation>
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
</transformers>
</configuration>

@wangzhigang1999 wangzhigang1999 mentioned this pull request May 13, 2026
Closed
4 tasks
@aajisaka

aajisaka commented May 15, 2026

Copy link
Copy Markdown
Member Author

@aajisaka Thank you for this work. Did you verify kyuubi-spark-authz-shaded module? Do the maven-shade-plugin configurations need to be changed?

kyuubi/extensions/spark/kyuubi-spark-authz-shaded/pom.xml

Lines 46 to 112 in be35121

<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.apache.kyuubi:*</include>
<include>org.apache.ranger:*</include>
<!-- RANGER-4225 (2.5.0) upgrades Jackson from 1.x to 2.x -->
<include>org.codehaus.jackson:*</include>
<include>com.fasterxml.jackson.core:*</include>
<include>com.fasterxml.jackson.module:*</include>
<include>com.fasterxml.jackson.jaxrs:*</include>
<include>com.sun.jersey:*</include>
<include>javax.ws.rs:jsr311-api</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/*.proto</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/LGPL2.1</exclude>
<exclude>META-INF/AL2.0</exclude>
<exclude>META-INF/ASL2.0</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/DEPENDENCIES</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
<exclude>META-INF/NOTICE.txt</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>LICENSE.txt</exclude>
<exclude>NOTICE.txt</exclude>
<exclude>mozilla/**</exclude>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>org.codehaus.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.org.codehaus.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.fasterxml.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.jersey</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.jersey</shadedPattern>
</relocation>
<relocation>
<pattern>com.sun.ws.rs.ext</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.sun.ws.rs.ext</shadedPattern>
</relocation>
<relocation>
<pattern>javax.ws.rs</pattern>
<shadedPattern>${kyuubi.shade.packageName}.javax.ws.rs</shadedPattern>
</relocation>
<relocation>
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
</transformers>
</configuration>

Sorry I forgot to verify. I removed Jackson 1 config because Ranger 2.8.0+ doesn't have its dependency. Also, updated the LICENSE file of the shaded jar.

@aajisaka aajisaka self-assigned this Jun 9, 2026
@aajisaka aajisaka added this to the v1.13.0 milestone Sep 6, 2026
@pan3793

pan3793 commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

If we have to make a breaking change here eventually, I wonder if we should do more things to move to Ranger 2.9 and PDP mode.

The current architecture- pulling all policies from the Ranger Admin Server on Spark application start, and evaluating authz requests on the client side (Spark driver), does not fit short-lived Spark ETL cases, and another disadvantage is that we have to bundle a heavy Ranger client that pulls in many dependencies. (easy to conflict, varies across versions, vulnerable)

In PDP mode, the Ranger client becomes lightweight and is almost an HTTP client, also no need to download the whole policies to the client. There are many organizations have similar enhancements to make the Ranger Spark Plugin fit large-scale lakehouses; now this is supported by the upstream Ranger community, we should migrate to it ASAP.

UT infrastructure might require a significant adjustment to allow us to adopt the Ranger PDP ...

@aajisaka

aajisaka commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Thank you @pan3793

While it's a significant change, it really makes sense to me. I'll try to move to Ranger 2.9 and support PDP mode.

@aajisaka
Upgrade Apache Ranger from 2.6.0 to 2.9.0 and migrate Spark authz plu…
e20a3b2 
…gin to Ranger PDP mode

Upgrade Apache Ranger from 2.6.0 to 2.9.0 to fix GHSA-c87w-642h-m97h,
and migrate the plugin to the new Ranger authorization API with Ranger
PDP mode as the default (suggested in review):

- SparkRangerAdminPlugin delegates to org.apache.ranger.authz.api
  RangerAuthorizer; the default implementation is RangerRemoteAuthorizer
  (the thin authz-remote client sending REST requests to the Ranger PDP
  server), and the embedded policy-download mode is kept as an option
  via ranger.authorizer.impl.class
- AccessRequest/AccessResource are rewritten for the new resource model
  (RRN-style resources, e.g. table:default/src, column:default/src/id)
- the shaded plugin bundles only ranger-authz-api, authz-remote, jackson,
  httpclient and commons-lang3; audits are recorded by the PDP server in
  PDP mode, and by the embedded authorizer in embedded mode
- the UT runs on the embedded authorizer with the local policy fixture,
  plus RangerRemoteAuthorizerSuite exercising the thin client against a
  mock PDP server (CI default), which can also run against a real Ranger
  PDP server via -Dranger.authz.remote.pdp.url

The security review findings are also addressed:

- resource names containing RRN metacharacters (backslash and slash) are
  escaped when building the RRN request, and blank resource names are
  denied instead of being evaluated against the wildcard marker
- a uri is authorized as alternative resource variants (the exact path
  and the path with a trailing slash), because the RRN request evaluates
  the sub-resources as conjunctive requirements instead of alternatives
- accesses are denied when the Ranger authorizer returns no result,
  instead of skipping the checks or leaving the row filter and the data
  mask unapplied
- the security-critical configurations (the authorizer implementation,
  the Ranger PDP server address, the Ranger service name, and the Ranger
  PDP client authentication and encryption settings) are read from
  ranger-spark-security.xml only, cannot be set or overridden by JVM
  system properties, and the plugin fails to initialize when the Ranger
  PDP server address is missing for the Ranger PDP mode
@aajisaka aajisaka changed the title Upgrade Apache Ranger from 2.6.0 to 2.8.0 [AUTHZ] Upgrade Apache Ranger to 2.9.0 and migrate Spark authz plugin to Ranger PDP mode Sep 12, 2026
@aajisaka

aajisaka commented Sep 12, 2026

Copy link
Copy Markdown
Member Author

Updated to support Ranger 2.9.0 and also updated the PR description.

@aajisaka
Fix Scalastyle violations in the Spark authz plugin
ebb9a79 
- keep the changed lines within 100 characters
- extend RangerRemoteAuthorizerSuite with KyuubiFunSuite instead of
  AnyFunSuite, and shut down the mock PDP server's executor pool on
  close so the suite's thread audit sees no leaked threads
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wForget wForget Awaiting requested review from wForget

Assignees

@aajisaka aajisaka

Labels

kind:build kind:documentation Documentation is a feature! module:authz module:extensions module:spark

Projects

None yet

Milestone

v1.13.0

Development

Successfully merging this pull request may close these issues.

3 participants

@aajisaka @wForget @pan3793