[ZEPPELIN-6717] Install locked Python dependencies in Linux CI

[miinhho]

What is this PR for?

Linux CI jobs repeatedly resolve the same Python 3.9 Conda dependencies during environment setup. This PR makes those jobs install from explicit Linux x86-64 Conda specifications instead, and pins the PyFlink packages installed with pip.

It also separates Maven, npm, Conda, and pip package caches by dependency type. The Livy job now caches the Spark distribution at the path used by its download script. Lock update and validation instructions are documented in testing/locks/README.md.

This targets dependency setup time.

What type of PR is it?

Improvement

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-6717

How should this be tested?

• Run the affected Linux CI jobs and confirm they install from the committed Conda specifications without dependency resolution.
• Confirm both PyFlink jobs install their pinned requirements and pass.
• Run CI again to check package-cache reuse and the Livy Spark distribution cache hit.
• Compare environment-setup durations separately from test durations.

Questions

• License files: No.
• Breaking changes for older versions: No product-runtime change. The explicit Conda specifications are Linux x86-64-specific.
• Documentation: Lock maintenance instructions are included.

[miinhho]

While working on this PR, I noticed that core.yml repeats several setup steps across jobs. I put together a follow-up commit that uses YAML anchors to reduce that duplication without changing the expanded job configuration.

I plan to propose it as a separate PR after this one is merged, so this PR can stay focused on dependency setup time.

[voidmatcha]

I confirmed that this PR substantially improves dependency setup time. As a small follow-up, I put together an example change that includes the Spark and Hadoop versions in the cache key and restore prefix, so a future version change cannot keep hitting the previous cache key. Please take a look.

[voidmatcha]

LGTM 👍