build: upgrade pyspark to 3.5 and delta spark to 3.2

[georgeRobertson2]

TLDR of changes

Upgrade PySpark to 3.5.2 and delta spark to 3.2.0

Delta spark maven package switches fro delta-core to delta-spark.

• https://mvnrepository.com/artifact/io.delta/delta-spark_2.13/3.2.0

Debating whether to upgrade spark xml to 0.18.0 as well.

• https://github.com/databricks/spark-xml
• https://mvnrepository.com/artifact/com.databricks/spark-xml

What kind of changes does this PR introduce?

Tick all that apply

• fix: A bug fix. Correlates with PATCH in SemVer
• feat: A new feature. Correlates with MINOR in SemVer
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• refactor: A code change that neither fixes a bug nor adds a feature
• perf: A code change that improves performance
• test: Adding missing or correcting existing tests
• build: Changes that affect the build system or external dependencies (example scopes: pip, docker, npm)
• ci: Changes to CI configuration files and scripts (example scopes: GitLabCI)

Please check if the PR fulfills these requirements

• I have read and followed the Contributing guidance
• Docs have been added / updated
• Tests and Linting in the CI are passing
• Changes have been reviewed and approved by a Project Maintainer

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud