housekeeping (tests, docs) (geopython#2091)

* housekeeping (tests, docs)

* fix ref

Author: tomkralidis

.github/workflows/main.yml

@@ -124,7 +124,6 @@ jobs:
         pip3 install -r requirements-django.txt
         python3 setup.py install
         pip3 install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
-        #pip3 install --upgrade rasterio==1.1.8
     - name: setup test data ⚙️
       run: |
         python3 tests/load_es_data.py tests/data/ne_110m_populated_places_simple.geojson geonameid
@@ -137,49 +136,20 @@ jobs:
         mysql -h 127.0.0.1 -P 3306 -u root -p'mysql' test_geo_app < tests/data/mysql_data.sql
         docker ps
         python3 tests/load_oracle_data.py
-    - name: run unit tests ⚙️
+    - name: run API tests ⚙️
+      run: pytest tests/api --ignore-glob='*_live.py'
+    - name: run Formatter tests ⚙️
+      run: pytest tests/formatter
+    - name: run Provider tests ⚙️
       env:
         POSTGRESQL_PASSWORD: ${{ secrets.DatabasePassword || 'postgres' }}
-      run: |
-        pytest tests/api --ignore=tests/api/test_admin_live.py
-        pytest tests/test_api_ogr_provider.py
-        pytest tests/test_base_provider.py
-        pytest tests/test_config.py
-        pytest tests/test_csv__formatter.py
-        pytest tests/test_csv__provider.py
-        pytest tests/test_django.py
-        pytest tests/test_elasticsearch__provider.py
-        pytest tests/test_opensearch__provider.py
-        pytest tests/test_esri_provider.py
-        pytest tests/test_filesystem_provider.py
-        pytest tests/test_geojson_provider.py
-        pytest tests/test_linked_data.py
-        pytest tests/test_mongo_provider.py
-        pytest tests/test_ogr_csv_provider.py
-        pytest tests/test_ogr_esrijson_provider.py
-        pytest tests/test_ogr_gpkg_provider.py
-        pytest tests/test_ogr_shapefile_provider.py
-        pytest tests/test_ogr_sqlite_provider.py
-        pytest tests/test_ogr_wfs_provider.py
-        pytest tests/test_postgresql_manager.py
-        # pytest tests/test_ogr_wfs_provider_live.py  # NOTE: these are skipped in the file but listed here for completeness
-        pytest tests/test_openapi.py
-        pytest tests/test_oracle_provider.py
-        pytest tests/test_parquet_provider.py
-        pytest tests/test_postgresql_provider.py
-        pytest tests/test_postgresql_mvt_provider.py
-        pytest tests/test_mysql_provider.py
-        pytest tests/test_rasterio_provider.py
-        pytest tests/test_sensorthings_edr_provider.py 
-        pytest tests/test_sensorthings_provider.py
-        pytest tests/test_socrata_provider.py
-        # pytest tests/test_socrata_provider_live.py.py  # NOTE: these are skipped in the file but listed here for completeness
-        pytest tests/test_sqlite_geopackage_provider.py
-        pytest tests/test_tinydb_catalogue_provider.py
-        pytest tests/test_tinydb_manager_for_parallel_requests.py
-        pytest tests/test_util.py
-        pytest tests/test_xarray_netcdf_provider.py
-        pytest tests/test_xarray_zarr_provider.py
+      run: pytest tests/provider --ignore-glob='*_live.py'
+    - name: run Manager tests ⚙️
+      env:
+        POSTGRESQL_PASSWORD: ${{ secrets.DatabasePassword || 'postgres' }}
+      run: pytest tests/manager
+    - name: run other tests ⚙️
+      run: pytest tests/other --ignore-glob='*_live.py'
     - name: failed tests 🚩
       if: ${{ failure() }}
       run: |

docs/source/data-publishing/ogcapi-features.rst

@@ -570,7 +570,7 @@ Custom SQL Manipulator Plugin
 The provider supports a SQL-Manipulator-Plugin class. With this, the SQL statement could be manipulated. This is
 useful e.g. for authorization at row level or manipulation of the explain plan with hints. 
 
-An example and more information about that feature can be found in the test class in tests/test_oracle_provider.py.
+More information and examples about this feature can be found in ``tests/provider/test_oracle_provider.py``.
 
 .. _Parquet:
 

docs/source/data-publishing/ogcapi-tiles.rst

@@ -201,6 +201,16 @@ This code block shows how to configure pygeoapi to read map tiles from a WMTS.
                  min: 0
                  max: 20
 
+
+Providing custom Tile Matrix Set definitions
+--------------------------------------------
+
+By default, pygeoapi provides the ``WorldCRS84Quad`` and ``WebMercatorQuad`` TMS
+definitions, for tile providers to use accordingly. Additional TMS definitions
+may be added in pygeoapi's ``resources/definitions/tiles`` (for example, by adding
+TMS definition files directly, volume mapping / Docker ``COPY``, Docker Compose ``volumes``, etc.).
+
+
 Data access examples
 --------------------
 

docs/source/development.rst

@@ -11,12 +11,12 @@ The pygeoapi codebase exists at https://github.com/geopython/pygeoapi.
 Pull Requests and GitHub Actions
 --------------------------------
 
-A given GitHub Pull Request is evaluated against the following GitHub actions:
+A given GitHub Pull Request is evaluated against the following GitHub Actions:
 
-- main: mainline testing harness (as defined in ``tests``)
-- flake8: code linting
-- docs: documentation updates (for files updated in ``docs/**.rst``)
-- vulnerabilities: Trivy vulnerability scanning
+- ``main``: mainline testing harness (as defined in ``tests``)
+- ``flake8``: code linting
+- ``docs``: documentation updates (for files updated in ``docs/**.rst``)
+- ``vulnerabilities``: Trivy vulnerability scanning
 
 Testing
 -------
@@ -25,6 +25,11 @@ pygeoapi uses `pytest <https://docs.pytest.org>`_ for managing its automated tes
 exist in ``/tests`` and are developed for providers, formatters, processes, as well as the
 overall API.
 
+- API specific tests can be found in ``/tests/api``
+- Provider specific tests can be found in ``/tests/provider``
+- Manager specific tests can be found in ``/tests/manager``
+- Additional/other tests can be found in ``/tests/other``
+
 Tests can be run locally as part of development workflow.  They are also run on pygeoapi’s
 `GitHub Actions setup`_ against all commits and pull requests to the code repository.
 
@@ -57,24 +62,23 @@ Using pre-commit
 ^^^^^^^^^^^^^^^^
 
 You may optionally use `pre-commit`_ in order to check for linting and other static issues
-before committing changes. Pygeoapi's repo includes a ``.pre-commit.yml``
+before committing changes. pygeoapi's repo includes a ``.pre-commit.yml``
 file, check the pre-commit docs on how to set it up - in a nutshell:
 
-- pre-commit is mentioned in pygeoapi's ``requirements-dev.txt`` file, so it will be included
-  when you pip install those
-- run ``pre-commit install`` once in order to install its git commit hooks.
-- optionally, run ``pre-commit run --all-files``, which will run all pre-commit hooks for all files in the repo.
-  This also prepares the pre-commit environment.
-- from now on, whenever you do a ``git commit``, the pre-commit hooks will run and the commit
-  will only be done if all checks pass
+- pre-commit is part of ``requirements-dev.txt`` file, so it will be included when installing same
+- run ``pre-commit install`` once in order to install its git commit hooks
+- optionally, run ``pre-commit run --all-files``, which will run all pre-commit hooks for all files
+  in the repository.  Note that this also prepares the pre-commit environment
+- When subsequent ``git commit`` commands are run, the pre-commit hooks will run and commit
+  on passing checks
 
 Building the documentation
 --------------------------
 
-To build the documentation in pygeoapi we use `Sphinx`_. The documentation is located in the docs folder.
+Documentation is managed using `Sphinx`_ and located in the ``docs`` directory.
 
 .. note::
-   For the following instructions to work, you must be located in the root folder of pygeoapi.
+   The following commands should be run from the root folder of the repository.
 
 Install the dependencies necessary for building the documentation using the following command:
 
@@ -93,12 +97,17 @@ Or using the following ``make`` command:
 
 .. code-block:: bash
 
-   make -C docs html
+   make -C docs/ html
+
+After building the documentation, the ``docs/build`` directory will contain the generated documentation. 
+
+To view the generated documentation locally, use one of the following options:
 
-After building the documentation, the folder ``docs/build`` will contain the website generated with the documentation. 
-Add the folder to a web server or open the file ``docs/build/html/index.html`` file in a web browser to see the contents of the documentation.
+- run ``python3 -m http.server`` and navigate to ``http://localhost:8000`` in a web browser.  To use a different port, use ``python3 -m http.server 8001``, for example, and navigate to ``http://localhost:8001``
+- add the directory to a web server
+- open the file ``docs/build/html/index.html`` file in a web browser
 
-The documentation is hosted on `Read the Docs`_. It is automatically generated from the contents of the ``master`` branch on GitHub.
+The documentation is hosted on `Read the Docs`_ and automatically generated from the contents of the ``master`` branch on GitHub.
 
 The file ``.readthedocs.yaml`` contains the configuration of the Read the Docs build. Refer to the `Read the Docs configuration file`_ documentation for more information.
 

docs/source/index.rst

@@ -21,7 +21,7 @@ reference documentation on all aspects of the project.
 
 .. note::
 
-   Did you know about the `pygeoapi workshop`_?  Ready to get your hands dirty?  Dive in!
+   Did you know about the `Diving into pygeoapi workshop`_?  Ready to get your hands dirty?  Dive in!
 
 .. toctree::
    :maxdepth: 4
@@ -61,4 +61,4 @@ Indices and tables
 * :ref:`modindex`
 * :ref:`search`
 
-.. _`pygeoapi workshop`: https://dive.pygeoapi.io
+.. _`Diving into pygeoapi workshop`: https://dive.pygeoapi.io

docs/source/ogc-compliance.rst

@@ -16,4 +16,4 @@ Setting up your own CITE testing instance
 -----------------------------------------
 
 Please see the pygeoapi `OGC Compliance <https://github.com/geopython/pygeoapi/wiki/OGCCompliance>`_
-for up to date information as well as technical details on setting up your own CITE instance.
+wiki page for up to date information as well as technical details on setting up your own CITE instance.

pygeoapi/api/__init__.py

@@ -63,7 +63,7 @@
     ProviderConnectionError, ProviderGenericError, ProviderTypeError)
 
 from pygeoapi.util import (
-    CrsTransformSpec, TEMPLATES, UrlPrefetcher, dategetter,
+    CrsTransformSpec, TEMPLATESDIR, UrlPrefetcher, dategetter,
     filter_dict_by_key_value, filter_providers_by_type, get_api_rules,
     get_base_url, get_provider_by_type, get_provider_default, get_typed_value,
     get_crs_from_uri, get_supported_crs_list, render_j2_template, to_json,
@@ -555,7 +555,7 @@ def __init__(self, config, openapi):
         self.default_locale = self.locales[0]
 
         if 'templates' not in self.config['server']:
-            self.config['server']['templates'] = {'path': TEMPLATES}
+            self.config['server']['templates'] = {'path': TEMPLATESDIR}
 
         if 'pretty_print' not in self.config['server']:
             self.config['server']['pretty_print'] = False

pygeoapi/config.py

@@ -3,7 +3,7 @@
 # Authors: Tom Kralidis <tomkralidis@gmail.com>
 #          Francesco Bartoli <xbartolone@gmail.com>
 #
-# Copyright (c) 2022 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 # Copyright (c) 2025 Francesco Bartoli
 #
 # Permission is hereby granted, free of charge, to any person
@@ -36,7 +36,7 @@
 import os
 import yaml
 
-from pygeoapi.util import to_json, yaml_load, THISDIR
+from pygeoapi.util import SCHEMASDIR, to_json, yaml_load
 
 LOGGER = logging.getLogger(__name__)
 
@@ -65,7 +65,7 @@ def get_config(raw: bool = False) -> dict:
 def load_schema() -> dict:
     """ Reads the JSON schema YAML file. """
 
-    schema_file = THISDIR / 'schemas' / 'config' / 'pygeoapi-config-0.x.yml'
+    schema_file = SCHEMASDIR / 'config' / 'pygeoapi-config-0.x.yml'
 
     with schema_file.open() as fh2:
         return yaml_load(fh2)

pygeoapi/models/provider/base.py

@@ -1,12 +1,12 @@
-# ****************************** -*-
-# flake8: noqa
 # =================================================================
 #
 # Authors: Antonio Cerciello <anto.nio.cerciello@gmail.com>
 #          Francesco Bartoli <xbartolone@gmail.com>
+#          Tom Kralidis <tomkralidis@gmail.com>
 #
 # Copyright (c) 2022 Antonio Cerciello
 # Copyright (c) 2025 Francesco Bartoli
+# Copyright (c) 2025 Tom Kralidis
 #
 # Permission is hereby granted, free of charge, to any person
 # obtaining a copy of this software and associated documentation
@@ -40,10 +40,9 @@
 import pydantic
 from pydantic import BaseModel
 
-from pygeoapi.util import THISDIR
+from pygeoapi.util import DEFINITIONSDIR
 
-
-TMS_DIR = THISDIR / 'resources' / 'tilematrixsets'
+TMS_DIR = DEFINITIONSDIR / 'tiles'
 
 
 class TilesMetadataFormat(str, Enum):
@@ -93,10 +92,19 @@ def __init__(self, directory: Path):
         self.directory = directory
 
     def load_from_file(self, filename: str) -> TileMatrixSetEnumType:
-        """Load a single TMS JSON file."""
+        """
+        Load a single TMS JSON file.
+
+        :param filename: filename of TMS
+
+        :returns: `TileMatrixSetEnumType` of TMS
+        """
+
         filepath = self.directory / filename
+
         with filepath.open(encoding='utf-8') as fh:
             data = json.load(fh)
+
         return TileMatrixSetEnumType(
             tileMatrixSet=data["id"],
             tileMatrixSetURI=data["uri"],
@@ -108,12 +116,18 @@ def load_from_file(self, filename: str) -> TileMatrixSetEnumType:
         )
 
     def create_enum(self) -> Enum:
-        """Create an Enum with all TileMatrixSets in the directory."""
+        """
+        Create an Enum with all TileMatrixSets in the directory.
+
+        :returns: `Enum` of `TileMatrixSetEnum`
+        """
+
         members = {}
-        for file_path in self.directory.glob("*.json"):
-            tms = self.load_from_file(file_path.name)
-            enum_name = tms.tileMatrixSet.upper().replace("-", "").replace(" ", "")
+        for filepath in self.directory.glob("*.json"):
+            tms = self.load_from_file(filepath.name)
+            enum_name = tms.tileMatrixSet.upper().replace("-", "").replace(" ", "")  # noqa
             members[enum_name] = tms
+
         return Enum("TileMatrixSetEnum", members)
 
 
@@ -199,7 +213,7 @@ class TileSetMetadata(BaseModel):
     license_: Optional[str] = None
     # Restrictions on the availability of the Tile Set that the user needs to
     # be aware of before using or redistributing the Tile Set
-    accessConstraints: Optional[AccessConstraintsEnum] = AccessConstraintsEnum.UNCLASSIFIED
+    accessConstraints: Optional[AccessConstraintsEnum] = AccessConstraintsEnum.UNCLASSIFIED  # noqa
     # Media types available for the tiles
     mediaTypes:  Optional[List[str]] = None
     # Type of data represented in the tileset

pygeoapi/openapi.py

@@ -4,7 +4,7 @@
 # Authors: Francesco Bartoli <xbartolone@gmail.com>
 # Authors: Ricardo Garcia Silva <ricardo.garcia.silva@geobeyond.it>
 #
-# Copyright (c) 2024 Tom Kralidis
+# Copyright (c) 2025 Tom Kralidis
 # Copyright (c) 2025 Francesco Bartoli
 # Copyright (c) 2023 Ricardo Garcia Silva
 #
@@ -47,7 +47,7 @@
 from pygeoapi.api import all_apis
 from pygeoapi.models.openapi import OAPIFormat
 from pygeoapi.util import (filter_dict_by_key_value, to_json, yaml_load,
-                           get_api_rules, get_base_url)
+                           get_api_rules, get_base_url, SCHEMASDIR)
 
 LOGGER = logging.getLogger(__name__)
 
@@ -62,8 +62,6 @@
     'pygeoapi': 'https://raw.githubusercontent.com/geopython/pygeoapi/master/pygeoapi/schemas/config/pygeoapi-config-0.x.yml'  # noqa
 }
 
-THISDIR = os.path.dirname(os.path.realpath(__file__))
-
 
 def get_ogc_schemas_location(server_config: dict) -> str:
     """
@@ -706,10 +704,9 @@ def get_visible_collections(cfg: dict) -> dict:
 
 
 def get_config_schema():
-    schema_file = os.path.join(THISDIR, 'schemas', 'config',
-                               'pygeoapi-config-0.x.yml')
+    schema_file = SCHEMASDIR / 'config' / 'pygeoapi-config-0.x.yml'
 
-    with open(schema_file) as fh2:
+    with schema_file.open() as fh2:
         return yaml_load(fh2)
 
 
@@ -957,10 +954,9 @@ def validate_openapi_document(instance_dict: dict) -> bool:
     :returns: `bool` of validation
     """
 
-    schema_file = os.path.join(THISDIR, 'schemas', 'openapi',
-                               'openapi-3.0.x.json')
+    schema_file = SCHEMASDIR / 'openapi' / 'openapi-3.0.x.json'
 
-    with open(schema_file) as fh2:
+    with schema_file.open() as fh2:
         schema_dict = json.load(fh2)
         jsonschema_validate(instance_dict, schema_dict)