Repository navigation

eea-crawler

Public

About

EEA Crawler contains the tasks (DAGs) used by Apache Airflow to index content from various EEA-Eionet websites into a central Elasticsearch (aka content hub).

airflow-dags crawler elasticsearch etl-pipeline indexing

Custom properties

1 star

3 watching

Report repository

Name		Name	Last commit message	Last commit date
dags		dags
logstash		logstash
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
DOCUMENTATION.md		DOCUMENTATION.md
Dockerfile		Dockerfile
README.md		README.md
docker-start.sh		docker-start.sh
pyproject.toml		pyproject.toml

Repository files navigation

Airflow and Logstash configurations for EEA-Crawler

See https://github.com/eea/eea.docker.airflow for integration and deployment.

Building this docker image will create a volume that contains the Python code DAGs required for the crawler operation.

Documentation for Airflow EEA Semantic Search Indexer

About

EEA Crawler contains the tasks (DAGs) used by Apache Airflow to index content from various EEA-Eionet websites into a central Elasticsearch (aka content hub).

airflow-dags crawler elasticsearch etl-pipeline indexing

Custom properties

1 star

3 watching

Report repository

Releases

Packages

Used by

Contributors

Languages