Set up airflow

[tdunning]

We need a practice instance of airflow to try out ideas.

This should use the following config options:

• db should be postgres
• webserver port 8091
• set up cron job to run every 30 seconds to pull from github and copy dags into airflow dir atomically (pull, cp to tmp, mv)
• logging to file with rotation

Questions

We need to answer and document a few questions:

• can we have a single airflow instance run production/staging with different user ID to protect data?
• do we need multiple airflow instances launched as different IDs to separate production and staging?
• if we have multiple instances, can we limit resource consumption before running with K8S?

Links

http://airflow.apache.org/docs/apache-airflow/stable/security/workload.html

[tdunning]

This relates to issue #7