diff --git a/docs/technical_documentation/concepts/clickhouse.rst b/docs/technical_documentation/concepts/clickhouse.rst index 31e1569..b0f7359 100644 --- a/docs/technical_documentation/concepts/clickhouse.rst +++ b/docs/technical_documentation/concepts/clickhouse.rst @@ -27,9 +27,16 @@ the variable **ASPECTS_RAW_XAPI_TABLE**: .. code-block:: yaml - ASPECTS_XAPI_DATABASE: "xapi" + # Derived from ASPECTS_XAPI_SOURCE: "openedx" (ASPECTS_VECTOR_DATABASE) when + # using Vector, "xapi" (RALPH_DATABASE) when using Ralph + ASPECTS_XAPI_DATABASE: "openedx" ASPECTS_RAW_XAPI_TABLE: "xapi_events_all" +Since ``ASPECTS_XAPI_DATABASE`` changes with the pipeline, the Alembic migration state is kept in a +separate, fixed database controlled by **ASPECTS_ALEMBIC_MIGRATIONS_DATABASE** (defaults to the +value of ``RALPH_DATABASE``, ``xapi``). This prevents Alembic from losing track of which migrations +have run when switching between Ralph and Vector. + From here, the main table is split into different tables and views for performance and reporting purposes: - Tables group xAPI statements by type (video, problem, enrollment, etc.) diff --git a/docs/technical_documentation/concepts/pipelines.rst b/docs/technical_documentation/concepts/pipelines.rst index ae41493..5c85271 100644 --- a/docs/technical_documentation/concepts/pipelines.rst +++ b/docs/technical_documentation/concepts/pipelines.rst @@ -3,10 +3,29 @@ Pipelines Aspects provide two Pipelines which are detailed below. +Vector Pipeline +############### + +As of Aspects version 5.0, the Vector pipeline is the default pipeline (previous +versions defaulted to a Celery / Ralph pipeline). It works by capturing the standard +output from the LMS logs and sending them directly to configured "sinks" or data +destinations. It implements two similar pipelines: one for xAPI data (enabled by default) +and one for tracking logs (disabled by default). + +Vector is lighter weight, and generally data will arrive faster. +It can also write a copy of every xAPI statement to an S3 compatible bucket as a backup, +which can later be restored to ClickHouse (see :ref:`backfill_s3`). + +To learn more about Vector, see the `Vector documentation `_. + +To configure Vector as your pipeline, see the :ref:`Quick Start - Vector guide `. + + Ralph Pipeline ############## -The Ralph pipeline is the default pipeline, and is the most robust. It will retry the +Prior to version 5.0, the Ralph pipeline was the default. It is now an alternative +pipeline, and remains the most robust. It will retry the most important failed events, and will catch most duplicates before they hit the database. This pipeline consist of a plugin in the LMS (`event-routing-backends`) that will send through HTTP the events to the Ralph API. @@ -19,18 +38,3 @@ Ralph is for sharing xAPI data using the LRS standard. To learn more about Ralph, see the `Ralph documentation `_. To configure Ralph as your pipeline, see the :ref:`Quick Start - Ralph guide `. - -Vector Pipeline -############### - -The Vector pipeline instead works by capturing the standard output from the LMS logs -and sending them directly to configured "sinks" or data destinations. It implements two -similar pipelines, one for xAPI data and one for tracking logs. - -Vector is lighter weight, and generally data will arrive a little faster, but doesn’t retry. -It can also be a good choice if you want to add other listeners for that data -(ex: to store xAPI statements to S3). - -To learn more about Vector, see the `Vector documentation `_. - -To configure Vector as your pipeline, see the :ref:`Quick Start - Vector guide `. diff --git a/docs/technical_documentation/concepts/ralph.rst b/docs/technical_documentation/concepts/ralph.rst index 5ddbcc3..e448719 100644 --- a/docs/technical_documentation/concepts/ralph.rst +++ b/docs/technical_documentation/concepts/ralph.rst @@ -14,10 +14,13 @@ Although Ralph has usages such as: - Validate xAPI statements. - Store events to different backends `backends `_. -In the aspects project, Ralph is optionally used as the API server that connects Open edX +In the aspects project, Ralph is an optional API server that connects Open edX and Clickhouse database. Ralph receives the xAPI statements from Open edX and stores them in the Clickhouse database after validating the data. +To use Ralph as your xAPI transport, you must set ``ASPECTS_XAPI_SOURCE: ralph`` and +``RUN_RALPH: True`` in your Tutor configuration. + By default, Ralph is connected to the Open edX platform via Event Routing Backends without any filter and receives all the xAPI statements. To learn more about event-routing-backends, please refer to the `documentation `_. diff --git a/docs/technical_documentation/concepts/vector.rst b/docs/technical_documentation/concepts/vector.rst index 419aa8d..666cc76 100644 --- a/docs/technical_documentation/concepts/vector.rst +++ b/docs/technical_documentation/concepts/vector.rst @@ -4,7 +4,7 @@ Vector ****** Vector is lightweight and ultra-fast tool for building observability pipelines. -In the Aspects project, Vector can optionally be used as a replacement for Ralph to +As of Aspects version 5.0, Vector is the default tool used to capture xAPI learner statements in the ClickHouse database, and/or as a way to store raw tracking log statements. It can be used as a general purpose log collector and forwarder. @@ -28,14 +28,21 @@ Vector can be deployed in two roles: - `Aggregator `_ - Receives data from Vector agents and sends it to a destination. +Aspects uses both roles, depending on the deployment type: -We use the role `Agent` to collect tracking logs and xAPI events from the lms and send it to -the Clickhouse sink. +- In Tutor ``local`` and ``dev`` (Docker Compose) deployments a single ``vector`` container runs as an + Agent. It reads the Docker logs of the LMS, CMS, worker, and job containers, transforms the events, + and writes them directly to the configured sinks. +- In Tutor ``k8s`` deployments a ``vector-agent`` DaemonSet runs one Agent pod per node. The agents + read the Kubernetes pod logs for the Open edX containers and forward the raw log lines to a + ``vector-aggregator`` StatefulSet. The aggregator does the parsing and transforms and writes to the + configured sinks, using a persistent volume for disk buffers so events are not lost if a sink is + temporarily unavailable. -For instance, Aspects has Sources configured for xAPI logging statements (optionally -generated by event-routing-backends), and for tracking log statements. Each of these have their -own Transforms, which validate that the log statements are JSON and forward them to Sinks which -store them in ClickHouse tables. +Aspects has Sources configured for xAPI logging statements (generated in-process by +event-routing-backends), and for tracking log statements. Each of these have their own Transforms, +which validate that the log statements are JSON and forward them to Sinks which store them in +ClickHouse tables. Those tables are controlled by the variables: @@ -43,6 +50,13 @@ Those tables are controlled by the variables: ASPECTS_VECTOR_DATABASE: "openedx" ASPECTS_VECTOR_RAW_TRACKING_LOGS_TABLE: "_tracking" - ASPECTS_VECTOR_RAW_XAPI_TABLE: "xapi_events_all" + ASPECTS_RAW_XAPI_TABLE: "xapi_events_all" + +Optionally, xAPI events can also be written to an S3 compatible bucket at the same time they are +written to ClickHouse. This provides a backup that can be restored with the +``xapi_block_storage_backfill`` command, see :ref:`backfill_s3` for details. + +Configuration options for both the sinks and the aggregator are described in the +:ref:`Quick Start - Vector guide `. To learn more about Vector, see the `Vector documentation `_. diff --git a/docs/technical_documentation/how-tos/02_upgrade.rst b/docs/technical_documentation/how-tos/02_upgrade.rst index 575e9fa..2a35f60 100644 --- a/docs/technical_documentation/how-tos/02_upgrade.rst +++ b/docs/technical_documentation/how-tos/02_upgrade.rst @@ -57,9 +57,42 @@ v1.x Nutmeg through Quince v2.x Redwood through Teak v3.x Ulmo v4.x Verawood and later +v5.x Verawood and later =============== ====================================== +Upgrading v4.x to v5.x +---------------------- + +Breaking Changes +================ + +The default data pipeline has changed from Ralph to Vector. This change improves performance and simplifies the architecture by eliminating the need to scale multiple Ralph containers and Celery workers for high-throughput scenarios. + +Key changes: + +- Vector is now the default for xAPI event ingestion (``RUN_VECTOR: True``, ``RUN_RALPH: False``, ``ASPECTS_XAPI_SOURCE: vector``) +- The ``ASPECTS_VECTOR_RAW_XAPI_TABLE`` setting has been replaced with ``ASPECTS_RAW_XAPI_TABLE`` +- The default database has changed from ``xapi`` (Ralph) to ``openedx`` (Vector). ``ASPECTS_XAPI_DATABASE`` is now derived from ``ASPECTS_XAPI_SOURCE`` and should not normally be set directly. +- Alembic migration state is now stored in the database named by the new ``ASPECTS_ALEMBIC_MIGRATIONS_DATABASE`` setting, which defaults to ``RALPH_DATABASE`` (``xapi``). This keeps the migration history stable when switching pipelines. Existing installs should leave this at the default so Alembic does not try to re-run all migrations. +- On Kubernetes, Vector is now deployed as a ``vector-agent`` DaemonSet plus a ``vector-aggregator`` StatefulSet with a persistent volume. The previous single ``vector`` DaemonSet configuration (``k8s.toml``) has been removed. See :ref:`vector`. +- A new, optional S3 sink can back up xAPI events and a new ``xapi_block_storage_backfill`` command restores them. See :ref:`backfill_s3`. + +To keep using Ralph as your data pipeline: + +.. code-block:: bash + + tutor config save --set ASPECTS_XAPI_SOURCE=ralph + tutor config save --set RUN_RALPH=True + tutor config save --set RUN_VECTOR=False + +This will configure Aspects to use Ralph with the ``xapi`` database, preserving your existing data. + +If you have customized ``ASPECTS_VECTOR_RAW_XAPI_TABLE`` in your configuration, update it to use ``ASPECTS_RAW_XAPI_TABLE`` instead. + +For new installations or users switching to Vector, your data will be stored in the ``openedx`` database. You can migrate existing data from the ``xapi`` database to ``openedx`` if needed. + + Upgrading v3.x to v4.x ---------------------- @@ -68,8 +101,9 @@ Breaking Changes Aspects v4 now uses Python 3.12 to match the Verawood release of Tutor. If you are running a named release before Verawood, you will need to upgrade to Verawood or later before upgrading Aspects to v4. + Upgrading v2.5 to v3.x ----------------------- +----------------------- Breaking Changes ================ diff --git a/docs/technical_documentation/how-tos/aspects_production_configuration.rst b/docs/technical_documentation/how-tos/aspects_production_configuration.rst index f6b6658..f6fe65e 100644 --- a/docs/technical_documentation/how-tos/aspects_production_configuration.rst +++ b/docs/technical_documentation/how-tos/aspects_production_configuration.rst @@ -10,6 +10,29 @@ Aspects can be configured to send xAPI events to ClickHouse in several different At a high level the options are: +Vector (default) +---------------- + +**Recommended for:** Most deployments, from resource-constrained Tutor local environments to larger production stacks. + +Vector is a log forwarding service that monitors the logs from docker containers or Kubernetes pods. It writes events directly to ClickHouse and automatically batches events based on volume. The LMS is configured to transform and log xAPI events in-process and Vector picks them up by reading the logs. + +Pros: + +- Removes the need to run or scale Ralph +- Automatic batching adjustments +- Fastest delivery times to ClickHouse +- Vector failures do not impact other systems +- Allows to backup and restore data from an S3 compatible backend + +Cons: + +- It is a new service for most operators +- Events are not de-duplicated before insert, which can result in some duplicate or incorrect data in a log replay or disaster recovery situation +- Needs a Vector agent pod on every Kubernetes node, plus an aggregator with persistent storage +- When run in-process, adds a small amount of overhead to any LMS request that sends an xAPI statement + + Celery tasks without batching (default as of 1.0.0) --------------------------------------------------- @@ -52,29 +75,6 @@ Cons: - Batching is not as well tested (as of Redwood) and may have edge cases until it has been used in production -Vector ------- - -**Recommended for:** Resource-constrained Tutor local environments, experienced operators on larger deployments. - -Vector is a log forwarding service that monitors the logs from docker containers or Kubernetes pods. It writes events directly to ClickHouse and automatically batches events based on volume. The LMS can be configured to transform and log xAPI events in-process and Vector will pick them up by reading the logs. - -Pros: - -- Removes the need to run or scale Ralph -- Automatic batching adjustments -- Fastest delivery times to ClickHouse -- Vector failures do not impact other systems - -Cons: - -- It is a new service for most operators -- Events are not de-duplicated before insert, which can result in some (mostly temporary) incorrect data in a disaster recovery -- Disaster recovery hasn't been tested with Aspects yet -- Needs a pod run for every LMS or CMS Kubernetes worker -- When run in-process, adds a small amount of overhead to any LMS request that sends an xAPI statement - - Event Bus (experimental) ------------------------ @@ -108,7 +108,9 @@ When in doubt, the simplest place to start with a production configuration is Ce Vector ------ -Generally the Aspects created Vector configuration should work in most cases. In Kubernetes environments you will need to make sure that a Vector pod is attached to each LMS/CMS worker. +Generally the Aspects created Vector configuration should work in most cases. In Kubernetes environments Aspects deploys a ``vector-agent`` DaemonSet, which reads pod logs on every node, and a ``vector-aggregator`` StatefulSet, which parses the events and writes them to ClickHouse. The aggregator uses a persistent volume for its disk buffers, so make sure your cluster can provision a volume of ``ASPECTS_VECTOR_AGGREGATOR_STORAGE_SIZE``. You can scale the aggregator with ``ASPECTS_VECTOR_AGGREGATOR_REPLICAS``. + +We recommend enabling the S3 backup sink in production so that events can be restored quickly if ClickHouse loses data. See the :ref:`Quick Start - Vector guide ` for the settings and :ref:`backfill_s3` for how to restore. Event bus --------- @@ -314,7 +316,8 @@ This is the time between now and the last xAPI event arriving. The frequency of count(*) as ttl_count, max(emission_time) as most_recent, date_diff('second', max(emission_time), now()) as lag_seconds - FROM xapi.xapi_events_all + -- use xapi.xapi_events_all if you are running the Ralph pipeline + FROM openedx.xapi_events_all FINAL FORMAT JSON diff --git a/docs/technical_documentation/how-tos/aspects_troubleshooting.rst b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst index b2f87d0..f53997c 100644 --- a/docs/technical_documentation/how-tos/aspects_troubleshooting.rst +++ b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst @@ -108,7 +108,8 @@ When running with the Vector data pipeline: - lms - cms -- vector +- vector (Tutor local / dev) +- vector-agent and vector-aggregator (Tutor k8s) When running with the event bus data pipeline: diff --git a/docs/technical_documentation/how-tos/aspects_without_tutor.rst b/docs/technical_documentation/how-tos/aspects_without_tutor.rst index 6d3f3b6..e866b59 100644 --- a/docs/technical_documentation/how-tos/aspects_without_tutor.rst +++ b/docs/technical_documentation/how-tos/aspects_without_tutor.rst @@ -141,7 +141,7 @@ In the next steps you'll combine the new services created by Tutor with your exi - superset / superset-worker / superset-beat - clickhouse (if enabled) - ralph (if enabled) -- vector (if enabled) +- vector (if enabled; on Kubernetes this is a vector-agent DaemonSet and a vector-aggregator StatefulSet) - aspects-docs (if enabled) Step 2a: Docker Compose diff --git a/docs/technical_documentation/how-tos/backfill.rst b/docs/technical_documentation/how-tos/backfill.rst index bb4313b..ffaea29 100644 --- a/docs/technical_documentation/how-tos/backfill.rst +++ b/docs/technical_documentation/how-tos/backfill.rst @@ -74,8 +74,55 @@ by: to your LMS performance) If you are running into repeated problems, you may wish to transform the tracking logs to file(s) on -S3 or Minio that can be loaded directly into the ClickHouse ``xapi.xapi_events_all`` table using -the ClickHouse S3 table function. +S3 or Minio that can be loaded directly into the ClickHouse raw xAPI table +(``openedx.xapi_events_all`` for Vector, ``xapi.xapi_events_all`` for Ralph) using the ClickHouse +S3 table function. + + +.. _backfill_s3: + +Restore xAPI data from the Vector S3 backup +########################################### + +If you have enabled the Vector S3 sink (see :ref:`quick-start-vector`), every xAPI event is also +written to your bucket as it is ingested. Those files can be loaded back into ClickHouse without +replaying tracking logs, which is much faster and is useful for: + +- Restoring data after a ClickHouse outage or data loss +- Importing data from another environment that writes to the same bucket +- Re-processing historical events + +The ``xapi_block_storage_backfill`` command runs on the ClickHouse service and inserts the matching +files directly into the raw xAPI table using the ClickHouse S3 table function. It uses the +``ASPECTS_XAPI_S3_*`` settings for the bucket, endpoint, and credentials. + +.. code-block:: console + + # Import every file in the bucket + tutor [dev|local|k8s] do xapi_block_storage_backfill + + # Filter by date, from year down to hour (24 hour clock). Single and double + # digit values are equivalent. + tutor local do xapi_block_storage_backfill --year 2026 --month 3 + tutor local do xapi_block_storage_backfill --year 2026 --month 03 --day 19 + tutor local do xapi_block_storage_backfill --year 2026 --month 03 --day 19 --hour 14 + + # Or give a glob path inside the bucket directly. This cannot be combined + # with the date options. + tutor local do xapi_block_storage_backfill --path 'xapi/2026/03/19/14/*.log.zst' + +Events restored this way may already exist in ClickHouse, so the command can optionally run an +``OPTIMIZE TABLE ... FINAL`` on the raw table and the main downstream tables afterwards to collapse +duplicates: + +.. code-block:: console + + tutor local do xapi_block_storage_backfill --year 2026 --month 3 --deduplicate + +.. warning:: + + ``OPTIMIZE TABLE FINAL`` can be resource intensive on large tables. Run it during low traffic + periods if you have a large dataset. .. _backfill_course_blocks: diff --git a/docs/technical_documentation/quickstarts/ralph.rst b/docs/technical_documentation/quickstarts/ralph.rst index fbe0ebf..73b7d2d 100644 --- a/docs/technical_documentation/quickstarts/ralph.rst +++ b/docs/technical_documentation/quickstarts/ralph.rst @@ -5,16 +5,22 @@ Ralph Installation instructions for Aspects are available on the plugin site: https://github.com/openedx/tutor-contrib-aspects -Ralph is the default option to send xAPI events to Clickhouse. To run it make sure to enable the `RUN_RALPH` option in the `config.yml` file. +Ralph is an alternative option to send xAPI events to Clickhouse, providing full xAPI +learning record store (LRS) statement support and deduplication (prior to Aspects version +5.0, Ralph was the default). To use Ralph as your xAPI pipeline, you need to enable it +and set it as the source in your `config.yml` file. .. code-block:: yaml RUN_RALPH: True + ASPECTS_XAPI_SOURCE: ralph - # We recommend only running Ralph or Vector for performance reasons, so - # suggest turning off Vector here + # We recommend only running one transport for performance reasons, so + # suggest turning off Vector if you are using Ralph for xAPI RUN_VECTOR: False +When ``ASPECTS_XAPI_SOURCE`` is set to ``ralph``, the xAPI data will be stored in the database defined by ``RALPH_DATABASE`` (defaults to ``xapi``). + Aspects provides the following configuration options: diff --git a/docs/technical_documentation/quickstarts/vector.rst b/docs/technical_documentation/quickstarts/vector.rst index 1195f53..ef7ae9f 100644 --- a/docs/technical_documentation/quickstarts/vector.rst +++ b/docs/technical_documentation/quickstarts/vector.rst @@ -5,25 +5,92 @@ Vector Installation instructions for Aspects are available on the plugin site: https://github.com/openedx/tutor-contrib-aspects -Vector is an alternative option to send xAPI events to Clickhouse. It can be run along with Ralph, but to optimize resources we encourage you to only use one. - -To configure Vector as the xAPI event handler, you can use the following configuration: +Vector is the default option to send xAPI events to Clickhouse in Aspects. It is enabled by default with the following settings: .. code-block:: yaml - # Disable ralph - RUN_RALPH: False - # Enable vector + # Default settings RUN_VECTOR: True - # Change the xAPI database to the one Vector uses - ASPECTS_XAPI_DATABASE: "openedx" + RUN_RALPH: False + ASPECTS_XAPI_SOURCE: vector + +When ``ASPECTS_XAPI_SOURCE`` is set to ``vector``, the xAPI data will be stored in the database defined by ``ASPECTS_VECTOR_DATABASE`` (defaults to ``openedx``). Aspects provides the following configuration options: .. code-block:: yaml - ASPECTS_CLICKHOUSE_VECTOR_USER: "vector" + # ClickHouse credentials used by Vector, generated automatically by Tutor + ASPECTS_CLICKHOUSE_VECTOR_USER: "ch_vector" ASPECTS_CLICKHOUSE_VECTOR_PASSWORD: "secure-password" + + # Which pipelines to run. Tracking logs contain PII and are off by default. + ASPECTS_VECTOR_STORE_XAPI: True + ASPECTS_VECTOR_STORE_TRACKING_LOGS: False + + # Database and tables that Vector writes to + ASPECTS_VECTOR_DATABASE: "openedx" + ASPECTS_RAW_XAPI_TABLE: "xapi_events_all" # The default name is used to keep backwards compatibility with Cairn ASPECTS_VECTOR_RAW_TRACKING_LOGS_TABLE: "_tracking" + + # Path to the Docker socket that Vector reads container logs from in + # Tutor local / dev. Only used with Docker Compose. + ASPECTS_DOCKER_HOST_SOCK_PATH: "/var/run/docker.sock" + + +S3 backup sink +############## + +Vector can write a copy of every xAPI event to an S3 compatible bucket (AWS S3, MinIO, etc.) at +the same time it writes to ClickHouse. This is a backup, not a fallback: events go to both sinks +in parallel. The S3 copy can later be restored into ClickHouse with the +``xapi_block_storage_backfill`` command, see :ref:`backfill_s3`. + +The sink is disabled until ``ASPECTS_XAPI_S3_BUCKET`` is set: + +.. code-block:: yaml + + ASPECTS_XAPI_S3_BUCKET: "xapi-events" + ASPECTS_XAPI_S3_REGION: "us-east-1" + # Only needed for non-AWS S3 compatible services, such as MinIO + ASPECTS_XAPI_S3_ENDPOINT: "http://minio:9000" + # If both are left empty Vector will use the default AWS credential chain + ASPECTS_XAPI_S3_ACCESS_KEY: "" + ASPECTS_XAPI_S3_SECRET_KEY: "" + + # Batching: a file is written when either limit is reached. Setting the + # timeout too low will create many small files in S3. + ASPECTS_XAPI_S3_SINK_MAX_EVENTS: "10000" + ASPECTS_XAPI_S3_SINK_TIMEOUT_SECS: "600" + +Files are written as zstd compressed, newline delimited JSON under the prefix +``xapi/////`` in the bucket. + + +Kubernetes aggregator +##################### + +In Tutor ``k8s`` deployments Vector runs as a ``vector-agent`` DaemonSet plus a +``vector-aggregator`` StatefulSet (see :ref:`vector`). The following settings control the +aggregator: + +.. code-block:: yaml + + # Port the agents send events to the aggregator on + ASPECTS_VECTOR_AGGREGATOR_PORT: "6000" + # Number of aggregator pods + ASPECTS_VECTOR_AGGREGATOR_REPLICAS: 1 + # Maximum size in bytes of the on-disk buffer for each sink (default 1 GiB). + # When the buffer is full Vector will block until the sink recovers. + ASPECTS_VECTOR_AGGREGATOR_BUFFER_MAX_SIZE: "1073741824" + # Size of the persistent volume claim backing the disk buffers + ASPECTS_VECTOR_AGGREGATOR_STORAGE_SIZE: "2Gi" + +``ASPECTS_VECTOR_AGGREGATOR_BUFFER_MAX_SIZE`` is also used for the disk buffer of the +single ``vector`` container in Docker Compose deployments. Make sure +``ASPECTS_VECTOR_AGGREGATOR_STORAGE_SIZE`` is large enough to hold the buffers for every +enabled sink. + +Any additional Vector configuration can be added with the ``vector-common-toml`` Tutor patch.