Vector default updates

docs/technical_documentation/concepts/clickhouse.rst

@@ -27,9 +27,16 @@ the variable **ASPECTS_RAW_XAPI_TABLE**:
 
 .. code-block:: yaml
 
-    ASPECTS_XAPI_DATABASE: "xapi"
+    # Derived from ASPECTS_XAPI_SOURCE: "openedx" (ASPECTS_VECTOR_DATABASE) when
+    # using Vector, "xapi" (RALPH_DATABASE) when using Ralph
+    ASPECTS_XAPI_DATABASE: "openedx"
     ASPECTS_RAW_XAPI_TABLE: "xapi_events_all"
 
+Since ``ASPECTS_XAPI_DATABASE`` changes with the pipeline, the Alembic migration state is kept in a
+separate, fixed database controlled by **ASPECTS_ALEMBIC_MIGRATIONS_DATABASE** (defaults to the
+value of ``RALPH_DATABASE``, ``xapi``). This prevents Alembic from losing track of which migrations
+have run when switching between Ralph and Vector.
+
 From here, the main table is split into different tables and views for performance and reporting purposes:
 
 - Tables group xAPI statements by type (video, problem, enrollment, etc.)

docs/technical_documentation/concepts/pipelines.rst

@@ -3,10 +3,29 @@ Pipelines
 
 Aspects provide two Pipelines which are detailed below.
 
+Vector Pipeline
+###############
+
+As of Aspects version 5.0, the Vector pipeline is the default pipeline (previous
+versions defaulted to a Celery / Ralph pipeline). It works by capturing the standard
+output from the LMS logs and sending them directly to configured "sinks" or data
+destinations. It implements two similar pipelines: one for xAPI data (enabled by default)
+and one for tracking logs (disabled by default).
+
+Vector is lighter weight, and generally data will arrive faster.
+It can also write a copy of every xAPI statement to an S3 compatible bucket as a backup,
+which can later be restored to ClickHouse (see :ref:`backfill_s3`).
+
+To learn more about Vector, see the `Vector documentation <https://vector.dev/docs/>`_.
+
+To configure Vector as your pipeline, see the :ref:`Quick Start - Vector guide <quick-start-vector>`.
+
+
 Ralph Pipeline
 ##############
 
-The Ralph pipeline is the default pipeline, and is the most robust. It will retry the
+Prior to version 5.0, the Ralph pipeline was the default. It is now an alternative
+pipeline, and remains the most robust. It will retry the
 most important failed events, and will catch most duplicates before they hit the database.
 This pipeline consist of a plugin in the LMS (`event-routing-backends`) that will send
 through HTTP the events to the Ralph API.
@@ -19,18 +38,3 @@ Ralph is for sharing xAPI data using the LRS standard.
 To learn more about Ralph, see the `Ralph documentation <https://openfun.github.io/ralph/>`_.
 
 To configure Ralph as your pipeline, see the :ref:`Quick Start - Ralph guide <quick-start-ralph>`.
-
-Vector Pipeline
-###############
-
-The Vector pipeline instead works by capturing the standard output from the LMS logs
-and sending them directly to configured "sinks" or data destinations. It implements two
-similar pipelines, one for xAPI data and one for tracking logs.
-
-Vector is lighter weight, and generally data will arrive a little faster, but doesn’t retry. 
-It can also be a good choice if you want to add other listeners for that data 
-(ex: to store xAPI statements to S3).
-
-To learn more about Vector, see the `Vector documentation <https://vector.dev/docs/>`_.
-
-To configure Vector as your pipeline, see the :ref:`Quick Start - Vector guide <quick-start-vector>`.

docs/technical_documentation/concepts/ralph.rst

@@ -14,10 +14,13 @@ Although Ralph has usages such as:
 - Validate xAPI statements.
 - Store events to different backends `backends <https://openfun.github.io/ralph/latest/features/backends/>`_.
 
-In the aspects project, Ralph is optionally used as the API server that connects Open edX
+In the aspects project, Ralph is an optional API server that connects Open edX
 and Clickhouse database. Ralph receives the xAPI statements from Open edX and stores them
 in the Clickhouse database after validating the data.
 
+To use Ralph as your xAPI transport, you must set ``ASPECTS_XAPI_SOURCE: ralph`` and
+``RUN_RALPH: True`` in your Tutor configuration.
+
 By default, Ralph is connected to the Open edX platform via Event Routing Backends without any filter
 and receives all the xAPI statements. To learn more about event-routing-backends, please
 refer to the `documentation <https://event-routing-backends.readthedocs.io/en/latest/>`_.

docs/technical_documentation/concepts/vector.rst

@@ -4,7 +4,7 @@ Vector
 ******
 
 Vector is lightweight and ultra-fast tool for building observability pipelines.
-In the Aspects project, Vector can optionally be used as a replacement for Ralph to
+As of Aspects version 5.0, Vector is the default tool used to
 capture xAPI learner statements in the ClickHouse database, and/or as a way to
 store raw tracking log statements. It can be used as a general purpose log collector
 and forwarder.
@@ -28,21 +28,35 @@ Vector can be deployed in two roles:
 - `Aggregator <https://vector.dev/docs/setup/deployment/roles/#aggregator>`_ - Receives data
   from Vector agents and sends it to a destination.
 
+Aspects uses both roles, depending on the deployment type:
 
-We use the role `Agent` to collect tracking logs and xAPI events from the lms and send it to
-the Clickhouse sink.
+- In Tutor ``local`` and ``dev`` (Docker Compose) deployments a single ``vector`` container runs as an
+  Agent. It reads the Docker logs of the LMS, CMS, worker, and job containers, transforms the events,
+  and writes them directly to the configured sinks.
+- In Tutor ``k8s`` deployments a ``vector-agent`` DaemonSet runs one Agent pod per node. The agents
+  read the Kubernetes pod logs for the Open edX containers and forward the raw log lines to a
+  ``vector-aggregator`` StatefulSet. The aggregator does the parsing and transforms and writes to the
+  configured sinks, using a persistent volume for disk buffers so events are not lost if a sink is
+  temporarily unavailable.
 
-For instance, Aspects has Sources configured for xAPI logging statements (optionally
-generated by event-routing-backends), and for tracking log statements. Each of these have their
-own Transforms, which validate that the log statements are JSON and forward them to Sinks which
-store them in ClickHouse tables.
+Aspects has Sources configured for xAPI logging statements (generated in-process by
+event-routing-backends), and for tracking log statements. Each of these have their own Transforms,
+which validate that the log statements are JSON and forward them to Sinks which store them in
+ClickHouse tables.
 
 Those tables are controlled by the variables:
 
 .. code-block:: yaml
 
     ASPECTS_VECTOR_DATABASE: "openedx"
     ASPECTS_VECTOR_RAW_TRACKING_LOGS_TABLE: "_tracking"
-    ASPECTS_VECTOR_RAW_XAPI_TABLE: "xapi_events_all"
+    ASPECTS_RAW_XAPI_TABLE: "xapi_events_all"
+
+Optionally, xAPI events can also be written to an S3 compatible bucket at the same time they are
+written to ClickHouse. This provides a backup that can be restored with the
+``xapi_block_storage_backfill`` command, see :ref:`backfill_s3` for details.
+
+Configuration options for both the sinks and the aggregator are described in the
+:ref:`Quick Start - Vector guide <quick-start-vector>`.
 
 To learn more about Vector, see the `Vector documentation <https://vector.dev/docs/>`_.

docs/technical_documentation/how-tos/02_upgrade.rst

@@ -57,9 +57,42 @@ v1.x             Nutmeg through Quince
 v2.x             Redwood through Teak
 v3.x             Ulmo
 v4.x             Verawood and later
+v5.x             Verawood and later
 ===============  ======================================
 
 
+Upgrading v4.x to v5.x
+----------------------
+
+Breaking Changes
+================
+
+The default data pipeline has changed from Ralph to Vector. This change improves performance and simplifies the architecture by eliminating the need to scale multiple Ralph containers and Celery workers for high-throughput scenarios.
+
+Key changes:
+
+- Vector is now the default for xAPI event ingestion (``RUN_VECTOR: True``, ``RUN_RALPH: False``, ``ASPECTS_XAPI_SOURCE: vector``)
+- The ``ASPECTS_VECTOR_RAW_XAPI_TABLE`` setting has been replaced with ``ASPECTS_RAW_XAPI_TABLE``
+- The default database has changed from ``xapi`` (Ralph) to ``openedx`` (Vector). ``ASPECTS_XAPI_DATABASE`` is now derived from ``ASPECTS_XAPI_SOURCE`` and should not normally be set directly.
+- Alembic migration state is now stored in the database named by the new ``ASPECTS_ALEMBIC_MIGRATIONS_DATABASE`` setting, which defaults to ``RALPH_DATABASE`` (``xapi``). This keeps the migration history stable when switching pipelines. Existing installs should leave this at the default so Alembic does not try to re-run all migrations.
+- On Kubernetes, Vector is now deployed as a ``vector-agent`` DaemonSet plus a ``vector-aggregator`` StatefulSet with a persistent volume. The previous single ``vector`` DaemonSet configuration (``k8s.toml``) has been removed. See :ref:`vector`.
+- A new, optional S3 sink can back up xAPI events and a new ``xapi_block_storage_backfill`` command restores them. See :ref:`backfill_s3`.
+
+To keep using Ralph as your data pipeline:
+
+.. code-block:: bash
+
+   tutor config save --set ASPECTS_XAPI_SOURCE=ralph
+   tutor config save --set RUN_RALPH=True
+   tutor config save --set RUN_VECTOR=False
+
+This will configure Aspects to use Ralph with the ``xapi`` database, preserving your existing data.
+
+If you have customized ``ASPECTS_VECTOR_RAW_XAPI_TABLE`` in your configuration, update it to use ``ASPECTS_RAW_XAPI_TABLE`` instead.
+
+For new installations or users switching to Vector, your data will be stored in the ``openedx`` database. You can migrate existing data from the ``xapi`` database to ``openedx`` if needed.
+
+
 Upgrading v3.x to v4.x
 ----------------------
 
@@ -68,8 +101,9 @@ Breaking Changes
 
 Aspects v4 now uses Python 3.12 to match the Verawood release of Tutor. If you are running a named release before Verawood, you will need to upgrade to Verawood or later before upgrading Aspects to v4.
 
+
 Upgrading v2.5 to v3.x
-----------------------
+-----------------------
 
 Breaking Changes
 ================

docs/technical_documentation/how-tos/aspects_production_configuration.rst

@@ -10,6 +10,29 @@ Aspects can be configured to send xAPI events to ClickHouse in several different
 
 At a high level the options are:
 
+Vector (default)
+----------------
+
+**Recommended for:** Most deployments, from resource-constrained Tutor local environments to larger production stacks.
+
+Vector is a log forwarding service that monitors the logs from docker containers or Kubernetes pods. It writes events directly to ClickHouse and automatically batches events based on volume. The LMS is configured to transform and log xAPI events in-process and Vector picks them up by reading the logs.
+
+Pros:
+
+- Removes the need to run or scale Ralph
+- Automatic batching adjustments
+- Fastest delivery times to ClickHouse
+- Vector failures do not impact other systems
+- Allows to backup and restore data from an S3 compatible backend
+
+Cons:
+
+- It is a new service for most operators
+- Events are not de-duplicated before insert, which can result in some duplicate or incorrect data in a log replay or disaster recovery situation
+- Needs a Vector agent pod on every Kubernetes node, plus an aggregator with persistent storage
+- When run in-process, adds a small amount of overhead to any LMS request that sends an xAPI statement
+
+
 Celery tasks without batching (default as of 1.0.0)
 ---------------------------------------------------
 
@@ -52,29 +75,6 @@ Cons:
 - Batching is not as well tested (as of Redwood) and may have edge cases until it has been used in production
 
 
-Vector
-------
-
-**Recommended for:** Resource-constrained Tutor local environments, experienced operators on larger deployments.
-
-Vector is a log forwarding service that monitors the logs from docker containers or Kubernetes pods. It writes events directly to ClickHouse and automatically batches events based on volume. The LMS can be configured to transform and log xAPI events in-process and Vector will pick them up by reading the logs.
-
-Pros:
-
-- Removes the need to run or scale Ralph
-- Automatic batching adjustments
-- Fastest delivery times to ClickHouse
-- Vector failures do not impact other systems
-
-Cons:
-
-- It is a new service for most operators
-- Events are not de-duplicated before insert, which can result in some (mostly temporary) incorrect data in a disaster recovery
-- Disaster recovery hasn't been tested with Aspects yet
-- Needs a pod run for every LMS or CMS Kubernetes worker
-- When run in-process, adds a small amount of overhead to any LMS request that sends an xAPI statement
-
-
 Event Bus (experimental)
 ------------------------
 
@@ -108,7 +108,9 @@ When in doubt, the simplest place to start with a production configuration is Ce
 Vector
 ------
 
-Generally the Aspects created Vector configuration should work in most cases. In Kubernetes environments you will need to make sure that a Vector pod is attached to each LMS/CMS worker.
+Generally the Aspects created Vector configuration should work in most cases. In Kubernetes environments Aspects deploys a ``vector-agent`` DaemonSet, which reads pod logs on every node, and a ``vector-aggregator`` StatefulSet, which parses the events and writes them to ClickHouse. The aggregator uses a persistent volume for its disk buffers, so make sure your cluster can provision a volume of ``ASPECTS_VECTOR_AGGREGATOR_STORAGE_SIZE``. You can scale the aggregator with ``ASPECTS_VECTOR_AGGREGATOR_REPLICAS``.
+
+We recommend enabling the S3 backup sink in production so that events can be restored quickly if ClickHouse loses data. See the :ref:`Quick Start - Vector guide <quick-start-vector>` for the settings and :ref:`backfill_s3` for how to restore.
 
 Event bus
 ---------
@@ -314,7 +316,8 @@ This is the time between now and the last xAPI event arriving. The frequency of
         count(*) as ttl_count,
         max(emission_time) as most_recent,
         date_diff('second', max(emission_time), now()) as lag_seconds
-    FROM xapi.xapi_events_all
+    -- use xapi.xapi_events_all if you are running the Ralph pipeline
+    FROM openedx.xapi_events_all
     FINAL
     FORMAT JSON
 

docs/technical_documentation/how-tos/aspects_troubleshooting.rst

@@ -108,7 +108,8 @@ When running with the Vector data pipeline:
 
 - lms
 - cms
-- vector
+- vector (Tutor local / dev)
+- vector-agent and vector-aggregator (Tutor k8s)
 
 When running with the event bus data pipeline:
 

docs/technical_documentation/how-tos/aspects_without_tutor.rst

@@ -141,7 +141,7 @@ In the next steps you'll combine the new services created by Tutor with your exi
 - superset / superset-worker / superset-beat
 - clickhouse (if enabled)
 - ralph (if enabled)
-- vector (if enabled)
+- vector (if enabled; on Kubernetes this is a vector-agent DaemonSet and a vector-aggregator StatefulSet)
 - aspects-docs (if enabled)
 
 Step 2a: Docker Compose

docs/technical_documentation/how-tos/backfill.rst

@@ -74,8 +74,55 @@ by:
   to your LMS performance)
 
 If you are running into repeated problems, you may wish to transform the tracking logs to file(s) on
-S3 or Minio that can be loaded directly into the ClickHouse ``xapi.xapi_events_all`` table using
-the ClickHouse S3 table function.
+S3 or Minio that can be loaded directly into the ClickHouse raw xAPI table
+(``openedx.xapi_events_all`` for Vector, ``xapi.xapi_events_all`` for Ralph) using the ClickHouse
+S3 table function.
+
+
+.. _backfill_s3:
+
+Restore xAPI data from the Vector S3 backup
+###########################################
+
+If you have enabled the Vector S3 sink (see :ref:`quick-start-vector`), every xAPI event is also
+written to your bucket as it is ingested. Those files can be loaded back into ClickHouse without
+replaying tracking logs, which is much faster and is useful for:
+
+- Restoring data after a ClickHouse outage or data loss
+- Importing data from another environment that writes to the same bucket
+- Re-processing historical events
+
+The ``xapi_block_storage_backfill`` command runs on the ClickHouse service and inserts the matching
+files directly into the raw xAPI table using the ClickHouse S3 table function. It uses the
+``ASPECTS_XAPI_S3_*`` settings for the bucket, endpoint, and credentials.
+
+.. code-block:: console
+
+    # Import every file in the bucket
+    tutor [dev|local|k8s] do xapi_block_storage_backfill
+
+    # Filter by date, from year down to hour (24 hour clock). Single and double
+    # digit values are equivalent.
+    tutor local do xapi_block_storage_backfill --year 2026 --month 3
+    tutor local do xapi_block_storage_backfill --year 2026 --month 03 --day 19
+    tutor local do xapi_block_storage_backfill --year 2026 --month 03 --day 19 --hour 14
+
+    # Or give a glob path inside the bucket directly. This cannot be combined
+    # with the date options.
+    tutor local do xapi_block_storage_backfill --path 'xapi/2026/03/19/14/*.log.zst'
+
+Events restored this way may already exist in ClickHouse, so the command can optionally run an
+``OPTIMIZE TABLE ... FINAL`` on the raw table and the main downstream tables afterwards to collapse
+duplicates:
+
+.. code-block:: console
+
+    tutor local do xapi_block_storage_backfill --year 2026 --month 3 --deduplicate
+
+.. warning::
+
+    ``OPTIMIZE TABLE FINAL`` can be resource intensive on large tables. Run it during low traffic
+    periods if you have a large dataset.
 
 
 .. _backfill_course_blocks:

docs/technical_documentation/quickstarts/ralph.rst

@@ -5,16 +5,22 @@ Ralph
 
 Installation instructions for Aspects are available on the plugin site: https://github.com/openedx/tutor-contrib-aspects
 
-Ralph is the default option to send xAPI events to Clickhouse. To run it make sure to enable the `RUN_RALPH` option in the `config.yml` file.
+Ralph is an alternative option to send xAPI events to Clickhouse, providing full xAPI
+learning record store (LRS) statement support and deduplication (prior to Aspects version
+5.0, Ralph was the default). To use Ralph as your xAPI pipeline, you need to enable it
+and set it as the source in your `config.yml` file.
 
 .. code-block:: yaml
 
     RUN_RALPH: True
+    ASPECTS_XAPI_SOURCE: ralph
 
-    # We recommend only running Ralph or Vector for performance reasons, so
-    # suggest turning off Vector here
+    # We recommend only running one transport for performance reasons, so
+    # suggest turning off Vector if you are using Ralph for xAPI
     RUN_VECTOR: False
 
+When ``ASPECTS_XAPI_SOURCE`` is set to ``ralph``, the xAPI data will be stored in the database defined by ``RALPH_DATABASE`` (defaults to ``xapi``).
+
 
 Aspects provides the following configuration options: