Add docs for Pixie Plugin System (#154)

aimichelle · web-flow · commit ca3c2a9624c4 · 2022-04-14T12:10:59.000-07:00
* plugin docs

Signed-off-by: Michelle Nguyen &lt;michellenguyen@pixielabs.ai&gt;

* add in reference index

Signed-off-by: Michelle Nguyen &lt;michellenguyen@pixielabs.ai&gt;

* update OTel Export link

Signed-off-by: Michelle Nguyen &lt;michellenguyen@pixielabs.ai&gt;

* address comments

Signed-off-by: Michelle Nguyen &lt;michellenguyen@pixielabs.ai&gt;
diff --git a/content/en/01-about-pixie/04-roadmap.md b/content/en/01-about-pixie/04-roadmap.md
@@ -30,4 +30,4 @@ We are building a data plane for Kubernetes rather than a full-fledged observabi
 * Non-Kubernetes environments
 * Alerting (available via Pixie’s New Relic One integration or [Slackbot](/tutorials/integrations/slackbot-alert) API integration)
 * Advanced dashboarding (available with Pixie's [Grafana plugin](/reference/plugins/grafana))
-* Long-term storage (available with [New Relic One integration](https://newrelic.com/platform/kubernetes-pixie))
+* Long-term storage (available with the [Pixie Plugin system](/reference/plugins/plugin-system) and [New Relic One integration](https://newrelic.com/platform/kubernetes-pixie))
diff --git a/content/en/01-about-pixie/05-faq.md b/content/en/01-about-pixie/05-faq.md
@@ -74,7 +74,7 @@ Pixie has a [1GiB memory requirement](/installing-pixie/requirements/#memory) pe
 
 Retention time depends on the level of traffic in your cluster, but will generally be on the order of hours. Each data table in Pixie (e.g. `http_events`) has its own maximum size. These tables collect data until the maximum size is reached, and then begin dropping the oldest data.
 
-We recommend integrating with third-party observability tools (such as [New Relic’s integration](https://newrelic.com/platform/kubernetes-pixie)) to provide long-term retention.
+Pixie's [Plugin System](/reference/plugins/plugin-system) integrates with other tools to for long-term retention. Other third-party observability tools (such as [New Relic’s integration](https://newrelic.com/platform/kubernetes-pixie)) may also provide this capability.
 
 ### How does Pixie secure its data?
 
@@ -146,7 +146,7 @@ Pixie does not yet offer full RBAC support. However, you can enable “Approvals
 
 Pixie does not offer alerting. However, alerts can be set up using Pixie’s API. For example, see the [Slackbot Alert Tutorial](/tutorials/integrations/slackbot-alert/).
 
-For comprehensive alerting, we recommend integrating with third-party observability tools (such as New Relic’s integration).
+For comprehensive alerting, we recommend integrating with third-party observability tools (such as New Relic’s integration). In the future, Pixie's [Plugin System](/reference/plugins/plugin-system) will allow easy integration with other tools to setup alerts using Pixie data.
 
 ### How do I delete a cluster?
 
diff --git a/content/en/05-reference/03-plugins/01-plugin-system.md b/content/en/05-reference/03-plugins/01-plugin-system.md
@@ -0,0 +1,51 @@
+---
+title: "Pixie Plugin System"
+metaTitle: "Reference | Plugins | Pixie Plugin System "
+metaDescription: "Pixie's Plugin System"
+order: 1 
+---
+
+Pixie's primary focus is to build a real-time debugging platform, not a full-fledged observability solution. Pixie provides a plugin system to integrate with external tools to enable capabilities like longterm data retention and alerting.
+
+- [*Longterm Data Retention*](#longterm-data-retention): Pixie only retains up to 24 hours of data. Leverage an external datastore for longterm data retention by sending Pixie data in the [OpenTelemetry](https://opentelemetry.io/) format. Future support will be added for querying longterm data from within the Pixie UI, allowing you to use PxL and Pixie's versatile live views.
+- *Alerts* (Coming Soon!): Power alerts using Pixie's rich dataset, all configurable from within Pixie's UI.
+
+## Enabling a Plugin
+
+Plugins can be configured from within the Admin UI. 
+
+1. Open the Live UI and navigate to the Plugins tab on the Admin page (`/admin/plugins`).
+2. Find the plugin you wish to enable and click the toggle to enable/disable the plugin.
+3. Expand the plugin row to specify the necessary configuration values. The required set of fields differs per plugin provider, and usually contains information such as API keys which are used for authentication.
+
+## Longterm Data Retention
+
+Enable a plugin which offers longterm data retention capabilities to send Pixie data to an external datastore. This plugin allows you to configure PxL scripts to export data at regularly scheduled intervals. This only currently supports exporting data through the [OpenTelemetry](https://opentelemetry.io/) format. 
+
+By default, the plugin provider has configured a set of preset scripts. These will automatically start running and exporting data from all clusters in the org as soon as the plugin is enabled. Users can also choose to export custom data by creating a custom export script.
+
+### Configuring Preset Export Scripts
+
+1. [Enable a plugin provider](#enabling-a-plugin) which supports longterm data retention. 
+2. Open the Live UI and navigate to the Data Export page in the sidebar (`/configure-data-export`).
+3. All preset scripts are listed under `<Plugin Provider> Scripts`.
+4. Click the toggle to enable/disable the export from the preset script.
+5. Click the gear icon to configure the preset script. Name, description, and PxL is uneditable. However, users may choose which clusters to export from and how frequently the script is run.
+6. Click "Save" to update the script with your changes.
+
+### Creating Custom Export Scripts
+
+1. [Enable a plugin provider](#enabling-a-plugin) which supports longterm data retention. 
+2. Open the Live UI and navigate to the Data Export page in the sidebar (`/configure-data-export`).
+3. Under `Custom Scripts`, click the `Create Scripts` button on the right-hand side.
+4. Enter a script name and description.
+5. Select which clusters to export from. If none is specified, data will be exported from all clusters in the org.
+6. Enter the PxL script. This script should contain a `px.export` call for exporting data to the OpenTelemetry format. See our [OpenTelemetry export tutorial](/tutorials/integrations/otel/) and [OpenTelemetry export reference docs](/reference/pxl/otel-export/) for more information. No `endpoint` needs to be specified to the `px.otel.Data` object, as this will be filled in automatically based on the selected plugin provider.
+7. Select how often this script should run.
+8. Select the plugin provider to export this data to.
+9. Click "Create" to save your settings. Data export should start immediately.
+
+## Contributing a Plugin
+
+Interested in contributing a plugin to our Plugin system? Check out our [Pixie Plugin Repository](https://github.com/pixie-io/pixie-plugin) and follow the provided instructions.
+
diff --git a/content/en/05-reference/03-plugins/02-grafana.md b/content/en/05-reference/03-plugins/02-grafana.md
@@ -2,7 +2,7 @@
 title: "Grafana Datasource Plugin"
 metaTitle: "Reference | Plugins | Grafana Datasource Plugin "
 metaDescription: "Pixie's Grafana Datasource Plugin"
-order: 1
+order: 2 
 ---
 
 The Pixie Datasource plugin allows you to visualize data from the Pixie observability platform in Grafana.
diff --git a/content/en/05-reference/03-plugins/index.md b/content/en/05-reference/03-plugins/index.md
@@ -5,6 +5,10 @@ metaDescription: "Reference docs for Pixie plugins."
 order: 3
 ---
 
-Plugins available for Pixie:
+Plugins available in Pixie:
+
+- [Pixie Plugin System](/reference/plugins/plugin-system)
+
+Plugins available for Pixie in other tools:
 
 - [Grafana Datasource Plugin](/reference/plugins/grafana)
diff --git a/content/en/05-reference/index.md b/content/en/05-reference/index.md
@@ -32,6 +32,7 @@ order: 50
 
 #### Plugins
 
+- [Pixie Plugin System](/reference/plugins/plugin-system)
 - [Grafana Datasource Plugin](/reference/plugins/grafana)
 
 #### PxL Language
