DWO Stress Testing #1353

Activity

1. dkwon17

   added this to Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Jul 3, 2025
2. ibuziuk

   moved this to Analyzing in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Jul 24, 2025
3. ibuziuk

   assigned

   rohanKanojia

   on Jul 24, 2025
4. rohanKanojia

   added this to Eclipse Che Team B Backlogon Jul 28, 2025
5. rohanKanojia

   moved this to Unplanned in Eclipse Che Team B Backlogon Jul 28, 2025
6. dkwon17

   moved this from Unplanned to 🚧 In Progress in Eclipse Che Team B Backlogon Jul 30, 2025
7. ibuziuk

   moved this from Analyzing to Todo in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Aug 14, 2025
8. ibuziuk

   moved this from Todo to In Progress in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Aug 14, 2025
9. rohanKanojia commented on Sep 3, 2025

   

   rohanKanojia

   on Sep 3, 2025

   Last edited by dkwon17

   Member

   More actions

   Load Test Implementation Progress

   Here's a quick summary of the progress on this issue so far:

   • DevTools Week Investigation: Investigated tools for load testing.

     • ChaosMonkey (Dev Tools Week Q2 2025 eclipse-che/che#23461 (comment))
     • k6 (https://grafana.com/docs/k6/latest/get-started/running-k6/)

   • Implement k6 Load Test PoC: Completed a proof-of-concept for k6 load tests (https://github.com/rohankanojia-testing/dwo-load-testing-using-k6-poc) with the following features:

     • Configurable number of Virtual Users (VUs)
     • Ability to run load tests from outside the cluster using the k6 binary
     • Ability to run load tests from within the cluster using the k6 operator
     • Observation of DevWorkspace Operator Pod CPU/memory usage during load tests
     • Creation of an automount ConfigMap/Secret for all DevWorkspaces created during tests

   • Merge with Existing Scripts: Integrated the PoC with the existing Eclipse Che load test script (https://github.com/eclipse-che/che/blob/main/tests/performance/load-tests/load-test.sh).

     • Added configuration to create DevWorkspaces in separate namespaces
     • Added configuration to specify the DevWorkspace manifest URL to be used during load tests
     • Added logging of events, DevWorkspace watches, and a list of failed DevWorkspaces to the logs/ directory

   • Integrate PoC into DevWorkspace Operator Code:

     • Created PR ci : test k6-operator install and load test execution in e2e workflow #1477
     • Exposed load tests via a make target (make test_load ...)
     • Run load tests in OpenShift CI after E2E tests
       • Added a step in .ci/openshift_e2e.sh to run load tests after E2E tests
       • The results from the CI OpenShift instance were inconclusive due to limited resources. Further investigation is needed to determine if we should run the tests with reduced load or on a different cluster.

   • Concurrency Tracking:

     • Noticed during a script-based investigation that the number of concurrently running DevWorkspaces was not matching the maximum number of VUs. Most DevWorkspaces were stuck in the Starting phase and were deleted as soon as they became Ready.
       • Added a configuration option to set a threshold for active workspaces, after which new DevWorkspaces will not be created.
       • Added an option to check if DevWorkspaces should be deleted at the end of each VU iteration.
     • With these new configuration options, we can now more accurately predict the number of DevWorkspaces created concurrently.

   • Pre-release Testing: Used these load tests for 3.23.0 pre-release testing on OpenShift instance provided by QE and documented the results.

   • Investigate Target Cluster: Investigate a target cluster to run these load tests before the release.

     • Cluster Bot (CPU exhaustion, preventing us from getting reliable performance results)
     • RH Demo Catalog (provides good cluster but has cost involved)
     • RH Resource Hub (CPU exhaustion, preventing us from getting reliable performance results)
     • Scale Lab (Under investigation)

   👍React with 👍1Reacted by David Kwon
10. ibuziuk

    moved this from In Progress to Blocked in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Sep 4, 2025
11. ibuziuk

    moved this from Blocked to In Progress in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Sep 25, 2025
12. dkwon17

    closed this as completedon Dec 3, 2025
13. github-project-automation

    moved this from In Progress to Done in Red Hat OpenShift Dev Spaces and Web Terminal Priorities [DEPRECATED]on Dec 3, 2025
14. github-project-automation

    moved this from 🚧 In Progress to ✅ Done in Eclipse Che Team B Backlogon Dec 3, 2025