Fix demo helper scripts that print null, fail on first run or watch the wrong entity

demos/moveit_pick_place/README.md

@@ -73,7 +73,27 @@ Use the interactive arm controller to send joint trajectories:
 ```
 
 The script sends goals directly to the `panda_arm_controller/follow_joint_trajectory` action.
-It works both from outside (via `docker exec`) and from inside the container.
+It works both from outside (via `docker exec`, no TTY required) and from inside the container.
+Where there is no `docker` CLI (inside the container), it uses the local `ros2`. Where there
+is one, it uses a local `ros2` only if `ros2 action list` shows the arm action; it asks twice,
+because a first listing without a running `ros2` daemon can miss it. Otherwise it runs the
+goal through `docker exec` in the demo container.
+
+`pick_place_loop.py` keeps sending its own goals to the same controller, so a manual move
+can be preempted mid-motion by the demo's own workload. `move-arm.sh` reports the goal's
+real final status: if the controller aborts it (`error_string: Current goal preempted by
+new incoming action`), the script prints `Failed: <pose> (status: ABORTED)` and exits
+non-zero instead of claiming success. `./move-arm.sh demo` runs all three steps regardless
+of earlier failures and reports each one; the command exits non-zero if any step failed.
+
+Each `ros2 action send_goal` run is limited to 30 seconds. The controller can fail to
+deliver the goal response to a freshly started CLI (the container log shows `Failed to send
+goal response`); it then never runs that goal. When the CLI printed `Sending goal:` and no
+goal response arrived, the script sends the goal again, up to three times. A goal that was
+accepted is never sent twice: if its result does not arrive in time, the script prints
+`Failed: <pose> (status: UNKNOWN)`. A goal that was never sent (no demo container, no action
+server within 30 seconds, a `ros2` error) is not sent again: the script prints the command's
+output and `Failed: <pose> (goal not sent: ...)`, and exits non-zero.
 
 ### 4. Viewing Logs
 
@@ -209,6 +229,10 @@ curl http://localhost:8080/api/v1/apps/move-group/operations | jq
 curl http://localhost:8080/api/v1/faults | jq
 ```
 
+While the fault manager is not available (for example right after startup), `GET /faults`
+answers `503`. `./check-faults.sh` then prints `Could not read faults (HTTP 503)` and exits
+non-zero; it reports "No active faults" only after a successful read of an empty list.
+
 ### Clear All Faults
 
 ```bash
@@ -308,8 +332,8 @@ The gateway supports condition-based triggers that fire when specific events occ
 
 ### How It Works
 
-1. `setup-triggers.sh` creates a trigger via `POST /api/v1/apps/manipulation_monitor/triggers`:
-   - **Resource:** `/api/v1/apps/manipulation_monitor/faults` (watches fault collection)
+1. `setup-triggers.sh` creates a trigger via `POST /api/v1/apps/manipulation-monitor/triggers`:
+   - **Resource:** `/api/v1/apps/manipulation-monitor/faults` (watches fault collection)
    - **Condition:** `OnChange` (fires on any new or updated fault)
    - **Multishot:** `true` (fires repeatedly, not just once)
    - **Lifetime:** 3600 seconds (auto-expires after 1 hour)
@@ -320,23 +344,23 @@ The gateway supports condition-based triggers that fire when specific events occ
 
 ```bash
 # Create a trigger
-curl -X POST http://localhost:8080/api/v1/apps/manipulation_monitor/triggers \
+curl -X POST http://localhost:8080/api/v1/apps/manipulation-monitor/triggers \
   -H "Content-Type: application/json" \
   -d '{
-    "resource": "/api/v1/apps/manipulation_monitor/faults",
+    "resource": "/api/v1/apps/manipulation-monitor/faults",
     "trigger_condition": {"condition_type": "OnChange"},
     "multishot": true,
     "lifetime": 3600
   }' | jq
 
 # List triggers
-curl http://localhost:8080/api/v1/apps/manipulation_monitor/triggers | jq
+curl http://localhost:8080/api/v1/apps/manipulation-monitor/triggers | jq
 
 # Watch events (replace TRIGGER_ID)
-curl -N http://localhost:8080/api/v1/apps/manipulation_monitor/triggers/TRIGGER_ID/events
+curl -N http://localhost:8080/api/v1/apps/manipulation-monitor/triggers/TRIGGER_ID/events
 
 # Delete a trigger
-curl -X DELETE http://localhost:8080/api/v1/apps/manipulation_monitor/triggers/TRIGGER_ID
+curl -X DELETE http://localhost:8080/api/v1/apps/manipulation-monitor/triggers/TRIGGER_ID
 ```
 
 ## Fault Injection Scenarios
@@ -445,7 +469,7 @@ Container scripts are stored under `/var/lib/ros2_medkit/scripts/moveit-planning
 | Docker build fails | Apt package missing | Check if MoveIt 2 Jazzy packages are available |
 | "MoveGroup not available" | Slow startup | Wait 60-90 seconds after container starts |
 | Controller not loading | Missing config | Verify `moveit_controllers.yaml` is correct |
-| Joint states empty | Controllers not loaded | Check `ros2 control list_controllers` inside container |
+| Joint states empty (`check-entities.sh` prints "Joint state data not available") | Controllers not loaded | Check `ros2 control list_controllers` inside container |
 | `ros2` CLI hangs in `docker exec` | DDS discovery across container boundaries | Use gateway REST API instead of `ros2` CLI for parameter/service operations |
 
 ## Comparison with Other Demos

demos/moveit_pick_place/check-entities.sh

@@ -40,24 +40,31 @@ echo_step "1. Areas (Functional Groupings)"
 curl -s "${API_BASE}/areas" | jq '.items[] | {id: .id, name: .name, description: .description}'
 
 echo_step "2. Components (Hardware/Logical Units)"
-curl -s "${API_BASE}/components" | jq '.items[] | {id: .id, name: .name, type: .type, area: .area}'
+curl -s "${API_BASE}/components" | jq '.items[] | {id: .id, name: .name, description: .description}'
 
 echo_step "3. Apps (ROS 2 Nodes)"
-curl -s "${API_BASE}/apps" | jq '.items[] | {id: .id, name: .name, category: .category, component: .is_located_on}'
+curl -s "${API_BASE}/apps" | jq '.items[] | {id: .id, name: .name, description: .description, component: .["x-medkit"].component_id}'
 
 echo_step "4. Functions (High-level Capabilities)"
-curl -s "${API_BASE}/functions" | jq '.items[] | {id: .id, name: .name, category: .category, hosted_by: .hosted_by}'
+curl -s "${API_BASE}/functions" | jq '.items[] | {id: .id, name: .name, description: .description}'
 
 echo_step "5. Sample Data (Joint States)"
 echo "Getting latest joint states from Panda arm..."
-curl -s "${API_BASE}/apps/joint-state-broadcaster/data/joint_states" 2>/dev/null | jq '{
-    joint_names: .data.name,
-    positions: .data.position,
-    velocities: .data.velocity
-}' || echo "   (Joint state data not available — robot may still be starting)"
+# An error body or a reply without data (no /joint_states publisher) has no
+# joint names, and jq still exits 0 on it.
+JOINT_STATES=$(curl -s "${API_BASE}/apps/joint-state-broadcaster/data/joint_states" 2>/dev/null) || JOINT_STATES=""
+if jq -e '.data.name | arrays | length > 0' <<< "${JOINT_STATES}" > /dev/null 2>&1; then
+    jq '{
+        joint_names: .data.name,
+        positions: .data.position,
+        velocities: .data.velocity
+    }' <<< "${JOINT_STATES}"
+else
+    echo "   (Joint state data not available - the robot may still be starting, or joint_state_broadcaster is not running)"
+fi
 
 echo_step "6. Faults"
-curl -s "${API_BASE}/faults" | jq '.items[] | {code: .code, severity: .severity, reporter: .reporter_id}'
+curl -s "${API_BASE}/faults" | jq '.items[] | {code: .fault_code, severity: .severity_label, status: .status, sources: .reporting_sources}'
 
 echo ""
 echo -e "${GREEN}✓ Entity hierarchy exploration complete!${NC}"

demos/moveit_pick_place/check-faults.sh

@@ -25,12 +25,19 @@ fi
 echo "✓ Gateway is healthy"
 echo ""
 
-# Get all faults
+# Get all faults. A failed read is not "no faults": the gateway answers 503
+# while the fault manager is unavailable (for example during startup).
 echo "📋 Active Faults:"
-FAULTS=$(curl -s "${API_BASE}/faults")
+RESPONSE=$(curl -s -m 30 -w '\n%{http_code}' "${API_BASE}/faults") || true
+HTTP_CODE="${RESPONSE##*$'\n'}"
+FAULTS="${RESPONSE%$'\n'*}"
 
-# Check if there are any faults
-FAULT_COUNT=$(echo "$FAULTS" | jq '.items | length')
+if [ "$HTTP_CODE" != "200" ] || ! FAULT_COUNT=$(echo "$FAULTS" | jq -e '.items | arrays | length' 2>/dev/null); then
+    DETAIL=$(echo "$FAULTS" | jq -r '[.message, .parameters.details] | map(select(. != null)) | join(": ")' 2>/dev/null)
+    echo "❌ Could not read faults (HTTP ${HTTP_CODE:-000})${DETAIL:+: ${DETAIL}}"
+    echo "   The fault list is unknown, not empty. Retry in a few seconds."
+    exit 1
+fi
 
 if [ "$FAULT_COUNT" = "0" ]; then
     echo "   No active faults — system is healthy! ✅"

demos/moveit_pick_place/move-arm.sh

@@ -12,13 +12,19 @@
 
 set -eu
 
-CONTAINER="${CONTAINER_NAME:-$(docker ps --format '{{.Names}}' | grep -E '^moveit_medkit_demo(_nvidia)?(_local)?$' | head -n1)}"
 ACTION="/panda_arm_controller/follow_joint_trajectory"
 JOINT_NAMES='["panda_joint1","panda_joint2","panda_joint3","panda_joint4","panda_joint5","panda_joint6","panda_joint7"]'
 
 # Duration in seconds for trajectory execution
 DURATION_SEC=3
 
+# Time limit for one `ros2 action send_goal` run, and how many runs per goal.
+# The controller can fail to deliver the goal response to a new CLI
+# ("Failed to send goal response"). It then never runs the goal and the CLI
+# waits forever, so a goal with no response is sent again.
+SEND_TIMEOUT_SEC=30
+SEND_ATTEMPTS=3
+
 # --- Preset joint positions (radians) ---
 # Ready: default MoveIt pose (from SRDF)
 READY="[0.0, -0.785, 0.0, -2.356, 0.0, 1.571, 0.785]"
@@ -45,6 +51,51 @@ RIGHT="[1.5, -0.785, 0.0, -2.356, 0.0, 1.571, 0.785]"
 WAVE="[0.0, -1.0, 0.0, -0.5, 0.0, 2.5, 0.785]"
 
 
+have() {
+    command -v "$1" &> /dev/null
+}
+
+# True only if a LOCAL ros2 can actually reach the target action server.
+# `ros2 node list` exits 0 even on an empty graph (wrong ROS_DOMAIN_ID, no
+# multicast route), so a host with ROS 2 sourced but not connected to the
+# demo looks identical to being inside the container. Checking that the
+# action itself is listed avoids that false positive. A cold listing (no
+# ros2 daemon yet) can miss a running server, so a miss is listed once more.
+can_reach_action_locally() {
+    have ros2 || return 1
+    ros2 action list 2> /dev/null | grep -qFx "${ACTION}" \
+        || ros2 action list 2> /dev/null | grep -qFx "${ACTION}"
+}
+
+# Picks how goals are sent: USE_LOCAL=true for the local ros2, else
+# `docker exec` into CONTAINER. Without a docker CLI the script runs inside
+# the container or on a ROS host, so the local ros2 is the only way.
+# On failure sets NOT_SENT_REASON and returns 1.
+USE_LOCAL=""
+CONTAINER=""
+NOT_SENT_REASON=""
+choose_transport() {
+    [[ -z "${USE_LOCAL}" ]] || return 0
+    if ! have docker; then
+        if ! have ros2; then
+            NOT_SENT_REASON="needs the docker CLI or a sourced ROS 2 (ros2 CLI), found neither"
+            return 1
+        fi
+        USE_LOCAL=true
+        return 0
+    fi
+    if can_reach_action_locally; then
+        USE_LOCAL=true
+        return 0
+    fi
+    CONTAINER="${CONTAINER_NAME:-$(docker ps --format '{{.Names}}' | grep -E '^moveit_medkit_demo(_nvidia)?(_local)?$' | head -n1)}"
+    if [[ -z "${CONTAINER}" ]]; then
+        NOT_SENT_REASON="no running moveit_medkit_demo container, start it with ./run-demo.sh or set CONTAINER_NAME"
+        return 1
+    fi
+    USE_LOCAL=false
+}
+
 send_trajectory() {
     local positions="$1"
     local label="$2"
@@ -64,27 +115,76 @@ send_trajectory() {
         }
     }"
 
-    # Check if we're inside the container or outside
-    if command -v ros2 &> /dev/null && ros2 node list &> /dev/null 2>&1; then
-        # Inside the container (or ROS 2 env is set up)
-        ros2 action send_goal "${ACTION}" \
-            control_msgs/action/FollowJointTrajectory \
-            "${goal_msg}" \
-            --feedback
-    else
-        # Outside — exec into container
-        docker exec -it "${CONTAINER}" bash -c "
-            source /opt/ros/jazzy/setup.bash && \
-            source /root/demo_ws/install/setup.bash && \
-            ros2 action send_goal ${ACTION} \
-                control_msgs/action/FollowJointTrajectory \
-                \"${goal_msg}\" \
-                --feedback
-        "
+    if ! choose_transport; then
+        echo "Failed: ${label} (goal not sent: ${NOT_SENT_REASON})" >&2
+        return 1
     fi
 
+    # `ros2 action send_goal` always exits 0, whatever the goal's outcome -
+    # the real result is in its own printed "Goal finished with status:"
+    # line, so capture output and parse that instead of the exit code.
+    # PYTHONUNBUFFERED keeps the lines printed before a timeout kills the CLI.
+    local attempt output rc
+    for ((attempt = 1; attempt <= SEND_ATTEMPTS; attempt++)); do
+        rc=0
+        if [[ "${USE_LOCAL}" == true ]]; then
+            output=$(PYTHONUNBUFFERED=1 timeout "${SEND_TIMEOUT_SEC}" \
+                ros2 action send_goal "${ACTION}" \
+                control_msgs/action/FollowJointTrajectory \
+                "${goal_msg}" \
+                --feedback 2>&1) || rc=$?
+        else
+            # Outside the container: exec into it. No -it: this must also
+            # work without a TTY (CI, a pipe), and the command needs no stdin.
+            # timeout runs in the container: killing `docker exec` would
+            # leave the CLI running there.
+            output=$(docker exec "${CONTAINER}" bash -c "
+                source /opt/ros/jazzy/setup.bash && \
+                source /root/demo_ws/install/setup.bash && \
+                PYTHONUNBUFFERED=1 timeout ${SEND_TIMEOUT_SEC} \
+                ros2 action send_goal ${ACTION} \
+                    control_msgs/action/FollowJointTrajectory \
+                    \"${goal_msg}\" \
+                    --feedback
+            " 2>&1) || rc=$?
+        fi
+        printf '%s\n' "${output}"
+        # An accepted or rejected goal has its answer.
+        if grep -qE '^(Goal accepted with ID|Goal was rejected)' <<< "${output}"; then
+            break
+        fi
+        # Never sent (no container, no action server, a CLI error): sending
+        # again changes nothing. rc 124 is the timeout.
+        if ! grep -q '^Sending goal:' <<< "${output}"; then
+            if ((rc == 124)) && grep -q '^Waiting for an action server' <<< "${output}"; then
+                echo "Failed: ${label} (goal not sent: no action server ${ACTION} within ${SEND_TIMEOUT_SEC} s)" >&2
+            elif ((rc == 124)); then
+                echo "Failed: ${label} (goal not sent: timed out after ${SEND_TIMEOUT_SEC} s)" >&2
+            else
+                echo "Failed: ${label} (goal not sent: exit status ${rc})" >&2
+            fi
+            return 1
+        fi
+        # Sent with no response: the controller never runs it, so send again.
+        if ((attempt < SEND_ATTEMPTS)); then
+            if ((rc == 124)); then
+                echo "No goal response within ${SEND_TIMEOUT_SEC} s, sending the goal again"
+            else
+                echo "No goal response (exit status ${rc}), sending the goal again"
+            fi
+        fi
+    done
+
+    local status
+    status=$(printf '%s\n' "${output}" | grep -F 'Goal finished with status:' | tail -n1 | sed -E 's/.*status: *//')
+
     echo ""
-    echo "✅ Done: ${label}"
+    if [[ "${status}" == "SUCCEEDED" ]]; then
+        echo "✅ Done: ${label}"
+        return 0
+    fi
+    echo "Failed: ${label} (status: ${status:-UNKNOWN})" >&2
+    return 1
 }
 
 show_menu() {
@@ -110,13 +210,19 @@ show_menu() {
 run_demo_cycle() {
     echo "🔄 Running pick → place → home cycle..."
     echo ""
-    send_trajectory "${PICK}" "pick"
+    local failed=0
+    send_trajectory "${PICK}" "pick" || failed=1
     sleep 2
-    send_trajectory "${PLACE}" "place"
+    send_trajectory "${PLACE}" "place" || failed=1
     sleep 2
-    send_trajectory "${READY}" "ready (home)"
+    send_trajectory "${READY}" "ready (home)" || failed=1
     echo ""
-    echo "🔄 Cycle complete!"
+    if [[ "${failed}" -eq 0 ]]; then
+        echo "🔄 Cycle complete!"
+    else
+        echo "🔄 Cycle complete with failures" >&2
+    fi
+    return "${failed}"
 }
 
 handle_choice() {
@@ -138,15 +244,20 @@ handle_choice() {
 
 # --- Main ---
 
-# If argument provided, run directly
+# If argument provided, run directly. Reflect the goal's real result in the
+# exit code instead of always exiting 0.
 if [[ $# -gt 0 ]]; then
-    handle_choice "$1"
-    exit 0
+    if handle_choice "$1"; then
+        exit 0
+    else
+        exit 1
+    fi
 fi
 
-# Interactive mode
+# Interactive mode. A failed goal reports failure and the menu continues -
+# it must not kill the session (set -e would, without this guard).
 while true; do
     show_menu
     read -rp "Choose position (1-8, d, q): " choice
-    handle_choice "${choice}"
+    handle_choice "${choice}" || true
 done

demos/multi_ecu_aggregation/Dockerfile

@@ -49,10 +49,6 @@ COPY src/ ${COLCON_WS}/src/multi_ecu_demo/src/
 COPY config/ ${COLCON_WS}/src/multi_ecu_demo/config/
 COPY launch/ ${COLCON_WS}/src/multi_ecu_demo/launch/
 
-# Copy container scripts
-COPY container_scripts/ /var/lib/ros2_medkit/scripts/
-RUN find /var/lib/ros2_medkit/scripts -name "*.bash" -exec chmod +x {} \;
-
 # Build all packages (skip test dependencies that aren't in ros-base)
 WORKDIR ${COLCON_WS}
 RUN bash -c "source /opt/ros/jazzy/setup.bash && \
@@ -61,6 +57,11 @@ RUN bash -c "source /opt/ros/jazzy/setup.bash && \
       --skip-keys='ament_cmake_clang_format ament_cmake_clang_tidy test_msgs example_interfaces sqlite3' && \
     MAKEFLAGS='-j 2' colcon build --executor sequential --symlink-install --cmake-args -DBUILD_TESTING=OFF"
 
+# Copy container scripts. Not a build input: kept after colcon build so a
+# script-only change does not invalidate the compile cache.
+COPY container_scripts/ /var/lib/ros2_medkit/scripts/
+RUN find /var/lib/ros2_medkit/scripts -name "*.bash" -exec chmod +x {} \;
+
 # Setup environment
 RUN echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc && \
     echo "source ${COLCON_WS}/install/setup.bash" >> ~/.bashrc