libsystemd: graceful service stop (stop handlers + stop_timeout_ms)

README.md

@@ -103,6 +103,7 @@ Recognized keys:
 | `description` | unit name  | Implemented | Free-form text, surfaced via `libsystemd_list()`'s `libsystemd_service_info_t.description` and printed by `service list`/`service status` - not otherwise interpreted |
 | `type`        | `simple`   | Implemented | `simple` (default, expected to keep running), `oneshot` (expected to run to completion - a clean exit is logged as informational, not a warning), or `library` (`exec` names a **Library** module instead of an Application - start loads+enables it, stop disables+unloads it; no process is spawned) - see [app/libsystemd/docs/configuration.md](app/libsystemd/docs/configuration.md#service-type) |
 | `restart`     | `no`       | Implemented | `no` (default), `always`, or `on-failure` - restarts the unit when its process exits **on its own** (not on `service stop`), via `dmosi`'s process exit-callback API. See [app/libsystemd/docs/configuration.md](app/libsystemd/docs/configuration.md#restart-supervision) |
+| `stop_timeout_ms` | `3000` | Implemented | How long a stop waits for a service that registered a stop handler (`libsystemd_set_stop_handler()`) to leave `main()` on its own before killing it; `0` kills at once. Services without a handler are killed at once, as before. See [app/libsystemd/docs/configuration.md](app/libsystemd/docs/configuration.md#graceful-stop) |
 
 An unset stream key leaves that stream at whatever default the spawned module
 would otherwise get; `stdout`/`stderr`/`stdlog` may point at the same path (as

app/libsystemd/CMakeLists.txt

@@ -31,6 +31,7 @@ set(DMOD_MAL_IMPLS          )
 dmod_add_library(${DMOD_MODULE_NAME} ${DMOD_MODULE_VERSION}
     # List of source files - can include C and C++ files
     src/serviceapi.c
+    src/stophandler.c
 )
 
 target_include_directories(${DMOD_MODULE_NAME}_if

app/libsystemd/docs/api-reference.md

@@ -18,6 +18,7 @@ a quick-reference summary, not a replacement for those.
 | `libsystemd_service_type_t` | enum | `LIBSYSTEMD_SERVICE_TYPE_SIMPLE` (default) / `LIBSYSTEMD_SERVICE_TYPE_ONESHOT` / `LIBSYSTEMD_SERVICE_TYPE_LIBRARY` - from the unit's `type` key, see [configuration.md](configuration.md#keys). |
 | `libsystemd_restart_policy_t` | enum | `LIBSYSTEMD_RESTART_NO` (default) / `LIBSYSTEMD_RESTART_ALWAYS` / `LIBSYSTEMD_RESTART_ON_FAILURE` - from the unit's `restart` key, see [configuration.md](configuration.md#keys). |
 | `libsystemd_service_info_t` | struct | `{ const char* unit_name; const char* description; libsystemd_service_type_t type; libsystemd_restart_policy_t restart_policy; libsystemd_service_status_t status; }` - passed to `libsystemd_list()`'s visitor. |
+| `libsystemd_stop_handler_t` | function pointer | `void (*)(void* arg)` - stop request handler registered with `libsystemd_set_stop_handler()`. Runs in the stopper's context; must only make the service leave and return at once. |
 | `libsystemd_visitor_t` | function pointer | `bool (*)(const libsystemd_service_info_t* info, void* user_ptr)` - return `false` to stop `libsystemd_list()` early. |
 
 ## Functions
@@ -114,10 +115,14 @@ For a `type=library` unit:
 
 ### `libsystemd_stop_service(const char* unit_name)`
 
-Looks `unit_name` up and stops it. For a `simple`/`oneshot` unit, kills its
-tracked process (`dmosi_process_kill`); if a restart-supervision callback is
-registered for the unit, unregisters it first, so a deliberate stop is never
-mistaken for a crash that needs restarting. For a `type=library` unit, instead
+Looks `unit_name` up and stops it. For a `simple`/`oneshot` unit, stops its
+tracked process: if a restart-supervision callback is registered for the
+unit, unregisters it first, so a deliberate stop is never mistaken for a
+crash that needs restarting. If the process registered a stop handler
+(`libsystemd_set_stop_handler()`), it is called and the process gets up to
+the unit's `stop_timeout_ms` to exit on its own - the call blocks meanwhile.
+Otherwise, or if it does not exit in time, it is killed
+(`dmosi_process_kill`). See [Graceful stop](configuration.md#graceful-stop). For a `type=library` unit, instead
 disables then unloads `exec` (`Dmod_DisableModule` + `Dmod_UnloadModule`) -
 disable before unload, since the dmod core refuses to unload a module that is
 still enabled.
@@ -154,6 +159,21 @@ exits right after adopting would kill what it just handed the unit to. See
 - `-ESRCH` - `pid` does not resolve to a live process.
 - `-ENOSYS` - process lookup is not connected on this build/platform.
 
+### `libsystemd_set_stop_handler(libsystemd_stop_handler_t handler, void* arg)`
+
+Registers `handler` as the calling process's stop request handler, replacing
+a previous one; `NULL` clears it. `libsystemd_stop_service()` calls it (in the
+stopper's context) before waiting up to `stop_timeout_ms` for the process to
+exit on its own, and kills the process only if it does not. The handler must
+only make the service leave - set a flag, post a semaphore - and return at
+once; it must not block or call into libsystemd. `arg` must stay valid until
+the handler is cleared or the process has terminated; clearing waits for an
+invocation still in progress. The registration is dropped automatically when
+the process terminates. See [Graceful stop](configuration.md#graceful-stop).
+
+- `-ESRCH` - the caller is not a dmosi process.
+- `-ENOMEM` - allocation failed, or the module is not initialized.
+
 ### `libsystemd_status(const char* unit_name, libsystemd_service_status_t* out_status)`
 
 Fills `*out_status` with the unit's current process state and PID. Reports

app/libsystemd/docs/configuration.md

@@ -36,6 +36,7 @@ else (non-`.ini` files) is silently skipped.
 | `description` | no   | Free-form text, surfaced (not parsed/interpreted) via `libsystemd_service_info_t.description` in `libsystemd_list()`, and printed by `service list`/`service status`. |
 | `type`     | no       | `simple` (default), `oneshot` or `library` - see [Service type](#service-type) below. |
 | `restart`  | no       | `no` (default), `always` or `on-failure` - see [Restart supervision](#restart-supervision) below. |
+| `stop_timeout_ms` | no | How long a stop waits for a service that registered a stop handler to exit on its own before killing it. Default `3000`; `0` kills at once. See [Graceful stop](#graceful-stop) below. |
 
 A unit file with no `exec` key fails to parse (`libsystemd_parse_file()`
 returns `-EINVAL`); `libsystemd_parse_dir()` logs that failure and skips the
@@ -145,6 +146,65 @@ Like the rest of this module, no internal locking is done - an application
 enabling `restart` from a multi-threaded environment is responsible for
 serializing its own calls into this module's API.
 
+### Graceful stop
+
+By default `libsystemd_stop_service()` (`service stop`, a device removal
+rule, a rescan, module unload) ends a unit's process with
+`dmosi_process_kill()`: its threads are deleted wherever they happen to be.
+Whatever the service held at that moment stays held - a driver's I/O lock
+taken inside an ioctl, a half-written filesystem structure, a mutex shared
+with another module.
+
+A service avoids that by registering a stop handler for its own process:
+
+```c
+static void on_stop(void* arg)            /* runs in the stopper's context */
+{
+    my_service_t* s = arg;
+    s->stop = true;                       /* only make main() leave ...        */
+    dmosi_semaphore_post(s->wakeup, 1);   /* ... and wake it if it is waiting */
+}
+
+int main(int argc, char* argv[])
+{
+    ...
+    libsystemd_set_stop_handler(on_stop, &s);
+    while (!s.stop)
+    {
+        dmosi_semaphore_wait(s.wakeup, 1, -1);
+        do_work(&s);                      /* never interrupted halfway */
+    }
+    libsystemd_set_stop_handler(NULL, NULL);  /* before freeing &s */
+    cleanup(&s);
+    return 0;
+}
+```
+
+A stop then runs:
+
+1. The restart-supervision callback is unregistered (a stop never triggers a
+   restart).
+2. The process's stop handler is called, in the caller's context. It must
+   return at once - no blocking, no calls into libsystemd.
+3. libsystemd waits up to `stop_timeout_ms` for the process to terminate on
+   its own (return from `main()`, so its cleanup runs).
+4. If it has not, it is killed with `dmosi_process_kill()` as before, and a
+   warning is logged.
+
+A process without a handler, a unit with `stop_timeout_ms=0`, and a unit
+stopping itself (the caller is the unit's own process) skip steps 2-3 and
+are killed at once - exactly the behavior before stop handlers existed.
+
+The stop call blocks for up to `stop_timeout_ms` while it waits. Keep the
+timeout above the longest piece of work the service does between two checks
+of its stop flag.
+
+The registration belongs to the calling *process*, not to a unit name, so it
+follows the process a unit tracks - including one adopted with
+`libsystemd_notify_main_pid()`. It is dropped automatically when the process
+terminates. Clear it (`NULL` handler) before freeing the handler's argument:
+that call waits for a handler invocation still in progress.
+
 ## Templates
 
 `libsystemd` supports `systemd`-style template units, identified by a literal

app/libsystemd/include/libsystemd.h

@@ -8,6 +8,7 @@
 dmod_libsystemd_api(1.0, int, _start_service, ( const char* unit_name, const char* user_value ));
 dmod_libsystemd_api(1.0, int, _stop_service, ( const char* unit_name ));
 dmod_libsystemd_api(1.0, int, _notify_main_pid, ( const char* unit_name, Dmod_Pid_t pid ));
+dmod_libsystemd_api(1.0, int, _set_stop_handler, ( libsystemd_stop_handler_t handler, void* arg ));
 dmod_libsystemd_api(1.0, int, _status, ( const char* unit_name, libsystemd_service_status_t* out_status ));
 dmod_libsystemd_api(1.0, int, _list, (libsystemd_visitor_t visitor, void* user_ptr));
 

app/libsystemd/include/libsystemd_types.h

@@ -48,6 +48,19 @@ typedef struct
     libsystemd_service_status_t     status;
 } libsystemd_service_info_t;
 
+/**
+ * @brief Stop request handler registered by a service with libsystemd_set_stop_handler()
+ *
+ * Called by libsystemd_stop_service() in the *stopper's* context (whichever
+ * thread asked for the stop), before it waits for the service's process to
+ * exit. It must only make the service leave on its own - set a flag, post a
+ * semaphore the service is waiting on - and return at once: it must not
+ * block, and must not call back into libsystemd.
+ *
+ * @param arg The pointer passed to libsystemd_set_stop_handler().
+ */
+typedef void (*libsystemd_stop_handler_t)(void* arg);
+
 /**
  * @brief pointer to a visitor function
  */

app/libsystemd/src/serviceapi.c

@@ -8,6 +8,21 @@
 // its surface (see the libsystemd_services_t doc in libsystemd_types.h), so the
 // registry's own list handling includes it here instead.
 #include "dmlist.h"
+#include "stophandler.h"
+
+/**
+ * @brief Default for a unit's "stop_timeout_ms" key
+ *
+ * How long libsystemd_stop_service() waits for a service that registered a
+ * stop handler (libsystemd_set_stop_handler()) to exit on its own before it
+ * falls back to dmosi_process_kill().
+ */
+#define LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS  3000u
+
+/**
+ * @brief Poll period while waiting for a stopping service to exit
+ */
+#define LIBSYSTEMD_STOP_POLL_MS             10u
 
 /**
  * @brief Full definition of the opaque @ref libsystemd_service_t handle
@@ -29,6 +44,7 @@ struct libsystemd_service
     char* description;                  //!< From the optional "description" key, or NULL if unset. Owned copy.
     libsystemd_service_type_t type;                    //!< From the optional "type" key ("simple"/"oneshot"). Defaults to ::LIBSYSTEMD_SERVICE_TYPE_SIMPLE.
     libsystemd_restart_policy_t restart_policy;         //!< From the optional "restart" key ("no"/"always"/"on-failure"). Defaults to ::LIBSYSTEMD_RESTART_NO.
+    uint32_t stop_timeout_ms;           //!< From the optional "stop_timeout_ms" key: how long a stop waits for a service with a stop handler to exit on its own before killing it. 0 = kill at once. Defaults to ::LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS.
     char* exec;                         //!< Module name (or file path) to spawn, from the "exec" key. Owned copy.
     int argc;                           //!< Number of entries in argv (always >= 1, argv[0] == exec).
     char** argv;                        //!< NULL-terminated argument vector (argc+1 entries, each an owned copy).
@@ -1145,14 +1161,85 @@ static bool libsystemd_start_service_visitor(void* data, void* user_data)
     return true;
 }
 
+/**
+ * @brief Wait until a process has terminated (or can no longer be found)
+ *
+ * The process is looked up by PID on every poll rather than held as a
+ * `dmosi_process_t`, so its structure going away while waiting is harmless.
+ *
+ * @param pid        Process to wait for.
+ * @param timeout_ms Upper bound of the wait.
+ *
+ * @retval true  The process is gone.
+ * @retval false It is still running after @p timeout_ms.
+ */
+static bool libsystemd_wait_for_exit(dmosi_process_id_t pid, uint32_t timeout_ms)
+{
+    uint32_t waited = 0;
+    for (;;)
+    {
+        dmosi_process_t process = dmosi_process_find_by_id(pid);
+        if (process == NULL || dmosi_process_get_state(process) == DMOSI_PROCESS_STATE_TERMINATED)
+        {
+            return true;
+        }
+        if (waited >= timeout_ms)
+        {
+            return false;
+        }
+        dmosi_thread_sleep(LIBSYSTEMD_STOP_POLL_MS);
+        waited += LIBSYSTEMD_STOP_POLL_MS;
+    }
+}
+
+/**
+ * @brief Ask a service's process to exit on its own, through its stop handler
+ *
+ * Skipped (returns false at once) when the unit's `stop_timeout_ms` is 0, when
+ * the process registered no handler with libsystemd_set_stop_handler(), and
+ * when the process is the caller itself - waiting for our own exit could only
+ * ever time out.
+ *
+ * @param service Service whose tracked `pid` is being stopped (must not be NULL).
+ *
+ * @retval true  The process exited on its own within `stop_timeout_ms`.
+ * @retval false It has to be killed (no handler, disabled, or it did not exit in time).
+ */
+static bool libsystemd_request_graceful_stop(libsystemd_service_t service)
+{
+    dmosi_process_id_t pid = (dmosi_process_id_t)service->pid;
+    dmosi_process_t self = dmosi_process_current();
+
+    if (service->stop_timeout_ms == 0 || (self != NULL && dmosi_process_get_id(self) == pid))
+    {
+        return false;
+    }
+    if (!libsystemd_stophandler_request(pid))
+    {
+        return false;
+    }
+    if (libsystemd_wait_for_exit(pid, service->stop_timeout_ms))
+    {
+        return true;
+    }
+
+    DMOD_LOG_WARN("Service '%s' did not stop within %u ms, killing it\n",
+                  service->unit_name, (unsigned)service->stop_timeout_ms);
+    return false;
+}
+
 /**
  * @brief Actually stop a service's process, without looking it up by name first
  *
  * Shared by libsystemd_stop_service() (which looks the service up by unit name
  * first) and libsystemd_stop_all_services_visitor() (which already has a direct
  * pointer while walking a registry that is about to be replaced/torn down).
  *
- * Resolves `service->pid` to a live `dmosi_process_t` and kills it via
+ * Resolves `service->pid` to a live `dmosi_process_t`. If that process
+ * registered a stop handler (libsystemd_set_stop_handler()), it is asked to
+ * exit on its own first and given up to the unit's `stop_timeout_ms`
+ * (libsystemd_request_graceful_stop()) - this blocks the caller for at most
+ * that long. Otherwise, or if it does not exit in time, it is killed via
  * `dmosi_process_kill()`. Always clears `service->pid` back to the "not
  * running" sentinel (-1) once the process is confirmed gone or killed, so a
  * subsequent libsystemd_start_service() call is never blocked by stale state.
@@ -1165,7 +1252,7 @@ static bool libsystemd_start_service_visitor(void* data, void* user_data)
  *
  * @param service Service to stop (must not be NULL).
  *
- * @retval 0      The service's process was found and killed successfully.
+ * @retval 0      The service's process was found and exited on request or was killed successfully.
  * @retval -ESRCH The service had no running process to stop (`pid` was already
  *                 <= 0, or the process it pointed to could no longer be found -
  *                 e.g. it already exited on its own).
@@ -1203,6 +1290,20 @@ static int libsystemd_stop_service_internal(libsystemd_service_t service)
 
     DMOD_LOG_INFO("Stopping service '%s'\n", service->unit_name);
 
+    if (libsystemd_request_graceful_stop(service))
+    {
+        service->pid = -1;
+        return 0;
+    }
+
+    // Looked up again: after a graceful attempt the old handle may be stale.
+    process = dmosi_process_find_by_id((dmosi_process_id_t)service->pid);
+    if (process == NULL)
+    {
+        service->pid = -1;
+        return 0;
+    }
+
     int result = dmosi_process_kill(process, 0);
     if (result != 0)
     {
@@ -2177,6 +2278,28 @@ static libsystemd_restart_policy_t libsystemd_parse_restart_policy(dmini_context
     return LIBSYSTEMD_RESTART_NO;
 }
 
+/**
+ * @brief Parse a unit's optional "stop_timeout_ms" key
+ *
+ * @param ctx  Unit's ini context (must not be NULL).
+ * @param exec The unit's `exec` value, used only in the warning message.
+ *
+ * @return The configured timeout, or ::LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS if
+ *         the key is absent or negative (the latter logged as a warning).
+ */
+static uint32_t libsystemd_parse_stop_timeout(dmini_context_t ctx, const char* exec)
+{
+    int value = dmini_get_int(ctx, NULL, "stop_timeout_ms", (int)LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS);
+    if (value < 0)
+    {
+        DMOD_LOG_WARN("Unit with exec '%s' has negative stop_timeout_ms %d, using %u\n",
+                      exec, value, (unsigned)LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS);
+        return LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS;
+    }
+
+    return (uint32_t)value;
+}
+
 /**
  * @brief Build a fresh `libsystemd_service_t` from an already-parsed ini context
  *
@@ -2215,6 +2338,7 @@ static int libsystemd_build_service_from_ctx(dmini_context_t ctx, libsystemd_ser
     new_service->description = NULL;
     new_service->type = LIBSYSTEMD_SERVICE_TYPE_SIMPLE;
     new_service->restart_policy = LIBSYSTEMD_RESTART_NO;
+    new_service->stop_timeout_ms = LIBSYSTEMD_DEFAULT_STOP_TIMEOUT_MS;
     new_service->exec = NULL;
     new_service->argc = 0;
     new_service->argv = NULL;
@@ -2232,6 +2356,7 @@ static int libsystemd_build_service_from_ctx(dmini_context_t ctx, libsystemd_ser
 
     new_service->type = libsystemd_parse_service_type(ctx, exec);
     new_service->restart_policy = libsystemd_parse_restart_policy(ctx, exec);
+    new_service->stop_timeout_ms = libsystemd_parse_stop_timeout(ctx, exec);
 
     bool ok = (new_service->exec = Dmod_StrDup(exec)) != NULL;
     ok = ok && libsystemd_build_argv(new_service, exec, args);
@@ -2906,6 +3031,13 @@ static int libsystemd_serviceapi_init(void)
         return -ENOMEM;
     }
 
+    int result = libsystemd_stophandler_init();
+    if (result != 0)
+    {
+        libsystemd_destroy_services(services);
+        return result;
+    }
+
     g_services = services;
 
     return 0;
@@ -2945,6 +3077,9 @@ static void libsystemd_serviceapi_deinit(void)
 
     libsystemd_destroy_devices(g_devices);
     g_devices = NULL;
+
+    // Last: stopping services above may still ask their stop handlers.
+    libsystemd_stophandler_deinit();
 }
 
 /**