Coordinator/place keepalive

[asher-pem-arm]

Background:

Labgrid currently supports place acquisition, but acquired places remain locked until they are released explicitly.
In practice, this can lead to stale locks, where a place stays occupied even after the user is no longer actively using it.

This reduces availability of shared lab resources and often requires manual intervention to recover locked places.

What problem this solves

This MR introduces lease-based place locking, allowing places to be acquired for a limited time and released automatically if the lease is not extended.

This prevents stale locks and keeps place usage self-cleaning, while also integrating naturally with the existing reservation and scheduler semantics.

Overview:

This change adds a new lease acquisition mode alongside the existing acquire mode:

• Acquire — default behaviour, locks a place indefinitely
• Lease — time-limited lock, requires an active reservation and must
  be extended explicitly

Leased places are released automatically when the associated reservation expires, is cancelled, or when lease extension fails.

Lease timing and limits

Lease timing is not user-configurable via CLI. Instead, it is governed by coordinator configuration:

• initial lease time is assigned automatically on lease
• each extend increases the lease by a fixed amount
• total lease time is capped by a maximum duration

This design ensures consistent behavior across users and prevents misuse or overly long locks.

Lease extension failure behavior

If extending a lease fails (for example due to exporter/resource errors),
the coordinator will cancel the lease and release the place.

This avoids leaving the system in a partially extended or inconsistent state,
especially when a place consists of multiple resources.

From the user perspective, a failed extend means the lease is no longer valid
and the place must be leased again.

CLI examples

Create a reservation:

labgrid-client reserve board=imx8 # the LG_TOKEN will be printed upon success

Lease a place using the dedicated lease command:

labgrid-client -p +<LG_TOKEN> lease

Extend the reservation to keep the lease alive:

labgrid-client extend <LG_TOKEN> --keepalive # automatically renews lease until failure or max duration

Acquire a place indefinitely (unchanged behaviour):

labgrid-client -p board-01 lock

or

labgrid-client -p board-01 acquire

[Emantor]

Commit 8a098cc should be dropped, it's a function rename now while the functionality was already merged in #1872.

[codecov]

Codecov Report

❌ Patch coverage is 4.70588% with 324 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.1%. Comparing base (915b591) to head (e65832f).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
labgrid/remote/coordinator.py	0.0%	189 Missing ⚠️
labgrid/remote/client.py	7.4%	100 Missing ⚠️
labgrid/remote/exporter.py	0.0%	26 Missing ⚠️
labgrid/remote/common.py	47.0%	9 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #1884     +/-   ##
========================================
- Coverage    46.0%   45.1%   -0.9%     
========================================
  Files         180     180             
  Lines       14464   14764    +300     
========================================
+ Hits         6654    6667     +13     
- Misses       7810    8097    +287     

Flag	Coverage Δ
3.10	45.1% <4.7%> (-0.9%)	⬇️
3.11	45.1% <4.7%> (-0.9%)	⬇️
3.12	45.1% <4.7%> (-0.9%)	⬇️
3.13	45.1% <4.7%> (-0.9%)	⬇️
3.14	45.1% <4.7%> (-0.9%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[asher-pem-arm]

@Emantor updated to drop other commit

[asher-pem-arm]

note: have added a new commit remote/coordinator: add exporter notification for lease start
to inform the exporter of lease start. Can squash this into the first commit as needed

[asher-pem-arm]

note: have added another new commit for an edge case we found: remote/coordinator: fail lease extension for orphaned resourcesagain can squash post review

[jluebbe]

While it's not really clear yet how we'd implement fencing (stopping access to resources) on the client on timeout (be it due to temporary network issues, suspend/resume on a laptop or something else), I'd prefer if we could extend the core acquire concept with timeouts.

This would allow us to cleanly extend this to cover normal interactive lock/unlock as well.

The new behavior could be opt-in via labgrid-client -p foo lock --keepalive. Then, you'd have to ensure that you keep sending keepalives for this place. In CI, this could just be a labgrid-client -p foo keepalive , because if the CI job is killed, the keepalive process is terminated with the rest.

[asher-pem-arm]

While it's not really clear yet how we'd implement fencing (stopping access to resources) on the client on timeout (be it due to temporary network issues, suspend/resume on a laptop or something else), I'd prefer if we could extend the core acquire concept with timeouts.

This would allow us to cleanly extend this to cover normal interactive lock/unlock as well.

The new behavior could be opt-in via labgrid-client -p foo lock --keepalive. Then, you'd have to ensure that you keep sending keepalives for this place. In CI, this could just be a labgrid-client -p foo keepalive , because if the CI job is killed, the keepalive process is terminated with the rest.

Hey @jluebbe ,

Thanks I can see the benefit of making timeout-based acquisition available to interactive lock users as well. We did consider that but for the shared CI use case this PR targets we think a separate reservation-backed lease provides stronger guarantees than adding keepalive directly.

A lease in this proposal:

• Must originate from a scheduler-managed reservation, so temporary access cannot bypass reservation allocation, filtering and priority
• Has a coordinator-defined maximum lifetime. A client that remains alive, loops indefinitely or is otherwise stuck cannot retain the place forever merely by continuing to send keepalives
• Couples reservation and place state. Expiry or cancellation of the reservation also releases the place, rather than introducing a second independent lifetime
• Extends the complete allocation through the coordinator and exporters. If one resource cannot be extended, the lease is cancelled and the place is released instead of leaving a partially valid multi-resource allocation
• Exposes an explicit leased state, lease start time and timeout, which makes the behaviour easier to inspect and reason about operationally

[Emantor]

While it's not really clear yet how we'd implement fencing (stopping access to resources) on the client on timeout (be it due to temporary network issues, suspend/resume on a laptop or something else), I'd prefer if we could extend the core acquire concept with timeouts.
This would allow us to cleanly extend this to cover normal interactive lock/unlock as well.
The new behavior could be opt-in via labgrid-client -p foo lock --keepalive. Then, you'd have to ensure that you keep sending keepalives for this place. In CI, this could just be a labgrid-client -p foo keepalive , because if the CI job is killed, the keepalive process is terminated with the rest.

Hey @jluebbe ,

Thanks I can see the benefit of making timeout-based acquisition available to interactive lock users as well. We did consider that but for the shared CI use case this PR targets we think a separate reservation-backed lease provides stronger guarantees than adding keepalive directly.

A lease in this proposal:

* Must originate from a scheduler-managed reservation, so temporary access cannot bypass reservation allocation, filtering and priority

The same could be true for a time-based lock based on the coordinator side policy chosen by the coordinator operator.

* Has a coordinator-defined maximum lifetime. A client that remains alive, loops indefinitely or is otherwise stuck cannot retain the place forever merely by continuing to send keepalives

The same is also true for a time-based lock which implements the keepalive state.

* Couples reservation and place state. Expiry or cancellation of the reservation also releases the place, rather than introducing a second independent lifetime

What do you mean by second independent lifetime here? Reservations allocation is the part that associates an explicit place with a reservation and naturally time based locks would need to also release the place once expired?

* Extends the complete allocation through the coordinator and exporters. If one resource cannot be extended, the lease is cancelled and the place is released instead of leaving a partially valid multi-resource allocation

Locking a place where the exporter fails to lock a resource should fail either way.

* Exposes an explicit leased state, lease start time and timeout, which makes the behaviour easier to inspect and reason about operationally

Please elaborate on this. The existing locking is known to existing users, there have to be good arguments why we need to extend beyond the known "locking/acquire" meaning and fold in a new meaning ("lease") here as well.

[asher-pem-arm]

thanks @Emantor these are fair points.

A timed-lock implementation could provide many of the same guarantees. We prefer the current design because it makes them mandatory for every lease and ties the reservation, place and resources together.

Every lease requires a reservation, so the scheduler always handles place selection, filtering and priority. A timed lock could be made to follow the same rules, but the lease design enforces them automatically regardless of how it is used.

The implementation also distinguishes a renewable deadline from an absolute maximum lifetime. Keepalives extend the current deadline, but never beyond lease_start_time + max_lease_duration. A timed lock could implement the same rule, although doing so requires adding lease start and maximum-duration semantics to the existing lock model.

By second independent lifetime I meant having both a reservation timeout and a separate timeout on the acquired place. Here the reservation timeout is the lease deadline, extension renews the complete allocation, while reservation expiry or cancellation releases it. This gives the reservation, place and resources one controlling lifecycle. A timed lock could be made to follow the same rules, but the lease design enforces them automatically regardless of how it is used.

On exporter failure I was distinguishing renewal from initial acquisition. Our implementation revalidates exporter resources whenever the deadline is extended. As you say a timed-lock implementation should use the same mechanism.

The explicit leased state is intentional. Existing users understand lock/acquire as ownership that remains valid until explicitly released. A lease has different failure semantics: it must be renewed, expires automatically and has an absolute lifetime cap. Representing that separately makes the distinction visible in the CLI, coordinator state, logs and CLI outputs (e.g. show and reservations), while preserving existing lock behaviour unchanged.

These capabilities could all be folded into a sufficiently extended timed-lock model, but at that point it would effectively have the same lease semantics under the existing lock name. For the scheduled CI use case, we think the explicit reservation-backed lease is clearer and harder to misuse.