Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ All commands follow a `:RESOURCE:ACTION:` naming convention. New commands must u
| `:PLACED:CREATE:`, `:PLACED:EVENT:` | Placed object (mine/explosive) lifecycle |
| `:ACE3:DEATH:`, `:ACE3:UNCONSCIOUS:` | ACE3 integration events |
| `:SYS:INIT:`, `:STORAGE:INIT:` | Initialize extension and storage |
| `:MISSION:START:`, `:MISSION:SAVE:` | Begin/end recording |
| `:MISSION:START:`, `:MISSION:SAVE:`, `:MISSION:DISCARD:` | Begin/end recording, or end it unsaved |
| `:MISSION:KEEP:` | Keep a recording the game moves on from (it met the addon's save limits) |

### Data Flow

Expand Down
44 changes: 37 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,15 +153,43 @@ paced. `0` disables pacing.
per mission (0 = unlimited). Once the limit is reached, spooling stops for that
mission, and a later full resend is partial.

The spool also survives the game server. If it crashes mid-mission, or a save
is not confirmed, the mission stays in `spoolDir`. The next extension process
checks `spoolDir` at startup and then every minute, resends such missions to
the web service, and deletes each spool once the web service has saved its
mission:
The spool also survives the game server. If it crashes mid-mission, or a
mission ends without a confirmed save, the mission stays in `spoolDir`. The
extension checks `spoolDir` at startup and then every minute, resends such
missions to the web service, and deletes each spool once the web service has
saved its mission. A mission resent in full is marked as recovered, so a web
service at protocol version 3 does not take it for the producer's next
mission and save the mission it records now as `partial`. An older web service
does, so with one the running process resends its own missions in full only
while it records none:
- A mission cut short by a crash is ended on the crashed process' behalf and
saved as `partial`, with everything recorded up to the last flush (at most
about a second is lost).
- A mission whose save was not confirmed is saved as recorded.
- A mission whose discard was not confirmed is not resent; the web service is
only told again to drop it, including any `partial` recording it saved of
the mission in the meantime. This needs the spool: without `spoolDir`, or
with a spool that hit its limit or failed, nothing is kept, and the web
service saves the mission as `partial` once its resume grace expires.

When the game moves on to the next mission without saving or discarding the
current one, for example after an admin `#missions`, the proxy ends it as
interrupted as soon as the next mission loads (`:STORAGE:INIT:`), and the web
service saves it as `partial`. Only if that is not confirmed within 30 seconds
does its spool stay behind for recovery. One that recorded nothing after its
start is discarded instead, when the web service supports it, and so is one
the addon would not save yet: it starts such a recording with `"discard"` and
sends `:MISSION:KEEP:` once it meets `OCAP_settings_minMissionTime`, so a
mission an admin only loaded and left with `#missions` or `#restart` leaves
no `partial` behind. A web service
without resume support is disconnected instead, which saves the mission as
`partial` there.

When the addon ends a recording that does not meet its save limits (for
example one shorter than `OCAP_settings_minMissionTime`), it sends
`:MISSION:DISCARD:`, and the proxy tells the web service to drop the recording
(protocol version 3) and deletes its spool. A web service that predates
discard keeps saving such recordings as `partial`.

Game servers can share one `spoolDir`, for example several instances started
from the same directory. Each spool has an operating system file lock
Expand Down Expand Up @@ -265,9 +293,11 @@ All commands follow a `:RESOURCE:ACTION:` naming convention. New commands must u
| Command | Purpose |
|---------|---------|
| `:SYS:INIT:` | Initialize extension, send `:SYS:READY:` callback |
| `:STORAGE:INIT:` | Initialize storage backend, ungate buffered handlers |
| `:MISSION:START:` | Start recording mission |
| `:STORAGE:INIT:` | Initialize storage backend once per process (later calls reuse it), ungate buffered handlers; answers with a `:STORAGE:OK:` callback, or `:STORAGE:ERROR:` with the error |
| `:MISSION:START:` | Start recording mission; answers with a `:MISSION:OK:` callback, or `:MISSION:FAILED:` with the error. An optional third argument, a registration token, is echoed as the callback's last value. An optional fourth, `discard`, drops the recording if the game moves on from it before `:MISSION:KEEP:` (`websocket`) |
| `:MISSION:KEEP:` | The recording met the addon's save limits: if the game moves on from it now, it is kept as `partial` (`websocket`) |
| `:MISSION:SAVE:` | End recording, flush data, upload if configured |
| `:MISSION:DISCARD:` | End recording without saving it (`memory`, `websocket`); answers with a `:MISSION:DISCARDED:` callback |
| `:MISSION:FOCUS_START:` | Set playback focus start frame |
| `:MISSION:FOCUS_END:` | Set playback focus end frame |
| `:SYS:VERSION:` | Get extension version |
95 changes: 81 additions & 14 deletions cmd/ocap_recorder/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"path/filepath"
"runtime"
"sync"
"sync/atomic"
"time"

"github.com/OCAP2/extension/v5/internal/api"
Expand Down Expand Up @@ -91,6 +92,10 @@ var (
// storageReady is closed when storage (DB or memory) is initialized and ready
storageReady = make(chan struct{})
storageReadyOnce sync.Once
// missionLoadedAt is when the last :STORAGE:INIT:, sent on every
// mission load, was dispatched. Compared with Before, which uses the
// monotonic clock, so a wall clock stepping back does not matter.
missionLoadedAt atomic.Pointer[time.Time]

// Services
parserService parser.Service
Expand Down Expand Up @@ -312,9 +317,43 @@ func initAPIClient() {

// handleNewMission handles the :MISSION:START: command: parses mission data,
// delegates DB persistence to the storage backend, and sets runtime state.
func handleNewMission(e dispatcher.Event) (any, error) {
// It answers with a :MISSION:OK: callback, or :MISSION:FAILED: with the error,
// so the addon never waits for a mission that did not start.
//
// An optional third argument is the addon's registration token, echoed in
// both callbacks, so a late answer to an earlier mission's start is not taken
// for this one's. An optional fourth, "discard", drops the recording if the
// game moves on from it before :MISSION:KEEP: (see storage.Abandonable).
func handleNewMission(e dispatcher.Event) (result any, err error) {
token := ""
if len(e.Args) > 2 {
token = util.TrimQuotes(e.Args[2])
}
reportFailure := func(msg string) {
if cbErr := a3interface.WriteArmaCallback(ExtensionName, ":MISSION:FAILED:", msg, token); cbErr != nil {
Logger.Warn("Failed to send MISSION:FAILED callback", "error", cbErr)
}
}
defer func() {
if r := recover(); r != nil {
reportFailure(fmt.Sprintf("panic during mission start: %v", r))
panic(r) // the dispatcher logs it
}
}()
result, err = startNewMission(e)
if err != nil {
reportFailure(err.Error())
return result, err
}
if err := a3interface.WriteArmaCallback(ExtensionName, ":MISSION:OK:", "OK", token); err != nil {
Logger.Warn("Failed to send MISSION:OK callback", "error", err)
}
return result, nil
}

func startNewMission(e dispatcher.Event) (any, error) {
if parserService == nil {
return nil, nil
return nil, fmt.Errorf("extension not initialized")
}

// 1. Parse (returns core types)
Expand All @@ -334,20 +373,27 @@ func handleNewMission(e dispatcher.Event) (any, error) {
MarkerCache.Reset()
EntityCache.Reset()

// 3. Start backend (handles DB ops for GORM/SQLite backends, stores missionID internally)
// 3. Start backend (handles DB ops for GORM/SQLite backends, stores missionID internally),
// once the previous mission's save or discard, dispatched before this
// start, has run.
saveState.waitIdle()
// A start that waited behind a save while the game loaded the next
// mission belongs to the mission that is gone: nothing would record it.
if loaded := missionLoadedAt.Load(); loaded != nil && !e.Timestamp.IsZero() && e.Timestamp.Before(*loaded) {
return nil, fmt.Errorf("the mission was unloaded before it started")
}
if storageBackend != nil {
if err := storageBackend.StartMission(&coreMission, &coreWorld); err != nil {
Logger.Error("Failed to start mission in storage backend", "error", err)
return nil, err
}
// Before :MISSION:OK:, so before the addon records anything.
if a, ok := storageBackend.(storage.Abandonable); ok && len(e.Args) > 3 && util.TrimQuotes(e.Args[3]) == "discard" {
a.SetDiscardAbandoned(true)
}
}

Logger.Info("New mission logged", "missionName", coreMission.MissionName)

// 4. ArmA callback
if err := a3interface.WriteArmaCallback(ExtensionName, ":MISSION:OK:", "OK"); err != nil {
Logger.Warn("Failed to send MISSION:OK callback", "error", err)
}
return nil, nil
}

Expand All @@ -360,11 +406,11 @@ func registerLifecycleHandlers(d *dispatcher.Dispatcher) {
})

d.Register(":STORAGE:INIT:", func(e dispatcher.Event) (any, error) {
go func() {
if err := initStorage(); err != nil {
Logger.Error("Storage initialization failed", "error", err)
}
}()
// A mission loaded: a start dispatched before it belongs to the
// previous one (see startNewMission).
loaded := e.Timestamp
missionLoadedAt.Store(&loaded)
go startStorage()
return nil, nil
})

Expand Down Expand Up @@ -403,7 +449,10 @@ func registerLifecycleHandlers(d *dispatcher.Dispatcher) {
return nil, nil
})

d.Register(":MISSION:START:", handleNewMission, dispatcher.Buffered(1), dispatcher.Blocking(), dispatcher.Gated(storageReady))
// Not blocking: a start waits for a save or discard still ending the
// previous mission, and a queue it filled must not hold up the game's
// thread. A refused start is reported to the addon by callExtension.
d.Register(":MISSION:START:", handleNewMission, dispatcher.Buffered(1), dispatcher.Gated(storageReady))

d.Register(":MISSION:SAVE:", func(e dispatcher.Event) (any, error) {
// Optional save-time tag override (addon exportData _tag). Sent as a
Expand All @@ -416,4 +465,22 @@ func registerLifecycleHandlers(d *dispatcher.Dispatcher) {
Logger.Info("Received :MISSION:SAVE: command, queuing async save", "tag", tag)
return triggerMissionSave(saveState, storageBackend, apiClient, tag)
})

// The addon ended a recording without saving it, for example one shorter
// than its minimum mission time.
d.Register(":MISSION:DISCARD:", func(e dispatcher.Event) (any, error) {
Logger.Info("Received :MISSION:DISCARD: command, queuing async discard")
return triggerMissionDiscard(saveState, storageBackend)
})

// The recording met the addon's save limits: if the game moves on from
// it now, it is kept (as partial) instead of dropped. Synchronous, so it
// is in place before a mission load that follows it.
d.Register(":MISSION:KEEP:", func(e dispatcher.Event) (any, error) {
Logger.Info("Received :MISSION:KEEP: command; an abandoned recording is kept")
if a, ok := storageBackend.(storage.Abandonable); ok {
a.SetDiscardAbandoned(false)
}
return nil, nil
})
}
146 changes: 146 additions & 0 deletions cmd/ocap_recorder/main_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
package main

import (
"testing"
"time"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/OCAP2/extension/v5/internal/dispatcher"
"github.com/OCAP2/extension/v5/internal/logging"
"github.com/OCAP2/extension/v5/internal/parser"
"github.com/OCAP2/extension/v5/pkg/a3interface"
"github.com/OCAP2/extension/v5/pkg/core"
)

// isolateMissionStart records callbacks and gives a test its own parser and
// no storage. Logger is left alone: the package init still reads it.
func isolateMissionStart(t *testing.T, p parser.Service) *callbackRecorder {
t.Helper()
rec := &callbackRecorder{}
a3interface.SetCallbackSinkForTest(rec.sink)
previousParser, previousBackend := parserService, storageBackend
t.Cleanup(func() {
a3interface.SetCallbackSinkForTest(nil)
parserService, storageBackend = previousParser, previousBackend
})
parserService = p
storageBackend = nil
return rec
}

func TestHandleNewMission_FailureIsReported(t *testing.T) {
rec := isolateMissionStart(t, nil)

_, err := handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-1"`}})
require.Error(t, err)
cb := rec.wait(t, ":MISSION:FAILED:")
require.Len(t, cb.data, 2)
assert.Contains(t, cb.data[0], "not initialized")
assert.Equal(t, `"tok-1"`, cb.data[1], "the addon's registration token is echoed")

rec.mu.Lock()
rec.calls = nil
rec.mu.Unlock()
parserService = parser.NewParser(Logger)
_, err = handleNewMission(dispatcher.Event{Args: []string{"not json"}})
require.Error(t, err)
cb = rec.wait(t, ":MISSION:FAILED:")
assert.Equal(t, `""`, cb.data[1], "an addon without a token gets an empty one")
}

// acceptingParser starts every mission.
type acceptingParser struct{ parser.Service }

func (acceptingParser) ParseMission([]string) (core.Mission, core.World, error) {
return core.Mission{MissionName: "M"}, core.World{WorldName: "W"}, nil
}

func TestHandleNewMission_WaitsForSaveOrDiscard(t *testing.T) {
rec := isolateMissionStart(t, acceptingParser{})

// The previous mission's discard was dispatched just before this start.
saveState.backendBusy.Store(true)
t.Cleanup(func() { saveState.backendBusy.Store(false) })
done := make(chan error, 1)
go func() {
_, err := handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-3"`}})
done <- err
}()
select {
case <-done:
t.Fatal("the start ran before the discard finished")
case <-time.After(100 * time.Millisecond):
}
saveState.backendBusy.Store(false)
require.NoError(t, <-done)
rec.wait(t, ":MISSION:OK:")
}

func TestHandleNewMission_RejectsStartOfUnloadedMission(t *testing.T) {
rec := isolateMissionStart(t, acceptingParser{})
previous := missionLoadedAt.Load()
t.Cleanup(func() { missionLoadedAt.Store(previous) })

// The start was dispatched, then the next mission loaded while it waited.
dispatched := time.Now()
loaded := dispatched.Add(time.Second)
missionLoadedAt.Store(&loaded)
_, err := handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-4"`}, Timestamp: dispatched})
require.Error(t, err)
cb := rec.wait(t, ":MISSION:FAILED:")
assert.Equal(t, `"tok-4"`, cb.data[1])

// A start dispatched after the load goes through.
_, err = handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-5"`}, Timestamp: dispatched.Add(2 * time.Second)})
require.NoError(t, err)
rec.wait(t, ":MISSION:OK:")
}

func TestHandleNewMission_OKEchoesToken(t *testing.T) {
rec := isolateMissionStart(t, acceptingParser{})

_, err := handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-2"`}})
require.NoError(t, err)
cb := rec.wait(t, ":MISSION:OK:")
assert.Equal(t, []string{`"OK"`, `"tok-2"`}, cb.data)
}

func TestHandleNewMission_DiscardAbandonedUntilKept(t *testing.T) {
rec := isolateMissionStart(t, acceptingParser{})
backend := &abandoningBackend{rec: rec}
storageBackend = backend

_, err := handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-6"`}})
require.NoError(t, err)
assert.Empty(t, backend.discard, "an addon that does not ask keeps an abandoned recording")

_, err = handleNewMission(dispatcher.Event{Args: []string{"{}", "{}", `"tok-7"`, `"discard"`}})
require.NoError(t, err)
assert.Equal(t, []bool{true}, backend.discard)

// The recording met the addon's save limits.
d, err := dispatcher.New(logging.NewDispatcherLogger(Logger))
require.NoError(t, err)
registerLifecycleHandlers(d)
_, err = d.Dispatch(dispatcher.Event{Command: ":MISSION:KEEP:"})
require.NoError(t, err)
assert.Equal(t, []bool{true, false}, backend.discard)
}

// panickingParser fails every mission start with a panic.
type panickingParser struct{ parser.Service }

func (panickingParser) ParseMission([]string) (core.Mission, core.World, error) {
panic("boom")
}

func TestHandleNewMission_PanicIsReported(t *testing.T) {
rec := isolateMissionStart(t, panickingParser{})

assert.Panics(t, func() { _, _ = handleNewMission(dispatcher.Event{}) }, "the dispatcher still sees the panic")
cb := rec.wait(t, ":MISSION:FAILED:")
require.Len(t, cb.data, 2)
assert.Contains(t, cb.data[0], "panic")
}
Loading
Loading