Skip to content

Read a video's start time from its own clock, not creation time minus duration - #832

Open
caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:fix-video-start-time
Open

caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:fix-video-start-time

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

The bug

A community report (Mapillary forum + FB group): a dashcam MP4 recorded at 4.91 fps, paired
with a GPX of the same drive, comes out with its frames on the wrong part of the route.
The same file + GPX works in Google Street View.

The reporter described it as the frame rate being misread. It isn't — the container is parsed
correctly (mdhd timescale 15712, stts 3186 × 3200 → exactly 4.91 fps, and
mp4_sample_parser reproduces every composition time to the microsecond). What is wrong is
the video's start time, and since the frames are located along the GPX by their absolute
timestamps, getting that wrong moves all of them together.

Probe.extract_stream_start_time() returned creation_time - duration. For the reported file
(creation_time 2026-08-08T08:47:32Z, duration 648.879837s) that is 08:36:43.120Z, while the
GPX starts at 08:47:32Z — the creation time, to the second. Every sampled frame came out one
full video duration early, and the workflow the README documents fails outright:

$ mapillary_tools video_process VIDEO_DIR --geotag_source gpx \
      --geotag_source_path R0024681.gpx --video_sample_distance -1 --video_sample_interval 20
...
32 image read in total
        32 (81.0 MB) MapillaryOutsideGPXTrackError
MapillaryProcessError: Failed to process 32 files.

The hard failure is the lucky case. When the track does reach back far enough — a GPX covering
a drive that the camera split across several files, which is how a ~5 GB-per-file dashcam
records — the frames from every file after the first are silently placed one file-length back
along the route. On this track that is up to ~1 km, and the images end up bunched onto road
they were not shot on, which is what the report describes.

Where it came from

The subtraction was introduced in 644701c (Feb 2019). The comment it deleted says what it was
for:

# Blackvue actually reports endtime in the created_at exif field

It was a BlackVue workaround that later got generalised to every video. ISO/IEC 14496-12
defines creation_time as the creation time of the presentation, which for a recording is the
moment it started, and that is what cameras and ffmpeg actually write.

The fix

Take the start time from the video's own telemetry when it has any, and fall back to
creation_time otherwise:

  1. --video_start_time, if given (unchanged)
  2. the first absolute GPS timestamp in the video's telemetry, mapped back to the video's time 0
  3. the container's creation_time

Step 2 is what keeps the cameras the old workaround was written for correct, without guessing:
their GPS clock is absolute UTC, so it is right for BlackVue, whose container time really is
the end of the recording, and for GoPro, whose container time is in local time (#819). Step 3
then applies to exactly the videos that have no telemetry to sync against — the plain MP4s
people geotag from a GPX.

Distance-based sampling already timestamps frames from that same GPS clock, so this also makes
the two sampling paths agree instead of differing by 8 hours on a GoPro.

Behaviour changes

  • Videos without telemetry (the reported case): frames are timestamped from creation_time
    instead of creation_time - duration, i.e. one duration later than before.
  • GoPro / BlackVue / CAMM with interval sampling: frames are timestamped from the GPS clock.
    For the hero8.mp4 fixture that moves the first sample from 2019_11_18_15_41_12_354 (the
    camera's local RTC, minus the duration, labelled UTC) to 2019_11_18_23_42_08_645 (real UTC,
    matching the GPS positions written into the same frames). This is Video Timestamping Bugs with GoPro MAX #819 items 2 and 4.
  • Distance-based sampling is unaffected for tracks with absolute timestamps; it already
    used them.
  • Users whose camera writes neither correctly still have --video_start_time.

Verification

On the reporter's files (10:49 video, 1630-point GPX), video_process --geotag_source gpx --video_sample_distance -1 --video_sample_interval 2 now yields 289 geotagged frames spanning
08:47:32 → 08:58:18, and every one sits 0.0000 m from the GPX position at its true video
time. Before the change: 0 frames, all rejected as outside the track.

pytest tests/unit (690 passed), pytest tests/integration (52 passed), ruff check,
ruff format --check and mypy all pass.

Not addressed here

--video_geotag_source gpx still rebases the GPX onto video time 0, so a GPX covering several
video files is injected whole into each of them. That is the documented limitation under
"Limitations of --video_geotag_source" and is a separate change.

… duration

A video's start time decides where its sampled frames land on a GPS track.
We derived it as creation_time - duration, which pushed every video one full
duration into the past unless its camera happened to stamp the creation time
at the end of the recording.

That subtraction came from 644701c (2019), whose removed comment says why:
"Blackvue actually reports endtime in the created_at exif field". The
workaround was later generalised to every video. ISO/IEC 14496-12 defines
creation_time as the creation time of the presentation, which for a recording
is the moment it started, and that is what cameras and ffmpeg write.

Take the start time from the video's own telemetry when it has some. That
clock is absolute UTC, so it is right for BlackVue, whose container time is
the end of the recording, and for GoPro, whose container time is in local
time (mapillary#819). Fall back to creation_time for the plain MP4s that get geotagged
from a GPX, which are exactly the videos that have no telemetry to sync
against.

Reported for a 10:49 dashcam video paired with a GPX starting at the video's
creation time: every sampled frame came out 648.88s early, so the documented
GPX workflow failed with MapillaryOutsideGPXTrackError on all of them. Where
the track does reach back far enough -- a GPX covering a recording split into
several files -- the frames were silently placed up to a kilometre off
instead. With this change all 289 sampled frames land on the track exactly.
@meta-cla meta-cla Bot added the cla signed label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant