Dev

DESCRIPTION

@@ -65,4 +65,4 @@ NeedsCompilation: yes
 Roxygen: list(markdown = TRUE)
 LinkingTo: 
     cpp11
-Config/roxygen2/version: 8.1.0
+Config/roxygen2/version: 8.1.1

NAMESPACE

@@ -28,6 +28,7 @@ export(filter_by_time_of_day)
 export(filter_by_trip_id)
 export(filter_by_weekday)
 export(fix_gtfs)
+export(fix_overnight_times)
 export(frequencies_to_stop_times)
 export(get_active_services)
 export(get_calendar_overlap)
@@ -66,6 +67,7 @@ export(remove_unused_ids)
 export(set_dwell_time)
 export(set_max_speed)
 export(set_min_dwell_time)
+export(set_min_speed)
 export(set_route_frequency)
 export(set_route_speed)
 export(set_trip_speed)

NEWS.md

@@ -1,12 +1,23 @@
 # gtfstools (development version)
 
+## Bug fixes
+
+- gtfstools functions now accept `dt_gtfs` objects whose tables were turned into tibbles, grouped data frames or plain data frames by other packages (e.g. after `dplyr::group_by()`). With `by_reference = TRUE`, an object with any such table now raises an informative error, since the edits can't be made by reference on a converted copy: convert it first with `as_dt_gtfs()` (#130, reported by @baarthur, who also opened PR #132).
+- Several functions that involved estimating trip speed and duration would
+drop trips that cross midnight if GTFS input had times in 00:MM:SS, which
+disrespects the GTFS standard to report times as 24:MM:SS in such cases. 
+This bug has been fixed.  Now all functions that calculate or change durations
+and speeds now fix these times using a new function `fix_overnight_times()`
+and return correct estimates for overnight trips.
+
+
 ## New features
 
+- New function `fix_overnight_times()` rewrites the times of trips that run past midnight but are written as early morning times (e.g. `"00:30:00"` instead of `"24:30:00"`), as required by the GTFS specification. `fix_gtfs()` now runs it before capping speeds and filling blank stop times, so these trips are no longer left with blank times.
 - `interpolate_stop_times()` gains `overwrite` and `keep_dwell_times`. With `overwrite = TRUE`, the times of every stop between the first and the last stop of each trip are replaced by interpolated ones, instead of only the blank ones. By default (`keep_dwell_times = TRUE`), these stops keep their original dwell times.
+- `fix_gtfs()` gains `overwrite_stop_times`. With `overwrite_stop_times = TRUE`, the times of every stop between the first and the last stop of each trip are replaced by interpolated ones, keeping their dwell times, instead of only filling blank times (see `interpolate_stop_times()`).
+- `set_dwell_time()` gains `strategy`. The default, `"shift_trip_end"`, shifts the later times of the trips, as before. `"keep_total_duration"` splits the change around each intermediate stop, keeping the trip durations.
 
-## Bug fixes
-
-- gtfstools functions now accept `dt_gtfs` objects whose tables were turned into tibbles, grouped data frames or plain data frames by other packages (e.g. after `dplyr::group_by()`), converting them back to `data.table` without modifying the input. `as_dt_gtfs()` does the same conversion. With `by_reference = TRUE`, an object with any such table now raises an informative error, since the edits can't be made by reference on a converted copy: convert it first with `as_dt_gtfs()` (#130, reported by @baarthur, who also opened PR #132).
 
 # gtfstools 2.0.0
 
@@ -40,7 +51,7 @@
 - New functions to edit feeds:
   - `set_route_frequency()` sets the headway of routes within a time of day, replacing their trips in it with a frequency-based template trip. Departures outside the time of day don't change.
   - `set_route_speed()` sets the average speed of all trips of the given routes, as `set_trip_speed()` does for trips.
-  - `set_dwell_time()`, `set_min_dwell_time()` and `set_max_speed()` set dwell times, raise dwell times at intermediate stops to a minimum and cap trip speeds, shifting the later times of each trip accordingly. The dwell times can be set within a time of day, and speeds capped only between two stops or for trips departing within a time of day, as in `set_trip_speed()`. Like `set_trip_speed()` and `set_route_speed()`, they and `set_route_frequency()` can be restricted to some transport modes with `route_type`.
+  - `set_dwell_time()`, `set_min_dwell_time()`, `set_max_speed()` and `set_min_speed()` set dwell times, raise dwell times at intermediate stops to a minimum, and cap or raise trip speeds to a maximum or minimum, shifting the later times of each trip accordingly. The dwell times can be set within a time of day, and speeds changed only between two stops or for trips departing within a time of day, as in `set_trip_speed()`. Like `set_trip_speed()` and `set_route_speed()`, they and `set_route_frequency()` can be restricted to some transport modes with `route_type`.
   - `interpolate_stop_times()` fills blank arrival and departure times, assuming a constant speed between stops with known times.
   - `interpolate_shape_dist_traveled()` (re)calculates `shape_dist_traveled` in `stop_times` and `shapes`, in meters.
   - `stop_times_to_frequencies()`, the counterpart of `frequencies_to_stop_times()`, approximates scheduled trips by frequency-based ones (#69).

R/build_graph.R

@@ -67,7 +67,10 @@
 #' should usually be filtered to a single day first, with [filter_by_date()]
 #' or [filter_by_weekday()]. Departures after midnight have times past
 #' "24:00:00", so use a time of day such as "24:00:00"-"26:00:00" for them.
-#' Trips whose times wrongly go back to "00:00:00" after midnight are placed
+#' Times after midnight written as early morning times (e.g. `"00:30:00"`
+#' instead of `"24:30:00"`) are first fixed with [fix_overnight_times()], as
+#' are `end_time`s earlier than `start_time`s in `frequencies`, so these trips
+#' are placed at their real times. Trips whose times still go back are placed
 #' by their earliest time.
 #'
 #' Within a trip, consecutive `stop_times` entries at the same stop are one
@@ -224,6 +227,13 @@ build_graph <- function(gtfs,
     gtfsio::assert_field_class(gtfs, "stops", "stop_name", "character")
   }
 
+  # times after midnight written as early morning times are fixed first, so
+  # trips are placed at their real times. the departures of trips listed in
+  # 'frequencies' come from its start and end times, which are fixed too
+  # (fields of the wrong class are reported later, when they are used)
+
+  gtfs <- fix_overnight_in_call(gtfs, frequencies = TRUE)
+
   network <- build_service_network(
     gtfs,
     from_secs,
@@ -510,7 +520,7 @@ build_service_network <- function(gtfs,
 
   # in-vehicle links connect the service nodes of consecutive visits of each
   # trip, whose rows are contiguous. times that are missing or that decrease
-  # (e.g. times after midnight written as 00:xx) are not used
+  # (by 12 hours or less, since the others were fixed) are not used
 
   has_next <- c(trip_run[-1L] == trip_run[-length(trip_run)], FALSE)
   current <- which(has_next)

R/filter_by_time_of_day.R

@@ -26,6 +26,12 @@
 #' @section Details:
 #' The time of day includes both `from` and `to`.
 #'
+#' Times after midnight written as early morning times (e.g. `"00:30:00"`
+#' instead of `"24:30:00"`), and `end_time`s earlier than `start_time`s in
+#' `frequencies`, are first fixed with [fix_overnight_times()], so these
+#' times are compared with the time of day as they should be, and the
+#' returned GTFS has them fixed.
+#'
 #' Trips listed in `frequencies` are filtered through their `frequencies`
 #' entries; their `stop_times` entries are templates and are kept whatever
 #' their times. With `keep = TRUE`, the entries that overlap the time of day
@@ -132,6 +138,12 @@ filter_by_time_of_day <- function(gtfs,
   to_secs <- string_to_seconds(to)
   if (from_secs > to_secs) stop("'from' must be lower than 'to'.")
 
+  # times after midnight written as early morning times are fixed first, so
+  # they are compared with the time of day as they should be. the returned
+  # feed keeps them fixed
+
+  gtfs <- fix_overnight_in_call(gtfs, frequencies = TRUE)
+
   # if the gtfs contains a frequencies table, it's important to keep the
   # stop_times of the trips listed in there intact.
   # this is because their stop_times are only templates, and the departure and

R/fix_gtfs.R

@@ -1,10 +1,10 @@
 #' Fix common problems in GTFS feeds
 #'
 #' Cleans a GTFS feed in one call, by removing duplicated entries and unused
-#' ids, optionally capping the speed of trips, filling blank stop times,
-#' optionally setting a minimum dwell time at intermediate stops and
-#' recalculating the distances travelled along shapes (replacing any existing
-#' ones).
+#' ids, fixing times after midnight written as early morning times, optionally
+#' capping the speed of trips, filling blank stop times, optionally setting a
+#' minimum dwell time at intermediate stops and recalculating the distances
+#' travelled along shapes (replacing any existing ones).
 #'
 #' @template gtfs
 #' @param max_speed A number. The maximum speed of trips, in the unit given by
@@ -18,6 +18,12 @@
 #' @param dwell_time_unit A string representing the time unit in which
 #'   `min_dwell_time` is given. One of `"s"` (seconds, the default), `"min"`
 #'   (minutes), `"h"` (hours) or `"d"` (days).
+#' @param overwrite_stop_times A logical. If `FALSE` (the default), only blank
+#'   stop times are filled. If `TRUE`, the times of every stop between the
+#'   first and the last stop of each trip are replaced by times interpolated
+#'   over the distance travelled, keeping their dwell times (see the
+#'   `overwrite` and `keep_dwell_times` arguments of
+#'   [interpolate_stop_times()]).
 #'
 #' @return The fixed GTFS object.
 #'
@@ -29,17 +35,25 @@
 #'    entries of every table that no remaining trip uses (see its
 #'    documentation for the exceptions). Note that this removes data from the
 #'    feed;
-#' 3. [set_max_speed()], only if `max_speed` is given, slows down the parts of
+#' 3. [fix_overnight_times()] rewrites the times after midnight written as
+#'    early morning times (e.g. `"00:30:00"` instead of `"24:30:00"`), so the
+#'    next steps work on these trips too. Only times that go back by more
+#'    than 12 hours are taken as times after midnight, and the times that
+#'    still go back are left unchanged, with a warning;
+#' 4. [set_max_speed()], only if `max_speed` is given, slows down the parts of
 #'    trips that are faster than `max_speed`. Only spans between stops with
-#'    known times are capped, and spans that end before they start (e.g.
-#'    times after midnight written as `"00:30:00"`) are left unchanged, with
-#'    a warning;
-#' 4. [interpolate_stop_times()] fills blank stop times from the times of the
+#'    known times are capped, and spans that still end before they start are
+#'    left unchanged, with a warning;
+#' 5. [interpolate_stop_times()] fills blank stop times from the times of the
 #'    stops around them. Speeds are capped before interpolating, so the filled
 #'    times follow the capped times. Interpolated times are rounded to the
 #'    second. Stops whose times can't be interpolated are left blank, with a
-#'    warning;
-#' 5. [set_min_dwell_time()], only if `min_dwell_time` is given, delays the
+#'    warning. With `overwrite_stop_times = TRUE`, the times of every
+#'    intermediate stop are replaced instead, from each trip's first and last
+#'    times, keeping the stops' dwell times. The trips' durations still
+#'    follow the capped times, and their travel time is spread evenly over
+#'    the distance;
+#' 6. [set_min_dwell_time()], only if `min_dwell_time` is given, delays the
 #'    departure from each intermediate stop whose dwell time is shorter than
 #'    `min_dwell_time`, so that vehicles stay there for at least that long.
 #'    The minimum is rounded to the second. The later times of each changed
@@ -48,10 +62,8 @@
 #'    filling blank times, so the filled stops also get the minimum. Only the
 #'    stops between the first and the last stop with times of each trip get
 #'    the minimum, so trips keep their start times. Stops with negative dwell
-#'    times (departures earlier than arrivals) also get it, see
-#'    [set_min_dwell_time()] for times after midnight written as
-#'    `"00:30:00"`;
-#' 6. [interpolate_shape_dist_traveled()], only if the feed has a `shapes`
+#'    times (departures earlier than arrivals) also get it;
+#' 7. [interpolate_shape_dist_traveled()], only if the feed has a `shapes`
 #'    table and `trips` has a `shape_id` column, replaces the
 #'    `shape_dist_traveled` columns of `stop_times` and `shapes` with the
 #'    distances, in meters, from the start of each shape. Existing values,
@@ -64,7 +76,8 @@
 #' above, so some warnings (e.g. about trips not linked to a shape) may be
 #' raised more than once.
 #'
-#' @seealso [remove_duplicates()], [remove_unused_ids()], [set_max_speed()],
+#' @seealso [remove_duplicates()], [remove_unused_ids()],
+#'   [fix_overnight_times()], [set_max_speed()],
 #'   [interpolate_stop_times()], [set_min_dwell_time()],
 #'   [interpolate_shape_dist_traveled()]
 #'
@@ -100,7 +113,8 @@ fix_gtfs <- function(
   max_speed = NULL,
   speed_unit = "km/h",
   min_dwell_time = NULL,
-  dwell_time_unit = "s"
+  dwell_time_unit = "s",
+  overwrite_stop_times = FALSE
 ) {
   gtfs <- assert_and_assign_gtfs_object(gtfs)
   checkmate::assert_number(
@@ -128,15 +142,19 @@ fix_gtfs <- function(
     upper = max_time_secs / dwell_unit_factors[[dwell_time_unit]],
     null.ok = TRUE
   )
+  checkmate::assert_logical(overwrite_stop_times, len = 1, any.missing = FALSE)
 
   # remove duplicates first, so the other steps don't see repeated rows
-
   gtfs_fixed <- remove_duplicates(gtfs)
 
   # remove unused ids
-
   gtfs_fixed <- remove_unused_ids(gtfs_fixed)
 
+  # fix times after midnight before the steps that use the times, which skip
+  # the spans that go back
+
+  gtfs_fixed <- fix_overnight_times(gtfs_fixed)
+
   # cap max speed. this is done before interpolating, so the interpolated
   # times follow the capped times
 
@@ -148,9 +166,12 @@ fix_gtfs <- function(
     )
   }
 
-  # interpolate stop_times
+  # fill blank stop times, or overwrite the times of intermediate stops
 
-  gtfs_fixed <- interpolate_stop_times(gtfs_fixed)
+  gtfs_fixed <- interpolate_stop_times(
+    gtfs = gtfs_fixed,
+    overwrite = overwrite_stop_times
+  )
 
   # set a minimum dwell time at intermediate stops. this is done after
   # interpolating, so the filled stops also get it. 'stop_times' is not the