Repository navigation
E52: latency between env client and bridge changes the time, not the weights - #118
Merged
Merged
Conversation
…latency between env client and bridge change the weights, and does it cost two one-way delays per step
…e, not the weights; up to 25 ms each way, SB3 ends on its in-process weights All 15 runs (seeds 0-2 x direct, 0, 1, 5, 25 ms) end on E50 in-process weights. Each step costs 2D plus 0.1-3.1 ms; P2 (band 2D to 2D + 3 ms) is falsified at 25 ms by 0.02-0.10 ms. After the runs, proxy_rtt.py shows the proxy itself accounts for about 1 ms; the rest grows with D and is not explained.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The question
The paper claims that latency between environment and trainer slows a run without changing what it collects. On one machine, with latency added between the env client and the bridge, does training end on the same weights? And does the time grow by two one-way delays per step?
The answer
The weights are the same at every delay, so P1 holds. SB3 PPO trained 16 Pendulum environments behind
delay_proxy.py, three seeds × {direct, 0, 1, 5, 25 ms each way}. All 15 runs end on E50's in-process weights for their seed. At 25 ms a run takes 22× as long and ends on the same bytes.The time came in a little above the band, so P2 is falsified at 25 ms. Each step cost 2D plus 0.1–3.1 ms. The registered band was 2D to 2D + 3 ms: inside it at 1 and 5 ms, and above it at 25 ms by 0.02–0.10 ms on every seed.
After the runs,
proxy_rtt.pymeasured the proxy alone. It accounts for about 1 ms: its two sleeps are rounded up inepoll_wait, as E47 found. The rest grows with D and is reported as not explained.