Follow-up for discussion from #122
AFAICT: We're not actually using the otel trace export ... we export to localhost, so it's effectively a no-op currently. ateom doesn't actually use the network meaningfully.
While the traces would be nice to have, we could have atelet relay these (e.g. we could use another unix socket for ateom traces => atelet), which is in-line with how we're trending for scheduling (aggregate at the atelet, avoid every worker talking directly to ate-apiserver) but repeated for tracing and the trace collector.
So far there doesn't seem to be any other use-case for inbound connections to the ateom.
I'm not sure there's a strong case for ateom to intrude on the worker's network at all, I could see potentially streaming snapshots but I think we're moving away from always pushing these over the network anyhow (#119, #227) so we'll probably have some local cache in atelet that we need to pull through anyhow.
It would be nice to just forward all traffic on all protocols from the pod IP to the interior sandbox and avoid users having to configure explicit actor ports, and more importantly avoid conflicts with any ports actors might desire.
I feel like this would've made more sense if we didn't think we'd be writing snapshots to local disk commonly (to avoid too much traffic on the nic) and if we thought we were going to eliminate atelet (#128 , currently I disagree and I think atelet will be critical for aggregating both scheduling and images / snapshots / ...), but even then debugging it may be ... confusing.
cc Tim Hockin (@thockin) Bowei Du (@bowei) Eitan Yarmush (@EItanya) Taahir Ahmed (@ahmedtd)
Follow-up for discussion from #122
AFAICT: We're not actually using the otel trace export ... we export to localhost, so it's effectively a no-op currently. ateom doesn't actually use the network meaningfully.
While the traces would be nice to have, we could have atelet relay these (e.g. we could use another unix socket for ateom traces => atelet), which is in-line with how we're trending for scheduling (aggregate at the atelet, avoid every worker talking directly to ate-apiserver) but repeated for tracing and the trace collector.
So far there doesn't seem to be any other use-case for inbound connections to the ateom.
I'm not sure there's a strong case for ateom to intrude on the worker's network at all, I could see potentially streaming snapshots but I think we're moving away from always pushing these over the network anyhow (#119, #227) so we'll probably have some local cache in atelet that we need to pull through anyhow.
It would be nice to just forward all traffic on all protocols from the pod IP to the interior sandbox and avoid users having to configure explicit actor ports, and more importantly avoid conflicts with any ports actors might desire.
I feel like this would've made more sense if we didn't think we'd be writing snapshots to local disk commonly (to avoid too much traffic on the nic) and if we thought we were going to eliminate atelet (#128 , currently I disagree and I think atelet will be critical for aggregating both scheduling and images / snapshots / ...), but even then debugging it may be ... confusing.
cc Tim Hockin (@thockin) Bowei Du (@bowei) Eitan Yarmush (@EItanya) Taahir Ahmed (@ahmedtd)