diff --git a/CHANGELOG.md b/CHANGELOG.md index fa8bca4..631b0d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,12 @@ gem but are what an operator runs against their own image. ## next / unreleased +### HotCell::Client + +#### Fixed + +* On macOS, a call to a full cell reports `capacity` instead of occasionally reporting `unavailable`. A full cell answers and closes the connection without reading the request, so the client's send fails. The client already ignored that failure as `EPIPE` or `ECONNRESET` and read the answer, but macOS sometimes raises `ENOTCONN` instead. (#110) + ### Yabeda::HotCell #### Fixed diff --git a/hotcell-client/lib/hot_cell/transport.rb b/hotcell-client/lib/hot_cell/transport.rb index 28d7715..1fadfae 100644 --- a/hotcell-client/lib/hot_cell/transport.rb +++ b/hotcell-client/lib/hot_cell/transport.rb @@ -13,9 +13,9 @@ class Socket # `timeout` covers the answer and not the connection. On Linux a blocking `connect` to a Unix socket waits # in the kernel while the listener's backlog is full, which is where a supervisor that stops calling # `accept` leaves it. `UNIXSocket.new` does not wait: Ruby opens every socket non-blocking and takes the - # kernel's EAGAIN for a connection in progress, so it returns a socket that never connected, the send - # fails with ENOTCONN, and the caller gets `unavailable` at once. That is Ruby's behavior rather than - # this gem's, so ControlTimeoutTest holds it. + # kernel's EAGAIN for a connection in progress, so it returns a socket that never connected. The send + # fails with ENOTCONN, which `deliver` ignores, then the read fails with EINVAL, and the caller gets + # `unavailable` at once. That is Ruby's behavior rather than this gem's, so ControlTimeoutTest holds it. # # If Ruby ever starts waiting, `connect_nonblock` plus `wait_writable` against a deadline would bound the # call only by spinning: on Linux an unconnected Unix socket is always writable, so it retries EAGAIN until @@ -39,9 +39,12 @@ def call(cell, line, descriptors, socket: cell.work_socket, timeout: cell.timeou # then fail while the answer is already waiting on the socket. So this ignores the failed send, and # `receive` reads the answer. If the peer closed without an answer, `receive` reports that the # supervisor is gone. + # + # Linux fails that send with EPIPE or ECONNRESET. macOS usually does too, but answers ENOTCONN when the + # close lands while its kernel is copying the request in, between checking the socket and queueing. def deliver(connection, line, descriptors) connection.send_message line, descriptors: descriptors - rescue Errno::EPIPE, Errno::ECONNRESET + rescue Errno::EPIPE, Errno::ECONNRESET, Errno::ENOTCONN nil end diff --git a/hotcell-client/test/classification_test.rb b/hotcell-client/test/classification_test.rb index c217807..695f911 100644 --- a/hotcell-client/test/classification_test.rb +++ b/hotcell-client/test/classification_test.rb @@ -244,27 +244,53 @@ def test_an_answer_that_cannot_be_read_is_a_verdict_rather_than_an_escaping_exce # then fails, but the answer is already on its socket. The answer is the verdict. A request larger than the # socket buffer makes the send fail every time. def test_an_answer_that_arrives_before_the_request_is_sent_is_still_the_verdict - refusal = HotCell::Response.failed(HotCell::Failure.new(code: "capacity", message: "the queue is full at 8")).to_line - - Dir.mktmpdir "hotcell-refusing" do |directory| - path = File.join(directory, "work.sock") - work = UNIXServer.new(path) - refuser = Thread.new do - connection = HotCell::Connection.new(work.accept) - connection.write_line refusal - connection.close - end + response = call_a_full_cell + + assert_equal "capacity", response.failure.code, response.failure.to_s + end - response = HotCell::Transport::Socket.new.call(nil, "x" * (4 * 1024 * 1024), [], socket: path, timeout: 5) + # macOS reports the same early close as ENOTCONN rather than EPIPE when the close lands while its kernel is + # copying the request in. That window is narrow, so the test above passes there almost every time. + def test_an_answer_that_arrives_before_a_send_refused_as_not_connected_is_still_the_verdict + failing_sends_with Errno::ENOTCONN do + response = call_a_full_cell assert_equal "capacity", response.failure.code, response.failure.to_s - ensure - refuser&.join - work&.close end end private + def call_a_full_cell + refusal = HotCell::Response.failed(HotCell::Failure.new(code: "capacity", message: "the queue is full at 8")).to_line + + Dir.mktmpdir "hotcell-refusing" do |directory| + path = File.join(directory, "work.sock") + work = UNIXServer.new(path) + refuser = Thread.new do + connection = HotCell::Connection.new(work.accept) + connection.write_line refusal + connection.close + end + + HotCell::Transport::Socket.new.call(nil, "x" * (4 * 1024 * 1024), [], socket: path, timeout: 5) + ensure + refuser&.join + work&.close + end + end + + # The send fails only once the cell's answer is on the socket, as it does when the cell has answered and closed. + def failing_sends_with(error) + original = HotCell::Connection.instance_method(:send_message) + HotCell::Connection.define_method(:send_message) do |*| + socket.wait_readable 5 + raise error, "sendmsg(2)" + end + yield + ensure + HotCell::Connection.define_method(:send_message, original) + end + # A cell whose supervisor answers one fixed line and closes, so the client reads exactly these bytes. # # It reads the request before answering, the way a real supervisor does. Answering without reading