Fix a macOS call to a full cell reporting unavailable instead of capacity

CHANGELOG.md

@@ -15,6 +15,12 @@ gem but are what an operator runs against their own image.
 
 ## next / unreleased
 
+### HotCell::Client
+
+#### Fixed
+
+* On macOS, a call to a full cell reports `capacity` instead of occasionally reporting `unavailable`. A full cell answers and closes the connection without reading the request, so the client's send fails. The client already ignored that failure as `EPIPE` or `ECONNRESET` and read the answer, but macOS sometimes raises `ENOTCONN` instead. (#110)
+
 ### Yabeda::HotCell
 
 #### Fixed

hotcell-client/lib/hot_cell/transport.rb

@@ -13,9 +13,9 @@ class Socket
       # `timeout` covers the answer and not the connection. On Linux a blocking `connect` to a Unix socket waits
       # in the kernel while the listener's backlog is full, which is where a supervisor that stops calling
       # `accept` leaves it. `UNIXSocket.new` does not wait: Ruby opens every socket non-blocking and takes the
-      # kernel's EAGAIN for a connection in progress, so it returns a socket that never connected, the send
-      # fails with ENOTCONN, and the caller gets `unavailable` at once. That is Ruby's behavior rather than
-      # this gem's, so ControlTimeoutTest holds it.
+      # kernel's EAGAIN for a connection in progress, so it returns a socket that never connected. The send
+      # fails with ENOTCONN, which `deliver` ignores, then the read fails with EINVAL, and the caller gets
+      # `unavailable` at once. That is Ruby's behavior rather than this gem's, so ControlTimeoutTest holds it.
       #
       # If Ruby ever starts waiting, `connect_nonblock` plus `wait_writable` against a deadline would bound the
       # call only by spinning: on Linux an unconnected Unix socket is always writable, so it retries EAGAIN until
@@ -39,9 +39,12 @@ def call(cell, line, descriptors, socket: cell.work_socket, timeout: cell.timeou
         # then fail while the answer is already waiting on the socket. So this ignores the failed send, and
         # `receive` reads the answer. If the peer closed without an answer, `receive` reports that the
         # supervisor is gone.
+        #
+        # Linux fails that send with EPIPE or ECONNRESET. macOS usually does too, but answers ENOTCONN when the
+        # close lands while its kernel is copying the request in, between checking the socket and queueing.
         def deliver(connection, line, descriptors)
           connection.send_message line, descriptors: descriptors
-        rescue Errno::EPIPE, Errno::ECONNRESET
+        rescue Errno::EPIPE, Errno::ECONNRESET, Errno::ENOTCONN
           nil
         end
 

hotcell-client/test/classification_test.rb

@@ -244,27 +244,53 @@ def test_an_answer_that_cannot_be_read_is_a_verdict_rather_than_an_escaping_exce
   # then fails, but the answer is already on its socket. The answer is the verdict. A request larger than the
   # socket buffer makes the send fail every time.
   def test_an_answer_that_arrives_before_the_request_is_sent_is_still_the_verdict
-    refusal = HotCell::Response.failed(HotCell::Failure.new(code: "capacity", message: "the queue is full at 8")).to_line
-
-    Dir.mktmpdir "hotcell-refusing" do |directory|
-      path = File.join(directory, "work.sock")
-      work = UNIXServer.new(path)
-      refuser = Thread.new do
-        connection = HotCell::Connection.new(work.accept)
-        connection.write_line refusal
-        connection.close
-      end
+    response = call_a_full_cell
+
+    assert_equal "capacity", response.failure.code, response.failure.to_s
+  end
 
-      response = HotCell::Transport::Socket.new.call(nil, "x" * (4 * 1024 * 1024), [], socket: path, timeout: 5)
+  # macOS reports the same early close as ENOTCONN rather than EPIPE when the close lands while its kernel is
+  # copying the request in. That window is narrow, so the test above passes there almost every time.
+  def test_an_answer_that_arrives_before_a_send_refused_as_not_connected_is_still_the_verdict
+    failing_sends_with Errno::ENOTCONN do
+      response = call_a_full_cell
 
       assert_equal "capacity", response.failure.code, response.failure.to_s
-    ensure
-      refuser&.join
-      work&.close
     end
   end
 
   private
+    def call_a_full_cell
+      refusal = HotCell::Response.failed(HotCell::Failure.new(code: "capacity", message: "the queue is full at 8")).to_line
+
+      Dir.mktmpdir "hotcell-refusing" do |directory|
+        path = File.join(directory, "work.sock")
+        work = UNIXServer.new(path)
+        refuser = Thread.new do
+          connection = HotCell::Connection.new(work.accept)
+          connection.write_line refusal
+          connection.close
+        end
+
+        HotCell::Transport::Socket.new.call(nil, "x" * (4 * 1024 * 1024), [], socket: path, timeout: 5)
+      ensure
+        refuser&.join
+        work&.close
+      end
+    end
+
+    # The send fails only once the cell's answer is on the socket, as it does when the cell has answered and closed.
+    def failing_sends_with(error)
+      original = HotCell::Connection.instance_method(:send_message)
+      HotCell::Connection.define_method(:send_message) do |*|
+        socket.wait_readable 5
+        raise error, "sendmsg(2)"
+      end
+      yield
+    ensure
+      HotCell::Connection.define_method(:send_message, original)
+    end
+
     # A cell whose supervisor answers one fixed line and closes, so the client reads exactly these bytes.
     #
     # It reads the request before answering, the way a real supervisor does. Answering without reading