Skip to content

Pass Async::Semaphore to replace Async::Idler, in order to control the exact concurrency limit. #4

Description

@shaokun

Async::Job::Processor::Redis::Server#initialize(...parent: nil), by default, the parent is nil, hence it's set to Async::Idler.new.
But I want to be able to control the exact number of concurrency, so I am trying to pass Async::Semaphore to parent.

I do it in this way:

# config/initializers/async_job.rb

Rails.application.configure do
  # Create a queue for the "default" backend:
  config.async_job.define_queue "default" do
    dequeue Async::Job::Processor::Redis, parent: Async::Semaphore.new(10)
  end
end

It's passed to the parent parameter, but the Server still dequeue as many jobs as possible without taking the concurrency limit into account.
From code:

def start!
  return false if @task
  
  @task = true
  
  @parent.async(transient: true, annotation: self.class.name) do |task|
    @task = task
    
    while true
      self.dequeue(task)
    end
  ensure
    @task = nil
  end
end

It looks like that the infinite loop is within the @parent.async, hence the concurrency limit doesn't work at all.
Should it be?

while true
  @parent.async(transient: true, annotation: self.class.name) do |task| do
    self.dequeue
  end
end

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions