Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add worker dequeue_interval attribute to control the dequeue poll rate #62

Closed
wants to merge 2 commits into from

Conversation

scottw
Copy link

@scottw scottw commented Dec 14, 2017

Summary

This change adds a new 'dequeue_interval' attribute to the worker that allows the poll time to be configurable.

Motivation

I was needing a faster poll time than the default 5s.

References

None I am aware of.

@kraih
Copy link
Member

kraih commented Dec 14, 2017

Didn't we discuss this just recently and concluded that it make no sense?

@scottw
Copy link
Author

scottw commented Dec 14, 2017

If there was discussion, I missed it. I'll search the archives to review—was this mailing list or irc?

@kraih
Copy link
Member

kraih commented Dec 14, 2017

Minion::Backend::Pg uses a pub/sub channel, so there is no delay whatsoever and the interval does nothing. Minion::Backend::SQLite had a bug that caused longer delays, but that has been fixed since and the interval is mostly irrelevant there too. Why do you need it?

@scottw
Copy link
Author

scottw commented Dec 14, 2017

I'm using the Postgres backend. My particular case involves adding many jobs into the queue at once, each with a 1s delay from the previous one—I want these to run no faster than 1 per second.

I have 1 worker running. The worker receives the first ready job and runs it, but the next time it polls for more work is 5s later, and so it takes the next 4 "ready" jobs. So on average, the worker is running 1 job per second, but in practice it's 5 jobs in a burst every 5s.

I'm already using a guard for rate limiting (I cannot have more than 100 jobs of a particular type at once). I'm open to other suggestion. I'm also happy to extract some generic sample code to illustrate my use case.

@kraih
Copy link
Member

kraih commented Dec 14, 2017

That doesn't sound right. Minion::Backend::Pg does not sleep for those 5 seconds, it will listen for pub/sub notifications and wake up as soon as a job has been enqueued. Maybe you're doing something else wrong or you're hitting a bug, but this patch is not the right solution.

@kraih kraih closed this Dec 14, 2017
@Grinnz
Copy link
Contributor

Grinnz commented Dec 14, 2017

I think the problem that's occurring is that all of the jobs are enqueued at once with delays, so when the first job completes there are no more enqueues occurring to send pubsub notifications. Thus it receives no notifications to wake up before the 5 seconds are up. In other words, the pubsub interruption is only really useful when enqueueing jobs to occur immediately, delayed jobs don't trigger it.

@scottw
Copy link
Author

scottw commented Dec 14, 2017

@Grinnz, I think you're right. All my jobs are enqueued fast (sometimes thousands) with delays to meter them. There may be an initial "pub" when they're enqueued, but all subsequent jobs are handled at 5s intervals. This PR fixes my symptom, but perhaps something in the pub/sub system needs to watch for the delayed jobs. I don't know Minion's architecture well enough to understand what's supposed to happen in this situation.

@kraih
Copy link
Member

kraih commented Dec 14, 2017

That does make sense, not sure there's a good solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants