Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not stop worker #78

Open
sunra opened this issue Dec 11, 2013 · 8 comments
Open

Can not stop worker #78

sunra opened this issue Dec 11, 2013 · 8 comments

Comments

@sunra
Copy link

sunra commented Dec 11, 2013

comand:
app/console bcc:resque:worker-stop -a

displayed:
Stopping 00000.net:24674:*...

but process stil working...

@mrbase
Copy link
Contributor

mrbase commented Jan 13, 2014

i have no problems starting and stopping workers, could you add a little more context ?

@slimus
Copy link

slimus commented Dec 23, 2015

I have some problem (in latest Mac os x and ubuntu 14.04.3 LTS)

posix_kill($pid, 3) return true, posix_get_last_error() return 0

If I replace 3 (SIGQUIT) to 9 (SIGKILL) all my workers was killed.
Which of context do you need? My steps for reproduce (from your docs):
0. symfony new test

  1. composer require bcc/resque-bundle
  2. "Add BCCResqueBundle to your application kernel"
  3. run from console: ./app/console bcc:resque:worker-start "*"
  4. run : ./app/console bcc:resque:worker-stop -a
  5. I see current processes: ps aux | grep resque and I look my worker still run

Also for information: if I run worker in foreground (--foreground) I can stop worker.
May be problem with nohup? If you need more information please let me know.

@PhilETaylor
Copy link

I have this issue on Mac, 27 July 2016... seems to still be a problem :(

@danhunsaker
Copy link

danhunsaker commented Jul 27, 2016

From the PHP-Resque Readme:

Signals also work on supported platforms exactly as in the Ruby version of Resque:

  • QUIT - Wait for job to finish processing then exit
  • TERM / INT - Immediately kill job then exit
  • USR1 - Immediately kill job but don't exit
  • USR2 - Pause worker, no new jobs will be processed
  • CONT - Resume worker.

Your workers aren't quitting because their jobs aren't done yet. If you must kill your workers immediately, use SIGINT instead (SIGTERM may result in an unclean shutdown, which is probably not what you want).

@PhilETaylor
Copy link

To be clear... my workers are not running jobs... there is nothing in the queue.... I issue command:

php app/console bcc:resque:worker-stop -a

It states its stopping them... but doesn't.

I still see them in the activity monitor and issuing the same stop command states that its stopping the same worker (same pid id) but again doesnt, even if I sudo.

@danhunsaker
Copy link

Aha. That explains a bit. You're starting multiple workers at once, yes? The parent process (the single PID the stop command lists) doesn't stick around to manage the workers it forks - it simply exits, meaning there's no process to catch the signals in the first place. This is a bug in a file which was originally intended to be a sample script, so wasn't deemed important to actually fix at the time. I've had a PR in to fix it for a few years, but it hasn't been merged upstream. The workaround for now is to only start single workers at a time, rather than entire pools as you really want.

Short of the BCCResque team switching their dependency to my PHP-Resque fork instead of the original, it doesn't look like there will be an actual fix anytime soon.

@PhilETaylor
Copy link

I think I have worked this out now.

If you start a load of workers... and then stop a load of workers - it works.

if you start a load of workers - flush the redis db (for some reason) - and then use the command to stop a load of workers - it doesnt stop the workers... The PHP processes live on and need to be manually killed.

@danhunsaker
Copy link

That sounds like the BCCResque guys added a bit of extra logic to clear out workers despite no management process. Each worker stores a small handful of keys in Redis to keep track of both their existence and their current status. Flushing the DB removes those keys, leaving the bundle script no way to determine which processes to kill. Of course, these keys are recreated by each worker when it pulls a job from the queue, so any workers that have done work since the flush will be killable again.

Not as elegant a solution, but a decent workaround while waiting for the upstream to progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants