python - How to give name to each node in celery -


we using celery our flask application. there many queues our various tasks , use supervisord run queues. use cloudamqp broker.

example supervisor configuration below:

[program:my-queue] command=/home/ubuntu/opt/proect/venv/bin/celery -a async_runner worker -q my_queue --loglevel=info --without-gossip --without-mingle --autoscale=1,1  -c 1 environment=pythonpath=/home/ubuntu/opt/project/,production_environment=true directory=/home/ubuntu/opt/project/app  process_name = %(program_name)s_%(process_num)02d user=ubuntu numprocs=2 autostart=true autorestart=true startsecs=10 stopwaitsecs = 600 priority=998 

we following error , queues stop.

[2017-04-06 12:43:06,759: warning/mainprocess] /home/ubuntu/opt/project/venv/local/lib/python2.7/site-packages/kombu/pidbox.py:75: userwarning: node named celery@ip-xxx-yy-yy-yy using process mailbox!  maybe forgot shutdown other node or did not properly? or if meant start multiple nodes on same host please make sure give each node unique node name!    warnings.warn(w_pidbox_in_use.format(node=self)) 

question: how give names each node?

when run celery -a async_runner status gives 5 nodes online message.

celery@ip-ip_value_here-: ok celery@ip-ip_value_here-: ok celery@ip-ip_value_here-: ok celery@ip-ip_value_here-: ok celery@ip-ip_value_here-: ok  5 nodes online. 

here how run 1 celery worker each queue in our company. entry point contains this:

echo queues: ${queues}  # comma separated list of queue names export num_queues=$(python -c "print len('$queues'.split(','))") echo num_queues: ${num_queues}  supervisord & 

the supervisor config file has this:

[program:worker] command=/path/to/celery_worker.sh %(process_num)s # supervisor uses special env_ prefix environment variable set above numprocs=%(env_num_queues)s 

and celery_worker.sh contains like:

queue=$(python -c "print '$queues'.split(',')[$1]")  # $1 = process_num above celery worker -n "worker.${queue}" -q ${queue}  # -n sets name of worker 

Comments