1

here is idea/workflow:

I have bunch of network devices that I need to communicate with.

a. client (flask/python) sends request: "mac-table sw1" to Celery.

b. celery look at available workers/tasks and if no workers responsible for sw1 exists - it will create one and pass job to it.

all consequent requests for sw1 will be forwarded to existing worker automatically (thus eliminating establishing sessions for every requests and limiting concurrent sessions to the device)

c. if worker idle for some time, it close connection to device and exit.

questions: is Celery good for this workflow ? do you know any example of similar workflow I can get ideas from.

Thank you!

Vitaly Nikolaev
  • 121
  • 1
  • 4

2 Answers2

0

If I understood well, what you want to achieve can be accomplished by dynamically creating a queue for particular job, picking one of the available workers, and subscribing it to that queue. It also requres some cleanup task that would remove unused queues (it can be a task that runs every N minutes that inspects queues, checks whether there is anything running there, if not unsubscribe the worker from that queue).

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
-1

Yes you can.

  • Call your celery task from Flask endpoint by passing the required task_id (Start by querying the task id, if it doesn't exist, call the task).
  • You can establish time limits for Tasks as mentioned here: Setting Time Limit on specific task with celery
@app.route('/task', methods=['POST'])
def upload(params):
    # query, validate and create new task
    do_task.apply_async(do_task, args=(params,), task_id="mac-table-sw1")

@celery.task(bind=True)
def do_task(self, params):
    pass

@app.route('/status/<task_id>')
def taskstatus(task_id):
    task = extract_keywords_task.AsyncResult("mac-table-sw1")
    return task.state
Abhilash
  • 2,026
  • 17
  • 21
  • Thank you, but, when I start my task with ID mac-table-sw1, in task I will create for example netmiko object (ssh connection to the switch), use it for current task, return result and celery will end task with ID mac-table-sw1 and delete netmiko object. Next time I run my task it will create NEW instance and I will have to do connection part again. – Vitaly Nikolaev Sep 04 '20 at 00:10