Home > Mobile >  How to combine multiple async tasks in Python?
How to combine multiple async tasks in Python?

Time:01-11

As a newbie I got stuck in here and would appreciate if you could help.

  1. Is there a way to shorten / combine the following async tasks?
  2. How can I repeat these set of tasks every n seconds forever?

Thank you in advance.

def get_tasks1(session1):
    task1 = []
    for symbol in symbols:
        task1.append(asyncio.create_task(session1.get(url1.format(symbol))))
    return task1

async def get_symbols1():
    async with aiohttp.ClientSession() as session1:
        task1 = get_tasks1(session1)
        responses = await asyncio.gather(*task1)
        for response in responses:
            results1.append(await response.json())

def get_tasks2(session2):
    task2 = []
    for symbol in symbols:
        task2.append(asyncio.create_task(session2.get(url2.format(symbol))))
    return task2

async def get_symbols2():
    async with aiohttp.ClientSession() as session2:
        task2 = get_tasks2(session2)
        responses = await asyncio.gather(*task2)
        for response in responses:
            results2.append(await response.json())

asyncio.run(get_symbols1())
asyncio.run(get_symbols2())

with open("Output.txt", "w") as text_file:
    text_file.write('%r\n%r\n' % (results1, results2))

CodePudding user response:

If you question is practical(you want to solve task rather than learn async deeply) I recommend:

  1. Use grequests package, which is based on popular requests package to query urls.

  2. If in future you want to perform some operations on received data, you probably will want to use Scrapy

  3. To run operation every N seconds:

from time import sleep

def operations(): 
    asyncio.run(get_symbols1())
    asyncio.run(get_symbols2())

    with open("Output.txt", "w") as text_file:
        text_file.write('%r\n%r\n' % (results1, results2))

while True:
    sleep(60) # 60 seconds
    operations()

But it is not safe to put this on scripts itself(possible memory leaking, process could be terminated by OS...), so consider using cron(if periods is more than 1 minute) or any other external tool, that will run your script periodically.

  •  Tags:  
  • Related