How to speedup FTX API call using asyncio?-CodePudding

I am running a program that makes the same API call, for each row in a dataframe. Since it is taking quite a long time I decided to try to learn and implement an async version with asyincio.

I'd split the dataframe in "N" (3 in this case) smaller dataframes, and for each one of those I'd create a coroutine that would be gathered and awaited by the main routine.

Here's what I've tried:

async def get_next_funding(df):
for idx in df.index:
    predicted_rate = ftx.get_future_stats(df.loc[idx]['name'])['nextFundingRate']
    df.at[idx, 'nextFundingRate'] = predicted_rate
    await asyncio.sleep(0.01)
return df


async def await_for_df(df_):
    # await é quem promete que a funcao vai ser executada, dando a mesma
    # para o event loop
    await get_next_funding(df_)
    return df_


async def main():
    # array_split returns a LIST of dataframes: [df1, ..., dfN]
    dfs = np.array_split(ftx.df, 3)
    # Could I use [await_for_df(dfs[x]) for x in dfs] ?
    results = await asyncio.gather(await_for_df(dfs[0]), await_for_df(dfs[1]), await_for_df(dfs[2]))

loop = asyncio.get_event_loop()
start = time.perf_counter()
asyncio.run(main())
end = time.perf_counter()
print('process finished in {} seconds'.format(end - start))

It works but it does not seem to run in parallel because it takes the same amount of time as my sync code. I have the feeling that the function ftx.get_future_stats() might be blocking eveything. Such function is a standard API call ( https://docs.ftx.com/#get-future-stats ).

What am I missing ? Thank you gents for any knowledge insigts.

Piero

CodePudding user response：

So, aparently I needed to execute the blocking function async get_next_funding(df) as a SYNC function inside a loop.run_in_executor() since the blocking fcn was not a async type.

Thank @gold_cy for the answer!

There's the modified code:

def get_next_funding(df):
    for idx in df.index:
        predicted_rate = ftx.get_future_stats(df.loc[idx]['name'])['nextFundingRate']
        df.at[idx, 'nextFundingRate'] = predicted_rate
    return df


async def await_for_df(df_):

    loop = asyncio.get_running_loop()
    r = await loop.run_in_executor(None, get_next_funding, df_)
    return r