Is there any library to support multiple asyncio event loop, one loop per thread/process

January 13, 2021 at 21:11


I am looking for an asyncio library to support multiple event loop, each per thread/process. Typical use case should be: Asyncio web crawler, in which there are multiple threads/processes, each runs an independent event loop to crawl remote data and does some processing. Given that the data is small enough, but there are a lot of them and time to process data is roughly equivalent to time to read it from server.

I thought of using separate executor poll to process data, so the asyncio event loop only needs to take care of reading data, but because the processing job is similar to executor poll, so using threaded executor poll may be hit by GIL. If using process executor poll, passing data between process via `multiprocessing `pipe/queue causes extra over head. More over, I want to have some interaction between event loops, like submitting a task to different event loops and await for its response. So I think having multiple processes, each process runs one event loops should fit the bill.

The closest one I found is Pulsar, but seems inactive for now. Another tentative one is Ray, but it's quite bigger than whet I need.

TLDR I would like to have multiple event loops, each one runs on a separated thread/process and can communicate between each other. Please let me know if there is anything like this.