Simple Multiprocessing with QuasiQueue

My third open source project this week is one of my favorites. QuasiQueue is a MultiProcessing library for Python that makes it super easy to have long running MultiProcess jobs. QuasiQueue handles process creation and cleanup, signal management, cross process communication, and all the other garbage that makes people hate dealing with multiprocessing.

I started on this library a couple of years ago and have pulled it out to use as the basis for multiple projects. QuasiQueue works by splitting the work into two components- the main process whose job it is to feed a Queue with work, and then read processes that take work off of the Queue to run. All the developers have to do is create two functions, a Reader and a Writer, and QuasiQueue will handle the rest.

QuasiQueue can be used for a number of purposes-

• Server/Client communication where the Writer waits for requests to come in and then passes the incoming sockets to the Readers. You could build an special purpose multiprocess HTTP server (or any protocol) this way.
• Dynamic Crawlers to constantly crawl and update data. I used this to crawl my Twitter followers and build network graphs to dynamically generate lists, as one really silly example.
• Local data processing, such as running image processing.

QuasiQueue is meant to be extremely simple- writing programs that utilize QuasiQueue is as simple as defining two functions. The output of the writer function will end up feeding the reader function. The only limitations on what can be sent between them is that python has to be able to "pickle" it, which means that most data (including sockets!) can be passed though.

import asyncio

from quasiqueue import QuasiQueue

async def writer(desired_items: int):
  """Feeds data to the Queue when it is low.
  """
  return xrange(0, desired_items)


async def reader(identifier: int|str):
  """Receives individual items from the queue.

  Args:
      identifier (int | str): Comes from the output of the Writer function
  """
  print(f"{identifier}")


runner = QuasiQueue(
  "hello_world",
  reader=reader,
  writer=writer,
)

asyncio.run(runner.main())

QuasiQueue has a Settings system built off of Pydantic. It's designed so multiple queues can coexist in the same codebase, and so the people deploying it can modify settings easily with Environment Variables.

I hope you find this project useful- pull requests and issues are welcome at Github, where you can also find an extensive Readme. This is the third project I've released this week, and one of my favorites- make sure to check back tomorrow for my fourth project release!