GSoC Logs: circus

Description

This is the post of logs recording the questions and barriers encountered when look into the code base of circus.

May-11

gen.Task equality in high version tornado

Some function in circus depend on outdated gen.Task, to run callback-based asynchronous function. The function has an argument callback and when the function run with gen.Task, the callback is then run after the execution of the function and the result of the callback function if returned as Future object.

In higher version of tornado, the concept of callback is deprecated. Some the programming pattern here are changed to the following code according to the reference from https://stackoverflow.com/questions/57103331/how-to-replace-yield-gen-taskfn-argument-with-an-equivalent-asyncio-express :

response = yield gen.Task(fn, request)

# is equivalent to
future = tornado.concurrent.Future()
fn(request, callback=future.set_result)
response = yield future

The fn function mentioned above is functions of zmq.eventloop.zmqstream.ZMQStream, which send and receive from a non-blocking socket, using tornado. The relevant methods are send(msg, callback) and on_recv(callback). The callback argement here used for assign the task to run when everytime the function send or on_recv is called. The functions locate in file circus/client.py::AsyncCircusClient::call which call the command set in the watcher of circus arbiter.

The call timing of the method add_done_callback

In new version of tornado (>=5.0), the Future in tornado is implemented from asyncio when available, so the behavior of method add_done_callback is slightly changed. If the Future is already done, this function makes no guarantee that the callback will be called immediately. So, use future_add_done_callback instead.

def future_add_done_callback(  # noqa: F811
    future: "Union[futures.Future[_T], Future[_T]]", callback: Callable[..., None]
) -> None:
    """Arrange to call ``callback`` when ``future`` is complete.

    ``callback`` is invoked with one argument, the ``future``.

    If ``future`` is already done, ``callback`` is invoked immediately.
    This may differ from the behavior of ``Future.add_done_callback``,
    which makes no such guarantee.

    .. versionadded:: 5.0
    """
    if future.done():
        callback(future)
    else:
        future.add_done_callback(callback)

May-21

RuntimeError: Event loop is closed for tornado < 5.0.2

Tutor Sebastiaan report that some tests failed when tornado==5.0.0. By check that, I assume the problem was caused from the bug of tornado<5.0.2 itself. From tornado release note 5.0.2, there is a bug that will failed in close the event loop in tearDown setup. Update the dependency tornado to higher version fix this. And to make sure this is not the fluke, I run the failed test independently, so the loop close procedure in tearDown setup will not take effect. Fortunately, the test passed as expected.