3

I am new to Python, playing around with stuff, trying to communicate python services via Kafka using Faust. So I have small PoC project. Faust app definition:

# app.py

import faust as f
from models import ReadRequest, ReadResponse

app = f.App("faust-app", broker="kafka://localhost:9092", store="rocksdb://")

topics = {"read-request": app.topic("read-request", value_type=ReadRequest)}


def get_app() -> f.types.AppT:
    return app


def get_topic(name: str) -> f.types.TopicT:
    return topics[name]

My DB reader agent:

# reader.py

import pandas as pd
from pymongo import MongoClient
from app import get_app, get_topic

client = MongoClient()

app = get_app()
req_topic = get_topic("read-request")


@app.agent(req_topic)
async def read_request(requests):
    async for request in requests:
        db = client.test
        coll = db[request["collection"]]
        result = coll.find(request["query"])
        df = pd.DataFrame(result)
        response = {
            "id": request["id"],
            "data": list(df.loc[:, df.columns != "_id"].to_dict(orient="records")),
        }
        print(response) # debug <1>
        yield response

Model definitions:

# models.py

import faust as f


class ReadRequest(f.Record):
    id: int
    collection: str
    query: dict

Test agent.ask()

# test.py

import asyncio
from reader import read_request
from models import ReadRequest


async def run():
    result = await read_request.ask(ReadRequest(id=1, collection="test", query={}))
    print(result) # debug <2>


if __name__ == "__main__":
    asyncio.run(run())

So I have zookeeper, kafka server, mongodb and faust worker reader running. Everything is using out of the box configs.

When I run python3 test.py I see debug <1> print output as expected, but debug <2> never goes off and execution hangs there.

Faust docs say enter image description here

So I assume that I am doing everything right. Anyone has any clues?

  • I would suggest to follow the [faust examples/agent.py](https://faust.readthedocs.io/en/latest/userguide/agents.html?highlight=examples%2Fagent.py) example mainly having 2 files one for the app and agent and the other for the producer and use the `loop.run_until_complete(run())` instead of `asyncio.run(run())`. – Ameida Jul 07 '21 at 09:47

0 Answers0