2

I've been trying to reproduce the publisher subscriber example: https://code.kx.com/wiki/Cookbook/publishsubscribe

Everything's working fine as far as it's purely q related.

I've been trying to reproduce the subscriber in a jupyter notebook using the PyQ kernel. I didn't manage to successfully get the subscriber output/feeds in a python object. Every single time, the subscriber prints in the kernel output (console).

I've also been trying to replicate in PyQ something similar to what I've seen on the java client for q without much success(https://code.kx.com/q/interfaces/java-client-for-q/)

What I would ideally like to achieve is:

  • running my publisher in command line (so far that was the very easy part)
  • having a python notebook that listens to the port where my publisher publishes and handles events when a new feed comes in
  • a websocket solution would be outstanding

Does anyone has some pointers on how to do that on the python side? any examples?

Thanks a lot, Yael Darmon

WillowOfTheBorder
  • 23
  • 5
  • You mention a websocket solution "would be outstanding" - you could take a look at this repo: https://github.com/jonathonmcmurray/ws.q `wsu.q` & `wschaintick.q` provide the framework for creating a WebSocket chained tickerplant, essentially republishing over websockets. Examples at bottom of README – Jonathon McMurray Jul 11 '18 at 16:34
  • Hi Jonathon, thanks a lot for your answer. Sadly the difficulty that I'm raising here is first and foremost the ability to get my result in a python object. I can see in your git repo that the important part I'm looking for is in node js. Would you know about a similar client in python? Thanks in advance, Yael – WillowOfTheBorder Jul 12 '18 at 08:21
  • node.js is just an example. any language that can act as a websocket client & parse JSON (that covers python for sure) can substitute in. there's nothing reliant on node.js in that repo. all that said, it's probably possible & better for performance to do something more directly in PyQ rather than rely on websockets – Jonathon McMurray Jul 12 '18 at 08:28
  • Couldn't manage to find any client example that works in python. Also on kx website, there's node js and java examples but no python example. I've tried to adapt it without success. but yes in the end I want something more directly in PyQ you're right – WillowOfTheBorder Jul 12 '18 at 09:40

1 Answers1

1

In order to get q output redirected to the Jupyter notebook you should specify -1 (for stdout) and -2 (for stderr) options to the %%q cell magic.

I want it in a python object

You can capture standard streams by redirecting them to a pipe. For example, on Linux you can do

>>> import os
>>> r,w = os.pipe()
>>> q('\\2 /dev/fd/%d' % w)
k('::')
>>> q('-2 "hello"')
k('-2')
>>> os.read(r, 5)
b'hello'

Note that I used stderr for illustration because redirecting stdout would interfere with the REPL display, but the same technique would work for stdout. The usual caveats for working with pipes apply. Pipes have a limited buffer, so you need to arrange regular reads. Typically this will involve some kind of event management which is outside of the scope of this answer. If you don't want to deal with such complexity, your best bet is to use a temporary file instead of a pipe. I would also recommend studying how "fdcap" fixture implemented in pytest.

Alexander Belopolsky
  • 2,228
  • 10
  • 26