4

I have a nodejs end point which will be called by a job with a POST HTTP call containing the details of the job data in JSON format?how do I listen to a webhook (https address) from python client to get the the job data?

https://company/api/bats_push

app.post("/api/bats_push",(req, res) => {
        //console.log("Calling bats_push...")
        const d = {
            method: req.method,
            headers: req.headers,
            query: req.query,
            body: ''
        }

        req.on('data', (c) => {
            d.body = d.body + c
        });


        req.on('end', () => {
            DATA.push(d);
            res.end('Saved');
        });
});

python client:

data = {'name': 'joe', 'age': 20}
webhook_url = 'http://localhost:3000/api/bats_push'
json_data = json.dumps(data)
print json_data
try:
    r = requests.post(webhook_url,data=json_data.encode("utf8"),verify=False,headers={"Content-Type": "application/json"},timeout=10.0)
    print "posted"
    print(r.status_code, r.reason)
    print r.url
except Exception as e:
    print (e)

Error:-

HTTPConnectionPool(host='localhost', port=3000): Read timed out. (read timeout=10.0)
Ritz
  • 1,193
  • 6
  • 15
  • 26
  • You should take a look at the [`websockets`](https://websockets.readthedocs.io/en/stable/) python package. – Thom Sep 25 '19 at 20:50
  • I tried to use websockets but am running into error,updated my question with the same... – Ritz Sep 26 '19 at 18:18
  • I am looking to listen over http which is not a websocket? – Ritz Sep 26 '19 at 23:01
  • 1
    If you are listening to a websocket, the URL should begin with `ws://` or `wss://` (if it's secure). Otherwise you are not passing a websocket URL. – Thom Sep 27 '19 at 19:06
  • To clarify, is the node process sending or receiving the POST request? – Fush Oct 03 '19 at 07:13
  • @Fush - the node process is POST'ing the request – Ritz Oct 03 '19 at 20:43

4 Answers4

3

Normally, a "webhook" is a feature of an online service that sends a request to your app at a public url where you have a HTTP server deployed, in response to some event within the service. The url needs to be configured in whatever is sending the request, it's not explicitly set in the receiving code.

The thing that listens to the webhook is an HTTP server with a suitable endpoint defined. To create a server you can use many different packages, but something simple like Flask is a good place to start.

The minimal view code would look something like:

@app.route('/api/bats_hook')
def bats_hook():
    data = request.get_json()
    print(data)
    return 'OK'

If whatever is sending the POST (your nodejs service?) needs to be able to send requests to your server, so you'll either need to deploy the server so it's publicly accessible, or set the service to send the requests to your public IP. If it's on the same machine you can use a private IP, or localhost.

On the other hand, if a node process is currently receiving the POST requests at https://company/api/bats_hook/, and you want to notify a python process whenever a job comes in, that's a different problem. You could either send that information on to a separate python server via axios.post or similar (this might be the easiest option), or store the data somewhere that the python process can access it. Redis would be a good option, because you can easily set up a publish/subscribe system, and the node and python processes can be on the same machine or on different machines. If they're on the same machine, you could also just dump the data to a log file and read that into python. Sending data from one process to another is called inter-process communication, and it gets complicated quickly.

If you were to use redis, with node_redis and redis-py, the implementation might look like:

// javascript
var redis = require("redis"),
    redisClient = redis.createClient();

app.post("/api/bats_hook", (req, res, next) => {
  console.log(req.query.params)
  console.log(req.body)
  console.log(req.body.name)
  // This publishes a message to the "bats hook channel"
  redisClient.publish("bats hook channel", JSON.stringify({
    params: req.query.params,
    body: req.body,
  }));
  res.status(200).json({
    message: "BATS object",
    posts: req.body
  }); 
});
# python
import redis

r = redis.Redis()
p = r.pubsub()
p.subscribe('bats hook channel')

# This blocks and reads any messages from the "bats hook channel" as they come in
for message in p.listen():
    print(message)

A lot depends on your intended usage. If the python part is going to be just used for debugging locally, you'd make different choices than if it needs to be deployed.

Fush
  • 2,469
  • 21
  • 19
  • let me re-clarify ,I already have a nodejs POST end point https://company/api/bats_hook ,I updated my question on how this looks like ,my question is how do I listen from a python client for any job notifies this endpoint.. – Ritz Oct 04 '19 at 21:16
  • If a node process is _sending_ a POST request to the `https://company/api/bats_hook/` url, then you need a python _server_ (not client) listening at that url. To do that, you need to deploy a server, and there are many ways to do that. Instructions for a Flask server, for example, are at https://flask.palletsprojects.com/en/1.1.x/deploying/ – Fush Oct 07 '19 at 05:49
  • @Ritz I've updated my answer based on what I think your setup might be. – Fush Oct 07 '19 at 06:51
  • how do I configure my node.js API to use REDIS_URL running @`bash-4.3# echo $REDIS_URL redis://redisdb:###REDACTEDPASSWORD###@orchard-ambassador:53854/0 ` – Ritz Oct 16 '19 at 03:28
  • I tried to use `redisClient = process.env.REDIS_URL` but whenever I post I get a `(500, 'Internal Server Error', u' \n\n\n\nError\n\n\n
    Internal Server Error
    \n\n\n')`
    – Ritz Oct 16 '19 at 03:58
  • You should be able to use `redisClient = redis.createClient({url: process.env.REDIS_URL});`. If it's not doing what it should, you should probably get it working in the [REPL](https://nodejs.org/docs/latest/api/repl.html) first - make sure you can publish a test message from node so it's picked up by your python process. – Fush Oct 22 '19 at 01:27
  • the code you posted works locally,how to we subscribe to the `bats hook channel'` over http ?I pushed the node.js code and it resides over `https://company.com/api/bats_holder` – Ritz Oct 25 '19 at 01:23
  • You need a redis server deployed that is accessible to both the node and python processes. If you're on AWS, you can use [elasticache](https://aws.amazon.com/elasticache/redis/). – Fush Oct 29 '19 at 03:31
  • I do have the redis server running on the server,its not clear to me how just subscribing to "bats hook channel" would work over a http server – Ritz Oct 29 '19 at 18:22
  • To talk to redis, the node and python processes both need to be able to "see" the server and the port redis is using, which is 6379 by default or 53854 if that's what you've configured. It doesn't go through http, it uses its own protocol. Is the python process running on the same machine as the node server? Is it on the same network/VPC? Or are you using it locally for debugging? To get it working I'd ssh into the box that will have the python process and use the redis-cli or python REPL to make sure you can connect to the redis server. – Fush Oct 30 '19 at 01:12
1

The setup you describe looks a bit weird, at least to me.

To clarify (mostly for myself):

  • You do have a node.js server serving the /api/bats_hook endpoint using POST on HTTP.
  • There are jobs running that access that enpoint now and then to obtain information.
  • You ask for a way to obtain that response (= job data) using Python.

The conditions can only hold if the job is the Python part that sends the POST request to your node.js server and receives the JSON on HTTP 200. Otherwise you'd try to wiretap to get foreign requests and their corresponding responses from the node.js server without being the actual communication partner.

If the job code is Python and you want to properly POST the node.js service, use the requests Python package. Sending a request and obtaining the JSON response is two lines of code then.

If the job code is not under your control or not the location where you want the Python "hook" to be active, that either smells like bad architecture or misusing a setup a way it's not intended. Please clarify.

jbndlr
  • 4,965
  • 2
  • 21
  • 31
  • To clarify the workflow is that are jobs running from another team(not in my control) that posts the result of that job(in JSON format) to the webhook @/`api/bats_hook`(this webhook is maintained by me) ,as a client I want to know the result of that job whenever the result is posted – Ritz Oct 07 '19 at 16:20
  • Well, that way you have chosen `node.js` for a reason and you should not try to relay code from `node.js` to Python (depending on your deployment architecture, this can involve more/new services). Instead stick to `node.js` or replace that one and use a pure Python web server serving the `/api/bats_hook` (e.g. Flask or FastAPI). – jbndlr Oct 08 '19 at 17:47
1

You save the posted data and return it to an interested consumer. It means you create a webhook bridge.

Example Node script:

const http = require('http');
const URL = require('url');

const DATA = [];

const routes = {
    '/api/bats_push': (req, res) => {
        const d = {
            method: req.method,
            headers: req.headers,
            query: req.query,
            body: ''
        }

        req.on('data', (c) => {
            d.body = d.body + c
        });

        req.on('end', () => {
            DATA.push(d);
            res.end('Saved');
        });
    },

    '/api/bats_pull': (req, res) => {
        const json = JSON.stringify(DATA, true); 
        DATA.length = 0;
        res.statusCode = 200;
        res.setHeader('content-type','application/json');
        res.end(json);
    }
};

const server = http.createServer((req, res) => {
    const reqUrl = URL.parse(req.url);
    const route = routes[reqUrl.pathname];
    if (route) { 
        req.query = reqUrl.query;
        route(req, res);
    } else {
        res.statusCode = 404;
        res.end('Not found');
    }
})

server.listen(8080,() => console.info('Server started'));

And, a python client script to test it:

import http.client
import json

print('\nPosting data...')

conn1 = http.client.HTTPConnection('localhost', 8080)
headers1 = {'content-type': 'application/json'}
body1 = {'name': 'joe', 'age': 20}

conn1.request('POST', '/api/bats_push?q=100', json.dumps(body1), headers1)

resp1 = conn1.getresponse()
print(resp1.status,resp1.reason, resp1.read())
conn1.close()

print('\nRetrieving data...')

conn2 = http.client.HTTPConnection('localhost', 8080)
headers2 = {'accept': '*'}

conn2.request('POST', '/api/bats_pull', None, headers1)

resp2 = conn2.getresponse()
resp2Json = json.loads(resp2.read())
print(resp2.status,resp2.reason, resp2Json)

output:

Posting data...
200 OK b'Saved'

Retrieving data...
200 OK [{'method': 'POST', 'headers': {'host': 'localhost:8080', 'accept-encoding': 'identity', 'content-length': '26', 'content-type': 'application/json'}, 'query': 'q=100', 'body': '{"name": "joe", "age": 20}'}]

Also, The HTTP request is saved "as is" . Contents of header, query and body are not processed. The clients should parse the data accordingly after pulling.

S.D.
  • 29,290
  • 3
  • 79
  • 130
  • this throws the error in https://stackoverflow.com/questions/58313447/read-timed-out-error-while-sending-a-post-request-to-a-node-js-api/58313913#58313913 – Ritz Oct 10 '19 at 03:39
  • @Ritz Updated answer. Works fine with python client. – S.D. Oct 10 '19 at 06:03
  • I still get the same error as in https://stackoverflow.com/questions/58313447/read-timed-out-error-while-sending-a-post-request-to-a-node-js-api/58313913#58313913 – Ritz Oct 11 '19 at 03:43
  • I updated my question with the exact `node.js` API and `python client` code am using,this only happens if I have the `headers={"Content-Type": "application/json"}` – Ritz Oct 11 '19 at 03:56
  • Do you have inputs?am so confused as to why sending POST request with `headers={"Content-Type": "application/json"}`gets Timeout error and it works fine without the headers ,looks like I am missing something fundamental – Ritz Oct 14 '19 at 19:44
  • @Ritz Have you modified the server script? The server and client scripts that are given here work properly for me. Also, the server and client here both send the `content-type` header. Note that the header name is lower case. – S.D. Oct 17 '19 at 08:49
0

Working with payment related systems in my career; in general webhooks are a callback that you get for a process initiated but it is separated from the main flow so that that memory is not held up without a reason.

Going by what you said

[Webhook] => Python => Node.js

and I wasn't able to understand what exactly is making a call for the process which adds the webhook flow in the whole cycle. Regardless it shouldn't change the answer.

So basically what you need is an API endpoint which will be called from this third party webhook. How you decide to facilitate that on your side using python is entirely your call. An API system up and running wouldn't be that complicated using frameworks like Django, performance etc. related questions is something I wouldn't dwell into.

Let me know if it clear to you.

Gandalf the White
  • 2,415
  • 2
  • 18
  • 39
  • Sorry,its not clear.To clarify the workflow is that are jobs running from another team(not in my control) that posts the result of that job(in JSON format) to the webhook @/`api/bats_hook` (this webhook is maintained by me) ,as a client I want to know the result of that job whenever the result is posted – Ritz Oct 07 '19 at 16:22
  • The process I have mentioned should work fine then, like I said how the request has been initiated shouldn't change anything in the answer. – Gandalf the White Oct 08 '19 at 05:11