0

I have the following code. I'm calling a source API endpoint using GET and am getting a stream of payload events in the response. I'm selecting specific payloads and am then posting them into another API endpoint. The code seems to work but when I evaluate the body in the destination payload response, I see the content in bytestring and it does not seem to be in json format. The content is some analytics data and does not seem to be incomplete; the json passed in definitely has a lot more detail.

CODE

import json
import requests
import get_token
from requests.auth import HTTPBasicAuth

source_streaming_url = 'https://livestream/api/stream/stream1'
destination_streaming_url = 'https://destinationstream/v1/streams/stream1'

def source_api():
    headers = {
        'x-api-key': <api_key>,
        'Authorization': 'Bearer ' + get_token.get_token().get('result').get('access_token'),
    }
    response = requests.get(source_streaming_url, headers=headers, stream=True)

    for line in response.iter_lines():
        if line:
            yield line


def main():
    for line in source_api():
        payload = json.loads(line.decode('utf-8'))
        evars_main = payload.get('variable1')
        if evars_main and evars_main.get('variable2'):
            if evars_main.get('variable2').get('myvar'):
            
                secrets =  <get_secret>

                destination_response = requests.post(
                        destination_streaming_url,
                        headers={"Content-Type": "application/json"},
                        data=payload,
                        auth=HTTPBasicAuth(secrets.get('user'), secrets.get('secret')),
                        stream=True
                        )
                print(destination_response.content)

Response content

reportSuite=stream1&exclude=id&exclude=value&exclude=description&pageEvent=id&pageEvent=type&pageEvent=description&pageEvent=linkURL&pageEvent=linkName&timeGMT=1682996441&receivedTimeGMT=1682996441&hitIdHigh=3614207338102325248&hitIdLow=46195734853736065634&mcVisIdHigh=2951833792067086269&mcVisIdLow=367663990508238318&visIdHigh=41978984813332068260&visIdLow=8004941667833356853&visIdType=5&props=prop29&props=prop40&props=prop73&props=prop30&props=prop33&props=prop21&props=site_section&props=prop48&evars=evars&events=event21&events=event89&geoCountry=jpn&geoRegion=13&geoCity=honan&geoZip=168-0072&geoDMA=30001&geoLatitude=35.63&geoLongitude=139.66&connectionType=LAN%2FWifi&topLevelDomain=.net&languageAbbrev=ja&language=Japanese&searchEngine=Microsoft+Bing&bot=%3CNot+a+bot%3E&operatingSystem=Windows+10&browserType=&browser=&javascriptVersion=1.6&monitorWidth=1600&monitorHeight=1600&monitorColorDepth=16+million+%2824-bit%29&carrier=&mobileDeviceType=&mobileDeviceName=&mobileManufacturer=&mobileScreenWidth=0
&mobileScreenHeight=0&mobileCookieSupport=False&mobileColorDepth=&mobileAudioSupport=&mobileVideoSupport=&pageURL=&pageName=&usesPersistentCookie=1&homePage=0&browserHeight=842&browserWidth=1479&javaEnabled=False&ip=<ip>&isErrorPage=False&purchaseId=&referrer=https%3A%2F%2Fwww.dr.com%2F&state=&userAgent=Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F112.0.0.0+Safari%2F537.36+Edg%2F&plugins=&currency=USD&hitSource=1&transactionId=&truncated=False&zip=&mcAudiences=values&mcAudiences=delim

To test this, I spun up a flask server and submitted the post there to retrieve the payloads being sent, and I see the same bytestring output as above. Any idea why this is in this format? How can I view the content in json format on the destination server? Here is the code that I used to spin up the flask server.

from flask import Flask, jsonify, request, Response, stream_with_context
import io
import json

app = Flask(__name__)

@app.route('/stream', methods=['POST'])
def stream():
    def generate():

        stream = io.BytesIO(request.get_data())
        while chunk := stream.read(1024):
            yield chunk
            print(chunk.decode()) # Display the received data to stdout
    return app.response_class(stream_with_context(generate()), mimetype='application/octet-stream')

if __name__ == "__main__":
    app.run(debug=True)
user2993021
  • 87
  • 1
  • 8
  • The answer you are after is simply because `https://destinationstream/v1/streams/stream1` returns data as [`application/x-www-form-urlencoded`](https://stackoverflow.com/questions/9870523/what-are-the-differences-between-application-json-and-application-x-www-form-url) rather than `application/json` - try sending an `Accept: application/json` header (duplicate what you had for `"Content-Type"` to `"Accept"`) and see if that makes a difference, otherwise you will need to convert that response to JSON if that's what you desired. – metatoaster May 02 '23 at 04:12
  • I added the header for "Accept: application/json" and that does not seem to work. I updated the flask server to print the incoming headers. The print(chunk.decode()) line still shows the data as keyword arguments or query parameters. ` Host: 127.0.0.1:5000 User-Agent: python-requests/2.28.2 Accept-Encoding: gzip, deflate Accept: application/json Connection: keep-alive Content-Type: application/json Content-Length: 1727 ` – user2993021 May 02 '23 at 04:38
  • Well, then you will have to make do with whatever response content that the `https://destinationstream` provided - without knowing exactly what that is, it becomes impossible to tell you how you can get the server to provide JSON. – metatoaster May 02 '23 at 04:47
  • I redirected the destination to flask and added `print(request.content_type)` in the flask generate() decorated function above and that shows 'application/json'. But when I print it using 'print(request.get_data())', it shows me the same content that I showed above in keyword argument format in bytes `b'reportSuite=stream1&exclude=id...'` – user2993021 May 02 '23 at 05:11
  • Could this be because the payload is sent as a stream of bytes and not as a json object when using 'stream=True' in requests? If that's the case then perhaps the content needs to be chunked. – user2993021 May 02 '23 at 05:20
  • You also specified `data=payload,` instead of `json=payload,` when making a request - it would be clear if your example and explanation is more clear in that the service you are dealing with is reflecting the request being made, because without knowing exactly how the data is being consumed it is impossible for others to determine the code you have written is actually what you want to do. To be clear, your title "Getting streamed response from one api endpoint and then pushing it into another" is the wrong focus, instead of the desired "I want the server to see the data I pushed as JSON" – metatoaster May 02 '23 at 05:29
  • I wanted to point out that this is a streamed response in the title and I'm afraid I didn't structure the title correctly. I didn't know where the problem was happening so I couldn't provide more a more explicit scope in my ask. I also tried to put as much detail as I could and provided every thing that was being asked. The problem indeed was the `data=payload` instead of `json=payload`. Thank you for that. – user2993021 May 02 '23 at 06:12
  • For future reference, it's more useful to break the problem down to avoid bringing information unrelated to the actual problem - using this example, if the streaming aspect was removed and the problem still persists, it might indicate to you that the problem might have to do with the `request.post` itself. – metatoaster May 02 '23 at 06:26

0 Answers0