0

The problem and output

When using my python script I'm getting the same response from a API call returning json, and it seems to happen for several hours concurrently.

I'm using the Coindesk BPI API, which updates once a minute. So as we know, Bitcoin's price doesn't stay level for 5 hours. See the output example below:

    # results.txt
    {"timestamp": 16-Apr-2020 22:50, "price": 7078, "gCount": 28, "rCount": 48}
    {"timestamp": 16-Apr-2020 23:00, "price": 7085, "gCount": 29, "rCount": 50}
    {"timestamp": 16-Apr-2020 23:10, "price": 7011, "gCount": 33, "rCount": 52}
    {"timestamp": 16-Apr-2020 23:20, "price": 7002, "gCount": 31, "rCount": 55}
    {"timestamp": 16-Apr-2020 23:30, "price": 7020, "gCount": 30, "rCount": 52}
    {"timestamp": 16-Apr-2020 23:40, "price": 7027, "gCount": 33, "rCount": 54}
    {"timestamp": 16-Apr-2020 23:50, "price": 7047, "gCount": 35, "rCount": 58}
    {"timestamp": 17-Apr-2020 00:01, "price": 7060, "gCount": 36, "rCount": 57}
    {"timestamp": 17-Apr-2020 00:10, "price": 7051, "gCount": 34, "rCount": 45}
    {"timestamp": 17-Apr-2020 00:20, "price": 7052, "gCount": 41, "rCount": 48}
    {"timestamp": 17-Apr-2020 00:31, "price": 7054, "gCount": 47, "rCount": 48}
    # It worked! Now the price is stuck for 2 get requests.
    {"timestamp": 17-Apr-2020 00:40, "price": 7054, "gCount": 48, "rCount": 47}
    {"timestamp": 17-Apr-2020 00:50, "price": 7054, "gCount": 50, "rCount": 48}
    {"timestamp": 17-Apr-2020 01:01, "price": 7051, "gCount": 48, "rCount": 43}
    # Price stuck again for around 30 get requests.
    {"timestamp": 17-Apr-2020 01:10, "price": 7051, "gCount": 46, "rCount": 47}
    {"timestamp": 17-Apr-2020 01:20, "price": 7051, "gCount": 49, "rCount": 46}
    {"timestamp": 17-Apr-2020 01:30, "price": 7051, "gCount": 48, "rCount": 47}
    {"timestamp": 17-Apr-2020 01:40, "price": 7051, "gCount": 50, "rCount": 48}
    {"timestamp": 17-Apr-2020 01:50, "price": 7051, "gCount": 50, "rCount": 52}
    {"timestamp": 17-Apr-2020 02:00, "price": 7051, "gCount": 51, "rCount": 56}
    {"timestamp": 17-Apr-2020 02:10, "price": 7051, "gCount": 50, "rCount": 55}
    {"timestamp": 17-Apr-2020 02:20, "price": 7051, "gCount": 57, "rCount": 57}
    {"timestamp": 17-Apr-2020 02:30, "price": 7051, "gCount": 48, "rCount": 54}
    {"timestamp": 17-Apr-2020 02:40, "price": 7051, "gCount": 52, "rCount": 54}
    {"timestamp": 17-Apr-2020 02:51, "price": 7051, "gCount": 54, "rCount": 57}
    {"timestamp": 17-Apr-2020 03:00, "price": 7051, "gCount": 53, "rCount": 59}
    {"timestamp": 17-Apr-2020 03:11, "price": 7051, "gCount": 53, "rCount": 59}
    {"timestamp": 17-Apr-2020 03:21, "price": 7051, "gCount": 50, "rCount": 55}
    {"timestamp": 17-Apr-2020 03:31, "price": 7051, "gCount": 51, "rCount": 55}
    {"timestamp": 17-Apr-2020 03:41, "price": 7051, "gCount": 52, "rCount": 56}
    {"timestamp": 17-Apr-2020 03:51, "price": 7051, "gCount": 50, "rCount": 55}
    {"timestamp": 17-Apr-2020 04:01, "price": 7051, "gCount": 48, "rCount": 56}
    {"timestamp": 17-Apr-2020 04:10, "price": 7051, "gCount": 39, "rCount": 50}
    {"timestamp": 17-Apr-2020 04:20, "price": 7051, "gCount": 39, "rCount": 49}
    {"timestamp": 17-Apr-2020 04:31, "price": 7051, "gCount": 41, "rCount": 53}
    {"timestamp": 17-Apr-2020 04:40, "price": 7051, "gCount": 43, "rCount": 53}
    {"timestamp": 17-Apr-2020 04:50, "price": 7051, "gCount": 39, "rCount": 51}
    {"timestamp": 17-Apr-2020 05:00, "price": 7051, "gCount": 37, "rCount": 52}
    {"timestamp": 17-Apr-2020 05:11, "price": 7051, "gCount": 38, "rCount": 54}
    {"timestamp": 17-Apr-2020 05:20, "price": 7051, "gCount": 31, "rCount": 49}
    {"timestamp": 17-Apr-2020 05:30, "price": 7051, "gCount": 0, "rCount": 0}
    {"timestamp": 17-Apr-2020 05:41, "price": 7051, "gCount": 32, "rCount": 49}
    {"timestamp": 17-Apr-2020 05:50, "price": 7051, "gCount": 37, "rCount": 49}
    {"timestamp": 17-Apr-2020 06:01, "price": 7051, "gCount": 39, "rCount": 51}
    {"timestamp": 17-Apr-2020 06:11, "price": 7051, "gCount": 41, "rCount": 47}
    {"timestamp": 17-Apr-2020 06:21, "price": 7051, "gCount": 42, "rCount": 46}
    # Now it works again as intended.
    {"timestamp": 17-Apr-2020 06:31, "price": 7082, "gCount": 45, "rCount": 49}
    {"timestamp": 17-Apr-2020 06:40, "price": 7084, "gCount": 48, "rCount": 50}
    {"timestamp": 17-Apr-2020 06:51, "price": 7095, "gCount": 45, "rCount": 51}
    {"timestamp": 17-Apr-2020 07:01, "price": 7097, "gCount": 44, "rCount": 45}
    {"timestamp": 17-Apr-2020 07:11, "price": 7068, "gCount": 45, "rCount": 46}
    {"timestamp": 17-Apr-2020 07:21, "price": 7070, "gCount": 43, "rCount": 45}

Python script and what I've tried

I'm using python 2.7 and requests. Requests does not cache by default. So I thought the connection just stays up randomly and python reuses it, getting the same json.

I tried to close the requests session by setting keep alive to false, by using the with block and by trying requests.session().close(). Below find the relevant python code:

import requests, json, sys, time
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry


def request_json():
    print 'Begin request to get the json...'

    # Try get request once
    response = requests_retry_session().get('https://api.coindesk.com/v1/bpi/currentprice/USD.json')
    if (response.status_code == 200):
        # Close the connection 
        # requests.session().close() <-- tried, doesn't do the trick
        print 'Fetched price succesfully.\n'
        return response.json()

    # If first request didn't succeed, retry 3 times using session 
    with requests.Session() as s:
        s.get('https://api.coindesk.com/v1/bpi/currentprice/USD.json')
        # Close the connection
        # s.config['keep_alive'] = False <-- tried, doesn't do the trick
        response = requests_retry_session(session=s).get(
            'https://api.coindesk.com/v1/bpi/currentprice/USD.json'
        )

    # When requests succeed using session
    if (response.status_code == 200):
        # Close the connection 
        # requests.session().close() <-- tried, doesn't do the trick
        print 'Fetched price succesfully.\n'
        return response.json()

    print 'Couldn\'t fetch price json.'
    return 'error'


def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,
):

    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )

    adapter = HTTPAdapter(max_retries=retry)
    session.mount('https://', adapter)

    return session




def get_price_data(json):

    price = str(json['bpi']['USD']['rate'])
    # Strip the ',' from price, convert to float and to int
    price = int(float(price.replace(',', '')))

    return price




def main():
    # Send a request for the bitcoin price json
    priceJson = request_json()
    # Check if the request and retries failed
    if (json == 'error'):
        print 'Terminating bitcoinPrice.py script.'
        sys.exit()

    # Get the data from the response json
    priceInt = get_price_data(priceJson)

    # Get timestamp as milliseconds
    milli_sec = int(round(time.time() * 1000))

    # Read the colordata from colors.txt
    # The format is: '63,61' where greenCount,redCount
    fh = open('colors.txt', 'r')
    colorData = fh.read()
    gCount = colorData.split(',')[0]
    rCount = colorData.split(',')[1]

    # Create a string in json format with the price and color data
    dataString = "{\"timestamp\": \"%d\", \"price\": \"%d\", \"gCount\": \"%s\", \"rCount\": \"%s\"}" % (milli_sec, priceInt, gCount, rCount)
    print dataString

    # Read and write to results.txt
    fh = open('results/results.txt', 'a')
    fh.write(dataString + '\n')
    fh.close()
    print '\nSuccesfully saved BTC price and color data to results.txt'




if __name__ == '__main__':
    main()

I couldn't reproduce the bug by running crontab every minute as a regular user and only this bitcoinPrice.py script.

The bug seems to occur with my root crontab running every 10 minutes, when a couple other scripts run before this one. The actual crontab, ran by root user, simplified has other scripts chained like this:

*/10 * * * * node script1.js && python2 script2.py && python2 bitcoinPrice.py && /home/user/clearcache.sh

All other scripts work as intended. The last script, clearcache.sh, resets caches and buffers as discussed here in the following way:

#!/bin/sh
sync; echo 3 > /proc/sys/vm/drop_caches

I would like to learn what's with this bug. In case I can't find a solution I will move into using curl and dumping the API json response into a file and reading it from there. Any ideas appreciated!

J. Suo
  • 1
  • 2
  • what is in 'priceEtc' variable which you are writing to file. don't see it is used anywhere in your program. – Amal Ts Apr 18 '20 at 12:13
  • @AmalTs Many thanks for taking the time to inspect my code. I updated the python code. – J. Suo Apr 19 '20 at 08:31
  • In my observation, this has nothing to do with crontab. Try passing `{'Cache-Control': 'no-cache'}` in the header to disable cache. As you are not able to reproduce it with the same script, this could be an environment specific issue. I would suggest to do some logging after each step to identify what is going wrong under the specific scenario. – Amal Ts Apr 20 '20 at 02:44
  • Appreciate your ideas. I tried with 'no-cache' and the problem persists. I will move into option B now and curl the json to a file and read it from there. – J. Suo Apr 21 '20 at 07:19

1 Answers1

0

I managed to solve it. Using curl regularly the same issue persisted, but I used this answers (https://stackoverflow.com/a/42263514/12965126) trick and added a unique query parameter (epoch time in ms) for each request ?$(date +%s).

curl https://api.coindesk.com/v1/bpi/currentprice/USD.json?$(date +%s) -o results/priceJson.txt

...and it works without any caches. Would work with the python requests now too using the same trick.

J. Suo
  • 1
  • 2