Apache and Python threading strange results

Question

I'm writing some code that queries SteamAPI for data on servers and then compiles a larger list and outputs JSON data.

It does this by:

requesting a list of all the servers Ip's (ip, port)
then sending a request to the ip/port
transforming the results,
appending it to a master list
formatting the final list using json.dumps()

It's written in python 2.7.6

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# gets all the miscreated servers

import Queue
import json
import unicodedata
from threading import Thread

import pyfscache
import valve.source.a2s
import valve.source.master_server

cache_it = pyfscache.FSCache('./cache/', days=0, hours=4, minutes=30)

# The main queue object
q = Queue.LifoQueue()
# this is the list we append to
final_servers_list = []


def print_headers():
    print "Content-type: application/json\n"
    print ""


def get_master_server_list():
    msq = valve.source.master_server.MasterServerQuerier()
    servers = msq.find(appid="299740")
    return servers


def normalize(data):
    if type(data) is unicode:
        return unicodedata.normalize('NFKD', data).encode('ascii', 'ignore')
    else:
        return data


def get_single_server_data():
    while not q.empty():  # check that the queue isn't empty
        try:
            server_address = q.get()
            _server = valve.source.a2s.ServerQuerier(server_address)
            info = _server.get_info()

            try:
                server_time = info['server_tags'].split(';')[0][-5:]
            except:
                server_time = '00:00'
            try:
                players = info['server_tags'].split(';')[1]
            except:
                players = '0'
            try:
                whitelisted = info['server_tags'].split(';')[2]
            except:
                whitelisted = '0'

            final_servers_list.append({'name': normalize(info['server_name']),
                                       'mapName': normalize(info['map']),
                                       'ip': normalize(server_address[0]),
                                       'port': normalize(server_address[1]),
                                       'time': normalize(server_time),
                                       'players': normalize(players),
                                       'whiteListed': normalize(whitelisted),
                                       'maxPlayers': normalize(info['max_players']),
                                       'version': normalize(info['version'])})
            q.task_done()
        except:
            q.task_done()


@cache_it
def get_server_list():
    master_server_list = get_master_server_list()
    for server in master_server_list:
        q.put(server)
    for i in range(200):
        t1 = Thread(target=get_single_server_data)  # target is the above function
        t1.start()  # start the thread
    q.join()
    return final_servers_list


if __name__ == '__main__':
    print_headers()
    print json.dumps(get_server_list())

Now this code works fine on my local machine running a scotchbox vagrant lamp stack The python is the same version on the server/dev machine.

I get what I expect on my machine I get about 500 servers back and all of the data exactly as I expect it.

However when I run this on the webserver running apache2 it spits me back a list that is in the 10,000 range with many of the results 10-20 times in a list. Even if I try to filter the results it's like some of the data is slightly different (because it was requested maybe a second later or a second time? and the server tags changed?)

I assume this has something to do with threading and apache with python and for some reason it not having some sort of Lock file, but I can not for the life of me figure this out. I thought I had it solved by doing a

source /etc/apache2/envvars

Then running the script for an ssh terminal

and it started working but then the next time the cache expired and the code was run it gave the same results back to me.

Any suggestions would be greatly appreciated because I'm banging my head against a wall here.

As a side note when I run apache2 -V

it spits out an error:

[Wed Feb 01 05:35:59.192112 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOCK_DIR} is not defined
[Wed Feb 01 05:35:59.192531 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Wed Feb 01 05:35:59.193300 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.214298 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
[Wed Feb 01 05:35:59.215112 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.215499 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.215708 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216057 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216272 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216595 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217080 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217475 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217812 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218115 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218369 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218657 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218885 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219117 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219348 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219631 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219845 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
AH00526: Syntax error on line 75 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}

It does this on both dev and production machine so I didn't think this was a major deal.

Finally the apache config of the site.

<VirtualHost *:80>
        ServerName "servers.miscreatedgame.com"
        ServerAdmin "csprance@entradainteractive.com"
        DocumentRoot "/var/www/servers.com/_build"

        # serverpanel appollo
        <Directory "/var/www/servers.com/_build">
                AddHandler cgi-script .py
                Options +ExecCGI
        </Directory>



        ErrorLog ${APACHE_LOG_DIR}/servers-error.log
        CustomLog ${APACHE_LOG_DIR}/servers-access.log combined

</VirtualHost>

Use thread local variables (see http://stackoverflow.com/questions/104983/what-is-thread-local-storage-in-python-and-why-do-i-need-it) rather than global variables — sureshvv, Mar 27 '17 at 17:50

score 2 · Answer 1 · answered Mar 21 '17 at 03:07

As @SergGr said, that definitely seems like a thread race condition between multiple requests. I would suggest trying to make your code reentrant. I would put the code that actually creates the Threads and constructs the whole server list in a separate process which then returns that to the process that handles the request from the user using ipc.

SergGr · Accepted Answer · 2017-03-21T02:25:29.543

In what environment that script is actually run on the server? Is it an independent script or is it run as a part of a web-server (Apache)? In the latter case, don't you happen to have several concurrent (HTTP) requests for the same data? Your q = Queue.LifoQueue() seems to be a globally shared variable that any processing request has access to and so all concurrent requests will fill the same "queue" (q). This might be the reason why it happens randomly: it happens only when there are concurrent requests for this data. If this is the case, the obvious way to fix it is to make q local variable to get_server_list and pass it explicitly to get_single_server_data using args parameter of the Thread constructor which is actually a good thing anyway. Obviously the same goes about final_servers_list you use for output.

Update

After some more thinking, what cleans the final_servers_list up? Assume you just run this HTTP request 10 times (sequentially, not concurrently). Why you don't expect to get the whole list of servers reported 10 times in the last response?

But proper solution is still the same: don't use global variable. It is much more reliable and future-proof then just clearing the list.

Apache and Python threading strange results

2 Answers2