How to run MQTT subscriber scripts written in python on google app engine

Question

I have written a python script for MQTT subscriber which works perfectly fine locally.

Python Subscriber script

import paho.mqtt.client as mqtt
import os,json

filePath = "logs.csv"

def initiateFile():
    if (os.path.exists(filePath) == False):
        fileObj = open(filePath, "w")
        fileObj.write("Label,x,y,z\n")

def readFile():
    data = open(filePath,'r').read()
    return data

def decodeJson(jsonString):
    jsonObject = json.loads(jsonString)
    label = jsonObject.keys()[0]
    x = jsonObject[label]['x']
    y = jsonObject[label]['y']
    z = jsonObject[label]['z']
    return label.encode("utf-8")+","+x+","+y+","+z;

def writeInFile(newData):
    oldData = readFile()
    fileObj = open(filePath, "w")
    fileObj.write(oldData+newData+"\n")

def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc)+" "+str(client))
    client.subscribe("sensors/test")
    initiateFile()

def on_message(client, userdata, msg):
    print msg.payload
    writeInFile(decodeJson(msg.payload))

def on_disconnect(client, userdata, rc):
    print("Disconnect, reason: " + str(rc))
    print("Disconnect, reason: " + str(client))

client = mqtt.Client()
client.username_pw_set(username, password)
client.connect(broker,port)
client.on_connect = on_connect
client.on_message = on_message
client.loop_forever()
client.on_disconnect = on_disconnect

But When I tried to deploy it on google cloud, I got "ImportError: No module named paho.mqtt.client" error.

Then I tried following solutions but got error

1.) Declare paho-mqtt library in app.yaml

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

libraries:
- name: paho-mqtt
  version: "1.3.0"

***ERROR: (gcloud.app.deploy) An error occurred while parsing file: [C:\Users\uni5p_000\Desktop\RMIT_Studies\Sem_1\Cloud_Computing\Practical\GOOGLE\python-docs-samples\appengine\standard\hello_world\app.yaml]
the library "paho-mqtt" is not supported
  in "C:\Users\uni5p_000\Desktop\RMIT_Studies\Sem_1\Cloud_Computing\Practical\GOOGLE\python-docs-samples\appengine\standard\hello_world\app.yaml", line 11, column 19***

2.) pip install paho-mqtt on Cloud shell

***OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/paho_mqtt-1.3.0.dist-info'***

How to proceed now?

Possible duplicate of [How to include third party Python libraries in Google App Engine?](https://stackoverflow.com/questions/14850853/how-to-include-third-party-python-libraries-in-google-app-engine) — hardillb, Sep 30 '17 at 11:04
If you are running in standard environment, this really won't work even once you sort out install and socket. Standard environment doesn't support long running queries. Front facing requests only 1 run for one minute, and task queues limit to 10 min. I suppose you could connect/disconnect constantly on each request triggered by cron — Tim Hoffman, Oct 02 '17 at 10:53

score 1 · Accepted Answer · answered Sep 30 '17 at 11:06

1

Google's documentation explains how to install libraries under a local lib directory.

Create a directory to store your third-party libraries, such as lib/.

mkdir lib

Use pip (version 6 or later) with the -t flag to copy the libraries into the folder you created in the previous step. For example:

pip install -t lib/ <library_name>

answered Sep 30 '17 at 11:06

hardillb

54,545
11
67
105

Thanks for your suggestion, its working now. But I am getting new error "FeatureNotEnabledError: The Socket API will be enabled for this application once billing has been enabled in the admin console." – Prince H Sep 30 '17 at 11:30
That error seams pretty self explanatory. If this answer has answered the original question asked then please mark it as so. – hardillb Sep 30 '17 at 11:31

How to run MQTT subscriber scripts written in python on google app engine

1 Answers1