I'm working on a Kivy app which needs to upload a file to S3 using boto3 but fails in the JSON decoder.py when trying to load the endpoint model and establish a client connection. This works fine locally running on 16.04 but fails after compiling with buildozer and running on Lollipop 5.1.1.
Buildozer log.
I/python (27222): [INFO ] [Base ] Leaving application in progress...
I/python (27222): Traceback (most recent call last):
I/python (27222): File "/build/phytogp/android/app/main.py", line 10, in <module>
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/app.py", line 828, in run
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/base.py", line 504, in runTouchApp
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/core/window/window_sdl2.py", line 663, in mainloop
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/core/window/window_sdl2.py", line 405, in _mainloop
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/base.py", line 342, in idle
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/base.py", line 327, in dispatch_input
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/base.py", line 293, in post_dispatch_input
I/python (27222): File "kivy/_event.pyx", line 718, in kivy._event.EventDispatcher.dispatch (kivy/_event.c:8191)
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivymd/ripplebehavior.py", line 68, in on_touch_up
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivymd/button.py", line 314, in on_touch_up
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivymd/button.py", line 206, in on_touch_up
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/uix/behaviors/button.py", line 179, in on_touch_up
I/python (27222): File "kivy/_event.pyx", line 714, in kivy._event.EventDispatcher.dispatch (kivy/_event.c:8146)
I/python (27222): File "kivy/_event.pyx", line 1225, in kivy._event.EventObservers.dispatch (kivy/_event.c:14035)
I/python (27222): File "kivy/_event.pyx", line 1109, in kivy._event.EventObservers._dispatch (kivy/_event.c:12816)
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/kivy/lang/builder.py", line 64, in custom_callback
I/python (27222): File "<string>", line 31, in <module>
I/python (27222): File "/build/phytogp/android/app/views/widgets/checkedin.py", line 112, in finish_button_on_release
I/python (27222): File "/build/phytogp/android/app/views/widgets/checkedin.py", line 144, in upload_report
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/boto3/__init__.py", line 83, in client
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/boto3/session.py", line 263, in client
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/session.py", line 826, in create_client
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/session.py", line 701, in get_component
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/session.py", line 897, in get_component
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/session.py", line 186, in create_default_resolver
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/loaders.py", line 420, in load_data
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/site-packages/botocore/loaders.py", line 175, in load_file
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/__init__.py", line 339, in loads
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/decoder.py", line 366, in decode
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/decoder.py", line 382, in raw_decode
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/scanner.py", line 38, in _scan_once
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/decoder.py", line 195, in JSONObject
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/scanner.py", line 40, in _scan_once
I/python (27222): File "/build/phytogp/android/platform/build/dists/phytogp/private/lib/python2.7/json/decoder.py", line 262, in JSONArray
I/python (27222): ValueError: Expecting , delimiter: line 63 column 29 (char 1666)
Connection command causing the issue is:
main.py
client = boto3.client(
's3',
aws_access_key_id='IVEGOTTHEKEY',
aws_secret_access_key='IVEGOTTHESECRET',
region_name='ap-southeast-2',
)
Looking at the stack trace it's falling over trying to load endpoints.json with the botocore.loader at the load_file line in the load_data method:
loaders.py
@instance_cache
def load_data(self, name):
for possible_path in self._potential_locations(name):
found = self.file_loader.load_file(possible_path)
if found is not None:
return found
# We didn't find anything that matched on any path.
raise DataNotFoundError(data_path=name)
Here it's looking for endpoints.json in the _potential_locations which are:
loaders.py
# The included models in botocore/data/ that we ship with botocore.
BUILTIN_DATA_PATH = os.path.join(BOTOCORE_ROOT, 'data')
# For convenience we automatically add ~/.aws/models to the data path.
CUSTOMER_DATA_PATH = os.path.join(os.path.expanduser('~'),
'.aws', 'models')
BOTOCORE_ROOT is where it should pick up these files. When deployed to Android BOTOCORE_ROOT is /data/data/com.phyto.phytogp/files/app/lib/python2.7/site-packages/botocore.
endpoints.json exists at this path. Transferring endpoints.json from tablet to PC and calling the load_file method of the JSONFileLoader works fine and returns the OrderedDict without the decoding failure.
The stack trace doesn't show that it failed to find the file by raising DataNotFoundError which suggests it found the file but couldn't parse it hence the json/decoder error.
I've tried forcing versions of boto3 and dependencies to the same used locally but no good. APK is built on the Buildozer_VM with the following spec:
Buildozer.spec
[app]
# (str) Title of your application
title = PhytoGP
# (str) Package name
package.name = phytogp
# (str) Package domain (needed for android/ios packaging)
package.domain = com.phyto
# (str) Source code where the main.py live
source.dir = .
# (list) Source files to include (let empty to include all the files)
source.include_exts = py,png,jpg,kv,atlas,ttf,db
# (list) List of inclusions using pattern matching
#source.include_patterns = assets/*,images/*.png
# (list) Source files to exclude (let empty to not exclude anything)
#source.exclude_exts = spec
# (list) List of directory to exclude (let empty to not exclude anything)
source.exclude_dirs = tests, bin
# (list) List of exclusions using pattern matching
#source.exclude_patterns = license,images/*/*.jpg
# (str) Application versioning (method 1)
version = 0.1
# (str) Application versioning (method 2)
# version.regex = __version__ = ['"](.*)['"]
# version.filename = %(source.dir)s/main.py
# (list) Application requirements
# comma seperated e.g. requirements = sqlite3,kivy
requirements = kivy,sqlite3,boto3==1.4.1,botocore==1.5.78,python-dateutil==2.6.0,jmespath==0.9.3,openssl,sparkpost,urllib3,requests,xhtml2pdf,reportlab,html5lib==1.0b8,webencodings,git+https://gitlab.com/kivymd/KivyMD.git
# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy
# (list) Garden requirements
#garden_requirements =
# (str) Presplash of the application
presplash.filename = %(source.dir)s/assets/presplash.png
# (str) Icon of the application
icon.filename = %(source.dir)s/assets/icon.png
# (str) Supported orientation (one of landscape, portrait or all)
orientation = landscape
# (list) List of service to declare
#services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY
#
# OSX Specific
#
#
# author = © Copyright Info
#
# Android specific
#
# (bool) Indicate if the application should be fullscreen or not
fullscreen = 1
# (list) Permissions
android.permissions = INTERNET
# (int) Android API to use
#android.api = 19
# (int) Minimum API required
#android.minapi = 9
# (int) Android SDK version to use
#android.sdk = 20
# (str) Android NDK version to use
#android.ndk = 9c
# (bool) Use --private data storage (True) or --dir public storage (False)
#android.private_storage = True
# (str) Android NDK directory (if empty, it will be automatically downloaded.)
#android.ndk_path =
# (str) Android SDK directory (if empty, it will be automatically downloaded.)
#android.sdk_path =
# (str) ANT directory (if empty, it will be automatically downloaded.)
#android.ant_path =
# (str) python-for-android git clone directory (if empty, it will be automatically cloned from github)
#android.p4a_dir =
# (str) The directory in which python-for-android should look for your own build recipes (if any)
#p4a.local_recipes =
# (str) Filename to the hook for p4a
#p4a.hook =
# (list) python-for-android whitelist
#android.p4a_whitelist =
# (bool) If True, then skip trying to update the Android sdk
# This can be useful to avoid excess Internet downloads or save time
# when an update is due and you just want to test/build your package
# android.skip_update = False
# (str) Bootstrap to use for android builds (android_new only)
# android.bootstrap = sdl2
# (str) Android entry point, default is ok for Kivy-based app
#android.entrypoint = org.renpy.android.PythonActivity
# (list) List of Java .jar files to add to the libs so that pyjnius can access
# their classes. Don't add jars that you do not need, since extra jars can slow
# down the build process. Allows wildcards matching, for example:
# OUYA-ODK/libs/*.jar
#android.add_jars = foo.jar,bar.jar,path/to/more/*.jar
# (list) List of Java files to add to the android project (can be java or a
# directory containing the files)
#android.add_src =
# (str) python-for-android branch to use, if not master, useful to try
# not yet merged features.
#android.branch = master
# (str) OUYA Console category. Should be one of GAME or APP
# If you leave this blank, OUYA support will not be enabled
#android.ouya.category = GAME
# (str) Filename of OUYA Console icon. It must be a 732x412 png image.
#android.ouya.icon.filename = %(source.dir)s/data/ouya_icon.png
# (str) XML file to include as an intent filters in <activity> tag
#android.manifest.intent_filters =
# (list) Android additionnal libraries to copy into libs/armeabi
#android.add_libs_armeabi = libs/android/*.so
#android.add_libs_armeabi_v7a = libs/android-v7/*.so
#android.add_libs_x86 = libs/android-x86/*.so
#android.add_libs_mips = libs/android-mips/*.so
# (bool) Indicate whether the screen should stay on
# Don't forget to add the WAKE_LOCK permission if you set this to True
#android.wakelock = False
# (list) Android application meta-data to set (key=value format)
#android.meta_data =
# (list) Android library project to add (will be added in the
# project.properties automatically.)
#android.library_references =
# (str) Android logcat filters to use
#android.logcat_filters = *:S python:D
# (bool) Copy library instead of making a libpymodules.so
#android.copy_libs = 1
# (str) The Android arch to build for, choices: armeabi-v7a, arm64-v8a, x86
android.arch = armeabi-v7a
#
# iOS specific
#
# (str) Path to a custom kivy-ios folder
#ios.kivy_ios_dir = ../kivy-ios
# (str) Name of the certificate to use for signing the debug version
# Get a list of available identities: buildozer ios list_identities
#ios.codesign.debug = "iPhone Developer: <lastname> <firstname> (<hexstring>)"
# (str) Name of the certificate to use for signing the release version
#ios.codesign.release = %(ios.codesign.debug)s
[buildozer]
# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2
# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1
# (str) Path to build artifact storage, absolute or relative to spec file
build_dir = /build/phytogp
# (str) Path to build output (i.e. .apk, .ipa) storage
# bin_dir = ./bin
# -----------------------------------------------------------------------------
# List as sections
#
# You can define all the "list" as [section:key].
# Each line will be considered as a option to the list.
# Let's take [app] / source.exclude_patterns.
# Instead of doing:
#
#[app]
#source.exclude_patterns = license,data/audio/*.wav,data/images/original/*
#
# This can be translated into:
#
#[app:source.exclude_patterns]
#license
#data/audio/*.wav
#data/images/original/*
#
# -----------------------------------------------------------------------------
# Profiles
#
# You can extend section / key with a profile
# For example, you want to deploy a demo version of your application without
# HD content. You could first change the title to add "(demo)" in the name
# and extend the excluded directories to remove the HD content.
#
#[app@demo]
#title = My Application (demo)
#
#[app:source.exclude_patterns@demo]
#images/hd/*
#
# Then, invoke the command line with the "demo" profile:
#
#buildozer --profile demo android debug
SSL comms are working fine when deployed and talking to SparkPost and another 3rd party API.
I'm out of ideas so any help would be great. Thanks
