2

I have a script that pulls from the city if Chicago and grabs a json file and then published to Pub Sub. Once the data gets into pub sub I have a dataflow template that pulls the data into Google Big Query. The final data move to BQ is failing and when I print the output in the script. I am getting a u' in front of all the fields which I believe is messing up the field match. Has anyone else had this issue and know what is wrong with my code and how to possibly remove the 'u'. I have tried multiple fix but none of them have worked. A sample out put is listed below:

('_last_updt', '2010-07-21 14:50:53.0'), ('_length', '0.69'), ('_lif_lat', '41.985032613'),

My code is listed below:

from __future__ import unicode_literals
from sodapy import Socrata
import json
from io import StringIO
from google.oauth2 import service_account
from oauth2client.client import GoogleCredentials
from google.cloud import pubsub_v1
import time
import datetime
import urllib
import urllib.request
import argparse
import base64

credentials = GoogleCredentials.get_application_default()
# change project to your Project ID
project="xxxx"
# change topic to your PubSub topic name
topic="xxxx"


res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')
res_body = res.read()
traffic=json.loads(res_body)
publisher = pubsub_v1.PublisherClient()
topicName = 'projects/' + project + '/topics/' + topic
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project,topic)
for key in traffic:
        publisher.publish(topicName,str.encode(str(key)))
        print(key.items())
john williams
  • 21
  • 2

1 Answers1

0

You are describing a Python construction: Unicode strings are shown with a u' prefix.

See:

For example, building an array with a "normal" string and an unicode one:

>>> [u'a', 'a']
[u'a', 'a']

Don't worry too much about it, they are identical strings:

>>> u'a' == 'a'
True

Now when you say "I am getting a u' in front of all the fields which I believe is messing up the field match." Where do you see this? Is this part of the Python code, or do you see these u' show up on the BigQuery web UI too?

Looking at the code you posted, is seems to be forcing all strings to be Unicode with from __future__ import unicode_literals:

>>> "a"
'a'
>>> from __future__ import unicode_literals
>>> "a"
u'a'
Felipe Hoffa
  • 54,922
  • 16
  • 151
  • 325
  • The data never makes it to Big Query as it doesn't match the fields. I made a small cloud function to test the import instead of using dataflow as a test. The error I get Unexpected token u' in JSON at position 1 and then it fails. – john williams Nov 13 '18 at 15:11
  • Sorry the error I am getting is "The error I get Unexpected token u in JSON at position 1 and then it fails". The error above had a ' in it which was not correct. So it does appear the u is causing issues during the translation. – john williams Nov 13 '18 at 15:19
  • If you have a minimal example that fails, please post the full coffe code of that example and the full error stack trace – Felipe Hoffa Nov 13 '18 at 16:36