2

I'm trying to download readable gtfs real time data (protocol buffer format) using Java so I can view it in a text file.

I tried a couple of approaches:

Approach #1:

URL url = new URL(uri); 
byte[] buffer = new byte[4096];
InputStream is = url.openStream();
byte[] buffer = new byte[4096];
InputStream is = url.openStream();
File file = new File("c:/protobuf_data.txt");
OutputStream output = new FileOutputStream(file);
int numOfBytesReadIntoBuffer = -1;
while((numOfBytesReadIntoBuffer = is.read(buffer)) != -1){
    output.write(buffer, 0, numOfBytesReadIntoBuffer);
}

results (snippet):
099700_L..S20150102*LÊ>0L 1637 8AV/RPY!¯¬œ¥¾¬œ¥"L22S(

Approach #2 (same results as approach #1): import org.apache.commons.io.IOUtils;

URL url = new URL(uri); 
InputStream is = url.openStream();
File file = new File("c:/protobuf_data.txt");
OutputStream output = new FileOutputStream(file);
byte[] bytes = IOUtils.toByteArray(is);
output.write(bytes);

I guess because they both write to the OutputStream the same way, the results are the same.

I also tried the suggestion found here but I just ended up getting errors: When using google protocol buffers to transfer String character,got messy code

I read through protocol buffer docs but I got more confused. https://developers.google.com/protocol-buffers/docs/encoding

I used com.sun.org.apache.xml.internal.security.utils.Base64 but I get an error. Approach #3 

URL url = new URL(uri);
InputStream is = url.openStream();

File file = new File("c:/users/Workstation/protobuf_data_bytes.txt");

OutputStream output = new FileOutputStream(file);

byte[] bytes = IOUtils.toByteArray(is);
Init.init();
byte[] decoded_bytes = Base64.decode(bytes);

error:
Exception in thread "main" com.sun.org.apache.xml.internal.security.exceptions.Base64DecodingException: Error while decoding

I also tried using java.util.Base64's wrap method to create an InputStream for decoding Base64 encoded byte stream but the data just got even more mangled.

Community
  • 1
  • 1
Village Idiot
  • 21
  • 1
  • 2
  • Protocol Buffers are a binary message format, and the .proto file describes the fields and data types of the message you're reading. So you're going to need the GTFS protocol message definition file from https://developers.google.com/transit/gtfs-realtime/gtfs-realtime-proto which describes how to encode and decode that GTFS message (assuming I guessed the right meaning of GTFS). You'll also need the Protocol Buffers compiler https://developers.google.com/protocol-buffers/docs/downloads which takes that .proto file as input and generates Java classes to encode and decode that message. – Jamie Flournoy Jan 03 '15 at 00:41

2 Answers2

1

The GTFS-realtime spec now includes code samples for parsing GTFS-realtime data in a variety of languages:

https://developers.google.com/transit/gtfs-realtime/code-samples

It's a good place to start when it comes to parsing GTFS-realtime data in your favorite language.

Brian Ferris
  • 7,557
  • 5
  • 25
  • 27
1

As others mentioned, the gtfs-realtime files are binary and require the special Protocol Buffer compiled code to parse. You're not going to get anywhere with Base64 decoding.

Alternative Method (Command Line + JSON):

However, if you're ONLY trying to parse gtfs-realtime files into a human-readable format, I wrote a standalone tool that converts GTFS-realtime into JSON: https://github.com/harrytruong/gtfs_realtime_json

Just download (no install), and run: gtfs_realtime_json <feed_url>

Here's a sample JSON output.

Harry Truong
  • 51
  • 4