How to download Mapbox PBF file using Java?

Question

I would like to download and parse a Mapbox PBF file from the web using Java. If I download the file manually, I am able to parse it without any errors. However, if I use Java code to download the file, and then try to parse the downloaded file, I get the following Exception:

Exception in thread "main"
com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:111)

By Googling this error, some people are saying the file is corrupt.

Is there something I am missing when trying to write this kind of binary data to file programmatically?

Here's some code showing two ways I am trying to download the file:

Technique 1

fileURL = "https://api.mapbox.com/v4/mapbox.mapbox-traffic-v1/17/36159/54906.vector.pbf?style=mapbox://styles/fnembhard/ck9inh9df002c1jpnccjg20pw@00&access_token=..."

public void saveFile1(String fileURL, String fileName){
    try{
        URL testURL = new URL(fileURL);
        if(testURL.getHost() != null) {
            InputStream in = new URL(fileURL).openStream();
            Files.copy(in, Paths.get(fileName), StandardCopyOption.REPLACE_EXISTING);
        }
    }
    catch(Exception e){
    e.printStackTrace();
    }
}

Technique 2

public void saveFile2(String fileURL, String fileName){
    try {
        URL testURL = new URL(fileURL);
        if (testURL.getHost() != null) {
            InputStream in = new URL(fileURL).openStream();
//                File f = new URL(fileURL).getFile();
            OutputStream outputStream = new FileOutputStream(fileName);
            int byteRead;
            while ((byteRead = in.read()) != -1) {
                outputStream.write(byteRead);
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

score 0 · Answer 1 · answered May 11 '20 at 13:39

0

Could you please provide how you are parsing the protobuf file?

To parse a protobuf file you need to know its schema: Generic Parsing of PB in java

If the bytes you are providing to FileDescriptorSet.parseFrom to populate a FileDescriptorSet are the binary protobuf contents (compiled schema) this should work.

If not, which is probably the case here, you would need a runtime .pbf parser. Unfortunately the Java Protobuf library does not have a way to load .pbf directly.

You can get a compiled schema by using the protoc command-line-tool with the --descriptor_set_out option.

Also see here: Protocol Buffers: How to parse a .proto file in Java

answered May 11 '20 at 13:39

Moritz

1,710
1
8
13

I am using the ElectronicChartCentre library: github.com/ElectronicChartCentre/java-vector-tile. ```InputStream inputStream = new FileInputStream(fileName); VectorTileDecoder d = new VectorTileDecoder(); VectorTileDecoder.FeatureIterable features = d.decode(inputStream.readAllBytes()); ``` I don't believe it is a parsing issue. I think it's the way I am downloading the file. If I manually download the file, I am able to parse it without any errors. When I compare the contents of a manually downloaded file with the one downloaded using Java code, both are different – Fitz May 14 '20 at 14:49

How to download Mapbox PBF file using Java?

1 Answers1