Getting GCS disk usage information via the Java API?

Question

I'd like to determine how much disk space my Google Cloud Storage (GCS) objects are using via GCS's Java API. Basically, I'd like something similar to Unix's df command:

>df
Filesystem        1K-blocks      Used Available Use% Mounted on
C:/Tools/cygwin64 248717308 217102536  31614772  88% /

This question discusses how to get GCS disk usage information using gsutil's du command. I would like to do something similar using the Java API, but haven't found anything at that level of granularity. I'm hoping I don't have to recursively process all of the objects in the container. Is there a better way?

score 1 · Answer 1 · answered Sep 21 '21 at 22:20

As the answers in that topic hit, the best option that doesn't require recursive enumeration is to fetch the storage.googleapis.com/storage/total_bytes metric. You can do that in Java with something like (modified from ):

MetricServiceClient metricServiceClient = MetricServiceClient.create();
# storage/total_bytes docs say this metric is emitted every 300s, so reading last 400 
long startMillis = System.currentTimeMillis() - (400000);
TimeInterval interval =
    TimeInterval.newBuilder()
        .setStartTime(Timestamps.fromMillis(startMillis))
        .setEndTime(Timestamps.fromMillis(System.currentTimeMillis()))
        .build();

ListTimeSeriesRequest.Builder requestBuilder =
    ListTimeSeriesRequest.newBuilder()
        .setName("")
        .setFilter("metric.type=\"storage.googleapis.com/storage/total_bytes\" AND metric.labels.gcs_bucket = \"bucketName\"")
        .setInterval(interval);

ListTimeSeriesPagedResponse response = metricServiceClient.listTimeSeries(requestBuilder.build());

System.out.println("Got timeseries: ");
for (TimeSeries ts : response.iterateAll()) {
  System.out.println(ts);
}

score 0 · Answer 2 · answered Sep 21 '21 at 19:35

0

There isn't API for that. Therefore, you need to get all the object metadata, get the size and aggregate the result.

In addition, take care to take into account all the object version, if your bucket is versioned.

Sorry, I broke your hope....

answered Sep 21 '21 at 19:35

guillaume blaquiere

66,369
2
47
76

score 0 · Answer 3 · answered Sep 28 '21 at 19:08

According to the Java and GCS (Google Cloud Storage) documentation, there is not a command that can be used to show the storage disk like the example you have with UNIX df.

Using java, you could write the following code to get the storage using these methods:

getTotalSpace(), getUsableSpace() and getFreeSpace()

It will depend on what kind of information you can get.

Code Example:

package example;

import java.io.File;

public class DiskSpaceDetail
{
    public static void main(String[] args)
    {   
        File file = new File("c:");
        long totalSpace = file.getTotalSpace(); //total disk space in bytes.
        long usableSpace = file.getUsableSpace(); ///unallocated / free disk space in bytes.
        long freeSpace = file.getFreeSpace(); //unallocated / free disk space in bytes.
        
        System.out.println(" === Partition Detail ===");
        
        System.out.println(" === bytes ===");
        System.out.println("Total size : " + totalSpace + " bytes");
        System.out.println("Space free : " + usableSpace + " bytes");
        System.out.println("Space free : " + freeSpace + " bytes");
        
        System.out.println(" === mega bytes ===");
        System.out.println("Total size : " + totalSpace /1024 /1024 + " mb");
        System.out.println("Space free : " + usableSpace /1024 /1024 + " mb");
        System.out.println("Space free : " + freeSpace /1024 /1024 + " mb");
    }
}

Output Example:

=== Partition Detail ===

=== bytes ===

Total size : 52428795904 bytes Space free : 33677811712 bytes Space free : 33677811712 bytes

=== mega bytes ===

Total size : 49999 mb Space free : 32117 mb Space free : 32117 mb

Note: Both getFreeSpace() and getUsableSpace() methods return the same total free disk space of a given partition.

Here is the documentation I reviewed for your awareness:

https://www.baeldung.com/java-google-cloud-storage

https://developers.google.com/api-client-library/java

Getting GCS disk usage information via the Java API?

3 Answers3