Using PHP to create a Pub/Sub notification for Google Cloud Storage

Question

So, I have a GKE-based function that does processing on an image file. I want this function to be triggered whenever a new file gets dumped into a bucket, using Pub/Sub. I have an external system that pushes the image file into the bucket. I was using object notifications to do this, now I want to use Pub/Sub.

Can I create a notification in GoogleCloudStorage that generates the Pub/Sub notification that my processing function will be pulling? I was looking at the PHP library (https://googleapis.github.io/google-cloud-php/#/docs/google-cloud/v0.162.0/storage/notification) to do this, but the documentation is ridiculously inadequate.

This code creates a notification, but seems strange that I should be supplying a notification ID, and I'm not sure what the payload is going to be...

$client=new StorageClient(['projectId'=><my project>, 'keyFile'=><key file contents>]);

$bucket_name = <my bucket name>;
$notification_id = '2482';  // ?????

$notification_params = [
        'eventType'     => 'OBJECT_FINALIZE',
        'payloadFormat' => 'JSON_API_V1',
        'bucketId'      => $bucket_name,
];

$bucket = $client->bucket( $bucket_name );

$notification = $bucket->notification( $notification_id );

Is this the correct way to create the notification I want? Do I specify my own id in this way? What happens if there is a collision?

thanks, andy

What do you mean at "if there is a collision"? Can you explain this as well? — Ricco D, Jul 06 '21 at 06:09
collision of the name specified in my code and the name of a topic already in existence? — Andy Wallace, Jul 06 '21 at 23:41
An error should appear if you defined a non existent topic in your code. — Ricco D, Jul 07 '21 at 02:04

score 0 · Answer 1 · answered Jul 06 '21 at 01:35

Yes it is possible to create a Pub/Sub notification when an object was pushed into the bucket.You need to first create a Pub/Sub topic and subscriber prior to creating the notification.

I used the sample code create pub/sub notification and just add defining of storage bucket:

use Google\Cloud\Core\Iam\PolicyBuilder;
use Google\Cloud\PubSub\PubSubClient;
use Google\Cloud\Storage\StorageClient;


$storage = new StorageClient();
$bucket = $storage->bucket('your-bucket'); // define bucket to be used

$pubSub = new PubSubClient();
$topicName = 'your-topic'; // define the topic to be used

// grant pubsub publisher role for topic
$serviceAccountEmail = $storage->getServiceAccount();
$topic = $pubSub->topic($topicName);
$iam = $topic->iam();
$updatedPolicy = (new PolicyBuilder($iam->policy()))
    ->addBinding('roles/pubsub.publisher', [
        "serviceAccount:$serviceAccountEmail"
    ])
    ->result();
$iam->setPolicy($updatedPolicy);

$notification = $bucket->createNotification($topicName, [
    'event_types' => [
        'OBJECT_DELETE',
        'OBJECT_METADATA_UPDATE'
    ]
]);

To test this, when the pub/sub notification is created:

Push an object in your bucket
Topic should send a message to the subscribers
Check message in the subscriber

Pushed object in test bucket:

Message/notification published to subscribers:

Sadly, it fails when trying to create the storage client with no parameters. If I specify my project and keyfile, it works up until it tries creating the PubSub client. Adding the project to that creates the PubSub client successfully, But then when I try creating the updated policy, I get: exception 'Exception' with message 'exception 'Google\Cloud\Core\Exception\BadRequestException' with message '{ "error": "invalid_grant", "error_description": "Bad Request" }' in /home/httpd/idxv3/vendor/google/cloud-core/src/RequestWrapper.php:368 — Andy Wallace, Jul 06 '21 at 23:51
It seems that your error might have a lot of possible causes as per [SO answer](https://stackoverflow.com/a/38433986/14733669). Can you also try setting the environment variable [GOOGLE_APPLICATION_CREDENTIALS](https://cloud.google.com/docs/authentication/production#passing_variable) in your machine? If none of the said solutions work, you can open another question for this issue so the community contribute some answers. — Ricco D, Jul 07 '21 at 01:58
But the docs from Google, and the error message specifically imply that I need to be using a Service Account to handle any sort of high load, which we're guaranteed to have. Putting the "accept this error" bandaid in place just pushes my need to get the service account auth working down the road. I'd rather knock that off now. — Andy Wallace, Jul 19 '21 at 18:10

Using PHP to create a Pub/Sub notification for Google Cloud Storage

1 Answers1