Stack Overflow
Stack Overflow

Questions tagged [gcs]

237 questions
4
votes
1 answer

How to create an eventarc trigger in terraform for GCS?

I would like to create an eventarc trigger for GCS object creation. According to the Eventarc documentation, this should use the direct GCS trigger. I can create it like this, but I don't know where to put the bucket name: resource…
Florian Feldhaus
  • 5,567
  • 2
  • 38
  • 46
3
votes
3 answers

matching engine index creation failed

I want to create Matching Engine Index in Vertex AI, I was following this article step by step and got stuck in matching engine index…
user393014
  • 445
  • 1
  • 8
  • 15
3
votes
1 answer

Creating and loading Avro file, file gets created but empty

I am reading a CSV file and loading it into an Avro file in the GCS bucket. The Avro file gets created but there is no data. There is data when I print. I checked the buffer but there is no data in the buffer as well. I tried writer.close() but I am…
Ananya Dwivedi
  • 33
  • 2
3
votes
2 answers

How do I write a partitioned parquet file from a big query table?

I have created a parquet file from a big query table like this: EXPORT DATA OPTIONS( uri='gs://path_for_parquet_file/*.parquet', format='PARQUET', overwrite=false ) AS SELECT * FROM…
Eze M
  • 31
  • 3
2
votes
0 answers

Server error: Internal server error: module 'google.auth.credentials' has no attribute 'CredentialsWithTokenUri'

I am trying to create Data Proc cluster with following python packages. "PIP_PACKAGES": "google-cloud-bigquery==3.10.0 google-resumable-media[requests]==2.5.0 google-cloud-storage==2.8.0 google-cloud-secret-manager==2.16.1 google-ads==21.0.0" The…
Dhomse N
  • 23
  • 2
2
votes
2 answers

GCS cloud storage standard pricing

I am using GCS standard bucket. If I am writing 100 GB of data from multiple processes and deleting after fetching it for once, how does the storage cost work ? Let's say I have 5 processes that writes 100 GB each and deletes them after using the…
Ram
  • 61
  • 4
2
votes
0 answers

How to acess GCS private Bucket file using Load Balancer

I am trying to upload dist folder in GCS private bucket and trying to access javascript file using load balancer facing access denied error can any one help how to access a file in private bucket other than https://storage.cloud.google.com/ this url…
nicky
  • 21
  • 2
2
votes
1 answer

How to start Airflow Dag with a past Data Interval Date

I am working in Ariflow 2.2.3 and I can't figure out how to trigger my dag with a past execution date. When I click Trigger dag with Config, I changed the calendar to the date I wanted but when I clicked run, I saw the run but it didn't run. I also…
Maggie
  • 41
  • 1
  • 8
2
votes
1 answer

Dataproc: PySpark logging to GCS Bucket

I have a pyspark job running in Dataproc. Currently, we are logging to console/yarn logs. As per our requirement, we need to store the logs in GCS bucket. Is there a way to directly log to files in GCS Bucket with python logging module? I have tried…
Ravi kiran
  • 43
  • 5
2
votes
1 answer

Getting /bin/bash^M: bad interpreter: No such file or directory after running initialisation script on Dataproc

I am getting the below error when I am trying to create a Dataproc cluster in GCP. I am using a windows machine to make changes to GCS bucket which will be used to initialize the dataproc…
sidharth vijayakumar
  • 1,190
  • 5
  • 29
2
votes
0 answers

GCP Vertex AI Workbench custom image - persistence with gcs don't work

While creating a workbench with a custom jupyter image I choose backup/ persistence with GCS (google cloud storage) and it doesn't work. For now, I tried to test with: jupyter/base-notebook:python-3.8.8…
Santhin
  • 21
  • 1
2
votes
0 answers

Fluentbit to GCS

I was wondering if Fluentbit had a Google Cloud Storage (GCS) output plugin? I found this fluent-bit-go-gcs repo on github but not sure if this is still something that's being maintained. Any examples of Fluentbit configs that output to GCS/other…
skarsky
  • 63
  • 1
  • 6
2
votes
1 answer

How to use trino to get prefix path in GCS

I'm using Trino with Hive+GCS connector to access my JSON files inside GCS. If I direct the external table to the path with the JSON it works, like: create table transactions( date DATE, transaction1 DOUBLE, TransactionType…
Matheus Tramontini
  • 322
  • 1
  • 13
2
votes
2 answers

Can you trigger Python Scripts from Dataproc?

I am experimenting with GCP. I have a local environment with Hadoop. It consists of files stored on HDFS and a bunch of python scripts which make API calls and trigger pig jobs. These python jobs are scheduled via cron. I want to understand the best…
JYosen
  • 89
  • 7
2
votes
1 answer

Why does GCS say there are no outputs found in GCS-backend?

I'm trying to understand why my data from my gcs backend is saying it does not have any outputs. I have a module called DB which creates a postgres database. I have a file called outputs.tf, where I have terraform { backend "gcs" { bucket =…
jdkealy
  • 4,807
  • 6
  • 34
  • 56
1
2 3
…
15 16