2

Using Argument setter by supplying the parameter value I want to make the Datafusion pipeline as resuable. As said by many other answer's have tried implementing using the cloud reusable pipeline example given in Google guide.I was not able to pass the parameter Json file.So how to create the API to that parameter Json file stored in Google storage.Please explain the values to be passed to Argument setter like URL,Request response etc., If any one of you had implemented in your projects.

Thank you.

Manasa
  • 31
  • 3
  • I have implemented by making the bucket public and was able to get the path for the API.But, any one let me know if this can be implemented without making the bucket as public .If yes, please let me know the solution of how to create the API end pint to the file in storage. – Manasa Mar 10 '20 at 06:49

4 Answers4

2

ArgumentSetter plugin reads from a HTTP endpoint and it must be publicly accessible as is depicted within the GCP documentation. Currently, there is not a way to read from a non-public file stored in GCS. This behavior has been reported in order to be improved to CDAP through this ticket.

Noe Romero
  • 390
  • 1
  • 6
1

Can you please provide what you've tried so far and where you're stuck?

The URL field in argument setter would contain the API endpoint you're making a call to. Make sure you include any headers your call would need like Authorization, Accept etc.

If you're having issues with argument setter a good check is to use Curl or any other tool to make sure you're able to talk to the endpoint you're trying to use.

Here's some documentation about Argument setter: https://github.com/data-integrations/argument-setter

itsanudeep
  • 26
  • 2
  • Am not able to create API end point initially.so here I made the bucket public and automatically I am able to see the end points value and copied it .If bucket is not public ,then how to create API end points.I am struck in creating API to the file stored in storage bucket. Any light on this would be helpful. – Manasa Mar 11 '20 at 06:35
0

Define a JSON file with appropriate name/value pairs. Upload it in a GCS bucket - make it public by changing permissions (add "allUsers" in permissions list). When you save it, the file will say "Public to Internet" Copy the https path to the file and use it in Arguments Setter. If you're able to access this path from curl/ your browser, Argument Setter will be able to do too..

There are other problems I've encountered while using Argument Setter though - the pipe doesn't supersede runtime arguments over default values provided in the URL many a times, specially when the pipe is duplicated.

To make file public

user13322341
  • 11
  • 3
0

You have to make your bucket public, currently there is no other way.

gsutil iam ch allUsers:objectViewer gs://BUCKET_NAME
Anna Christiane Kolandjian
  • 256
  • 2
  • 10