1

Wanted to send a file to an http end point url using Data fusion. Making this http call as a pipeline alert at the completion of the pipeline.

enter image description here

enter image description here

This is not working. Getting 500 response from API. Can someone help me on how do I send the file?

If this can be achieved with http plugin as sink, please let me know the steps.

aruna j
  • 91
  • 5
  • Which datafusion version are you using? It might version mismatch. Could you provide full error log? Did you follow any tutorial/guide? – PjoterS Sep 14 '21 at 10:26
  • DataFusion version -6.1.3. – aruna j Sep 14 '21 at 11:20
  • Did you try to use never version of Data fusion? Could you provide full error message? – PjoterS Sep 15 '21 at 13:27
  • From the information you included above, it seems like you are attempting to send the GCS resource URL to the HTTP endpoint. Is that intended? If you wish to send the actual file contents, you will need to read the file using a source (e.g. GCS Source), convert the contents to base64, then send the value to the endpoint. – Dennis Li Sep 20 '21 at 20:52

1 Answers1

1

I doubt you can send files using HTTP. You can create a request using the HTTP plugin as I answered one of your previous question - Data Fusion - Issue with http post plugin.

In HTTP Callback you can find information:

This action can be used when you want to perform an HTTP request at the end of a pipeline run.

Using this plugin, you can send a request or text, not a file.

If you would like to send a file, you should use File like it's mentioned in this thread: Some questions about google Data fusion

Regarding Error 500

The HyperText Transfer Protocol (HTTP) 500 Internal Server Error server error response code indicates that the server encountered an unexpected condition that prevented it from fulfilling the request.

It means that there is an issue with your endpoint. It might be a firewall or HTTP server issue. Too many possibilities of what could be wrong with current details.

In addition, you can find some additional information in doc Ingest CSV data to BigQuery using Cloud Data Fusion - Batch ingestion

PjoterS
  • 12,841
  • 1
  • 22
  • 54