4

Within Palantir Foundry platform, I am working in Data integration. I need some help as I am new to Palantir software. Is there any documents, white-papers, links or tutorials on this topic?

How do I integrate data from another source, for example Amazon S3 or HDFS?

fmsf
  • 36,317
  • 49
  • 147
  • 195
Saurabh Chakraborty
  • 101
  • 1
  • 11

2 Answers2

6

Palantir usually makes available on the customer company intranet a Q&A forum to help developers and address issues.

Moreover they provide also on-site support and there is on Foundry workspace home page a section called Training & learning, providing examples and tutorial to get started.

Matteo Guarnerio
  • 720
  • 2
  • 9
  • 26
5

To integrate data from another Platform you'll need a source and a sync in data connection. You'll need to have platform permissions to create these, not all users can since it can involve the organisation data governance policies.

Assuming you don't have a source with a valid configuration for S3. You'll need to create one. On Data Connection, click "Sources" and then click "New Source". You can then do this in two ways:

  • Use prebuilt S3 source: Click File System in the new New Source drop down and follow the wizard steps
  • Use a custom connector such as magritte-rest: Click Custom in the same dropdown.

For the magritte-rest:

  • Select either one of the available agents, or Cloud ingest depending on your preferences
  • Give it a name and save it into a folder.
  • Add configuration like:
type: magritte-rest
url: 'https://foobar.organization.s3.amazonaws.com'

Now to create the Sync, use a configuration similar to this:

type: rest-source-adapter
method: GET
path: the/path/in/s3/yourdata
outputFileType: csv

Other output file types are also supported (json, zip, ...)

fmsf
  • 36,317
  • 49
  • 147
  • 195