1
I am using a Copy Data activity with a Source that is a REST API POST Method using body as a Kusto query. But while using POST method  collection only 1000 record showing as a web page limourceitation.
 
[Source][1]
[Source Dataset][1]

And Sink dataset is SQL database
The REST API endpoint looks something like this: 

{
  "totalRecords": 2145,
  "count": 1000,
  "data": [
    {

     }
  ],
  "facets": [],
  "resultTruncated": "false",
  "$skipToken": "ew0KICAiJGlkIjogIjEiLA0KICAiTWF4Um93cyI6IDEwMDAsDQogICJSb3dzVG9Ta2lwIjogMTAwMCwNCiAgIkt1c3RvQ2x1c3RlclVybCI6ICJodHRwczovL2FyZy13ZXUtdGhyZWUtc2YuYXJnLmNvcmUud2luZG93cy5uZXQiDQp9"
}

I know through powershell this is possible Like a Do-while loop until $skipToken is null, but in ADF not sure how to perform. Can anyone suggest a solution?

Siddharth
  • 11
  • 2

1 Answers1

-1

Azure Data Factory $Skiptoken (for Rest API POST Method) - Workaround?

From the information you provided we can conclude that you want to perform pagination on Rest API Till the the $Skiptoken is null to perform this Azure Data Factory, provides Pagination rules Where you can set the end condition for the pagination of Rest API

Reference Images:

  • My sample data enter image description here

  • EndCondition enter image description here

Here I am providing the EndCondition like when the $.page.number From the Output of Rest API Is Empty It Will stop that pagination.

Reference: Set end condition

Pratik Lad
  • 4,343
  • 2
  • 3
  • 11
  • In below link you will get to know what exactly I am looking for this is possible with powershell with "do while " process but not sure what need to be done in ADF... https://learn.microsoft.com/en-us/rest/api/azureresourcegraph/resourcegraph(2020-04-01-preview)/resources/resources?tabs=HTTP – Siddharth Dec 09 '22 at 10:54