1

I need to pull the data from Azure Sentinel in an Incremental manner.

E.g:

  • step 1: Need a daily login details to my UI from Sentinel(using KQL)
  • step 2: create a pipeline from ADF
  • step 3: Load the data in tables

Is there any Lync Services or connectors available for Azure Sentinel or Azure Log Analytics to connect Azure Data Factory?

Erik Oppedijk
  • 3,496
  • 4
  • 31
  • 42
subham das
  • 9
  • 1

2 Answers2

1

There are two methods witch depend on the method of authentication to the API.

The first is with a service principle, High level steps: Import Azure Monitor log data into Azure Data Factory Blog on the topic: https://datasavvy.me/2020/12/24/retrieving-log-analytics-data-with-data-factory/comment-page-1/#comment-28467

Second is with Managed Identity:

  1. first give ADF access to Log Analytics using IAM How can I use this API in Azure Data Factory
  2. Then connect to Log Analytic API with Web activity or a copy activity (these are the two i got working).

Web Activity

enter image description here

URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/query

Body: {"query":"search '*'| where TimeGenerated >= datetime(@{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(@{pipeline().parameters.it_endDate}) | distinct $table "}

Copy Activity

First the linked service. enter image description here

ADF Datasets: enter image description here

Base URL: URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/

Copy Source: enter image description here

Body: { "query": "@{item()[0]} | where TimeGenerated >= datetime(@{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(@{pipeline().parameters.it_endDate})" }

Additional: The body code above, gets a list of the table names in log analytics using the web activity. Which I then pass to the Copy Activity to exports copy of the data for each table.

Aaron C
  • 301
  • 1
  • 8
0

We have used the below Kusto query , to pull the list of login that where happened on a particular computer during the last one day

SecurityEvent
| where  TimeGenerated  >= ago(1d)
| where Computer == "<vmname>"

You can refer this documentation , for more sample kusto query queries.

Is there any Lync Services or connectors available for Azure Sentinel or Azure Log Analytics to connect Azure Data Factory?

No we don't have any direct connectors available to connect Azure log analytics workspace or Azure Sentinel with Azure data factory.

if you want to use the log analytics workspace/Azure sentinel data in ADF you need to export the data to either storage account(blob storage) or Azure Events hubs & load that data from blob storage to data factory as explained in this documentation.

You can refer these documentations to export the data from the Azure log analytics workspace to storage account , Azure sentinel data to azure storage account

VenkateshDodda
  • 4,723
  • 1
  • 3
  • 12
  • Only issue with the default Log analytics export to storage account is that you can't export custom tables (I suspect this is partly due to a naming issue, but that is sidetracking). So if you have custom tables.. you end up having to create a second export process (messy!). – Aaron C Jul 25 '22 at 22:53