Kusto data ingestion from an Azure Function App ends with a 403

Question

I try to ingest data from azure function app into a ADX database. I followed the instruction found in the the article here.

The difference is, I'd like to insert data into the table. I struggle with a 403 error "Principal 'aadapp=;' is not authorized to access table"

What I did: I have created a AAD App with the following API permissions: AAD App configured permission

I configured the database via Kusto Explorer:

.add database myDB ingestors ('aadapp=;') 'theAADAppname'

.add table PressureRecords ingestors ('aadapp=;') 'theAADAppname'

.add table TemperatureRecords ingestors ('aadapp=;') 'theAADAppname'

My code:

 var kcsbDM = new KustoConnectionStringBuilder($"https://ingest-{serviceNameAndRegion}.kusto.windows.net:443/").WithAadApplicationKeyAuthentication(
            applicationClientId: "<my AD app Id>",
            applicationKey: "<my App Secret from Certificates & secrets>",
            authority: "<my tenant Id>");

        using (var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(kcsbDM))
        {

            var ingestProps = new KustoQueuedIngestionProperties(databaseName, tableName);
            ingestProps.ReportLevel = IngestionReportLevel.FailuresAndSuccesses;
            ingestProps.ReportMethod = IngestionReportMethod.Queue;
            ingestProps.JSONMappingReference = mappingName;
            ingestProps.Format = DataSourceFormat.json;

            using (var memStream = new MemoryStream())
            using (var writer = new StreamWriter(memStream))
            {
                var messageString = JsonConvert.SerializeObject(myObject); // maps to the table / mapping 
                writer.WriteLine(messageString);
                writer.Flush();
                memStream.Seek(0, SeekOrigin.Begin);

                // Post ingestion message
                ingestClient.IngestFromStream(memStream, ingestProps, leaveOpen: true);
            }

the steps you've taken _seem_ to be ok. It's impossible to debug this without additional context (e.g. the _full_ exception / error message you get) - if you don't want to include those on this SO thread, please consider opening a support ticket for your cluster via the Azure portal — Yoni L., Mar 16 '20 at 16:42

score 0 · Answer 1 · answered Mar 17 '20 at 07:07

The issue is that the mapping you are using in this ingestion command does not match the existing table schema (it has additional columns). In these cases Azure Data Explorer (Kusto) attempts to add the additional columns it finds in the mappings. Since the permission that the app has is 'ingestor', it cannot modify the table structure and thus the ingestion fails.

In your specific case, your table has a column that is written in a specific casing and in the ingestion mapping the same column has a different casing (for one character) so it is treated as a new column.

We will look into providing a better error message in this case.

Jean · Accepted Answer · 2020-03-18T14:53:37.483

Update: the issue is fixed in the system and now it works as expected.

Avnera thanks for your hint, potential it is an issue because of the Real vs double translation. In one of my first try I used double in the table and that worked. That is not longer possible, looks the supported data types changed.

My current configuration:

.create table PressureRecords ( Timestamp:datetime, DeviceId:guid, Pressure:real )

.create-or-alter table PressureRecords ingestion json mapping "PressureRecords"
'['
'{"column":"TimeStamp","path":"$.DateTime","datatype":"datetime","transform":null},'
'{"column":"DeviceId","path":"$.DeviceId","datatype":"guid","transform":null},'
'{"column":"Pressure","path":"$.Pressure","datatype":"real","transform":null}'
']'

public class PressureRecord
{
    [JsonProperty(PropertyName = "Pressure")]
    public double Pressure { get; set; }
    [JsonProperty(PropertyName = "DateTime")]
    public DateTime DateTime { get; set; } = DateTime.Now;
    [JsonProperty(PropertyName = "DeviceId")]
    [Key]
    public Guid DeviceId { get; set; }
}

no, this error is not related to data types. The issue is that the column "Timestamp" in the table appears as "TimeStamp" in your mapping. Because column names are case sensitive we think that "TimeStamp" is a new column and we try to create it, and fail because ingestor does not have permission to change the table schema. — Avnera, Mar 17 '20 at 11:27
Perfect, thanks a lot, the permission issue is fixed. Unfort. I have the next issue with that: the messageString from my code example is "{"Pressure":97914.0,"DateTime":"2020-03-17T12:34:49.8888901+00:00","DeviceId":"f5693ab4-4550-4b4e-afc6-95a15c370078"}" but each operation ends with a "Stream_NoDataToIngest: Value '' used in a switch/case is invalid". Is it because of a wrong formatting if the string or a wrong order of the fields ? — Jean, Mar 17 '20 at 13:00

Kusto data ingestion from an Azure Function App ends with a 403

2 Answers2