Stack Overflow
Stack Overflow

Questions tagged [azure-synapse-analytics]

89 questions
2
votes
1 answer

PySpark with SynapseSQL throws error when using queries with restricted keywords in column name

I'm quering SQL Database on Azure Synapse Analytics server with PySpark. However, some of my column names may contain restricted keyword(s). Here, I've prepared a generic example which I could bypass with changing the name of c.Name as something…
Lukasz
  • 269
  • 4
  • 12
2
votes
1 answer

Unable to saveAsTable in pyspark dataframe : apache.spark.sql.delta.schema.InvariantViolationException error: Exceeds char/varchar type length

I am trying to merge 5 dataframes from csv's and creating new dataframe than in the second step I am creating empty table with custom schema, now I am want to load the records from the dataframe. Here is the step by step details. creating…
Vikram Singh Yadav
  • 33
  • 4
1
vote
1 answer

Azure Synapse Pipeline Get Metadata Last Modified Date on file instead of on folder

I am trying to get the Last Modified date on the files (CSV linked service) in my repository using Get Metadata activity. Field List However, I am getting the same date/time for all of the files... the LastModified date on the folder instead of the…
CFCJB
  • 11
  • 4
1
vote
1 answer

Problems in UDF function in Spark 3.3 (scala)

We are changing some code from spark 2.4 to spark 3.3 and we are having problems with our UDF function. We generate the JAR (after modifying the library versions in the POM) with all the logic and run it from Azure in an Apache Spark Pool. When we…
jmarco10
  • 451
  • 1
  • 7
  • 21
1
vote
1 answer

Write delta table to ADLS from Azure Synapse Pyspark notebook

I'm tying to write delta formatted table to ADLS Gen2 from Azure Synapse Pyspark notebook with Serverless SQL pool. It is throwing me error, while writing to ADLS Gen2 as below. Py4JJavaError: An error occurred while calling o3898.save. : Operation…
subro
  • 1,167
  • 4
  • 20
  • 32
1
vote
1 answer

regexMatch and regexReplace functions in Azure Synapse Analytics are not yielding results in simple SELECT statements

I'm trying to execute simple SELECT statements in dedicated SQL Pool's script in Azure Synapse. One of those were: SELECT regexMatch('abctest','^[A-Za-z]*$'); This statement should yield return True because it tests whether the string contains only…
AAA6
  • 67
  • 3
  • 9
1
vote
1 answer

Load MongoDB data incrementally through Azure data factory

I would like to load data from MongoDB incrementally to Azure storage using Azure data factory. I couldn't find any relavent documentation to do this..Appreicate if there is a way to achieve this with Azure data factory. I have already checked the…
Suresh
  • 9
  • 3
1
vote
1 answer

Execute Serverless SQL Pool external table / Views from Synapse notebook using Python

I have a Serverless SQL pool which contains external tables/Views write against ADLS Gen2 parquet files. I want to call external table / views from synapse notebook using python. FYI. Server less SQL Pool Identity is Azure Active directory. Can…
Sunny
  • 13
  • 4
1
vote
0 answers

How to make use of custom jar in Synapse Notebook after uploading it in Azure Synapse Workspace?

I am trying to add my customized jar in Azure Synapse Workspace to make use of user defined function (udf) present in the jar while running the sql query in Synapse Notebook. An Example: There is udf named as MapCloud() registered in UDFHelper.scala…
Pranay Ramtekkar
  • 11
  • 3
1
vote
2 answers

Is it possible to create temporal table in Azure synapse datawarehouse

I m trying to create temporal table in azure synapse datawarehouse. CREATE TABLE dbo.Department ( DeptID INT NOT NULL PRIMARY KEY CLUSTERED , DeptName VARCHAR(50) NOT NULL , ManagerID INT NULL , ParentDeptID INT NULL , ValidFrom…
Vaishnavi S
  • 25
  • 1
  • 5
1
vote
1 answer

How can I connect to Azure Synapse Dedicated SQL pool ( SQL DW) from powershell using service principal authentication?

I want to deploy my SQL scripts to synapse from powershell using Invoke-Sqlcmd utility We have disable username/password authentication and the only way to do it is by using Managed Identity/AD Service Principal. How can I leverage Powershell to…
azuresnowflake1
  • 135
  • 1
  • 10
1
vote
1 answer

Access dedicated SQL Pool from Synapse Analytics notebook

I am new to synapse analytics and I want to create a notebook, which could be used further as a databrick in the pipeline in order to process the data in one of the tables from the DB. I want to do everything in PySpark, so I am wondering how to…
Eli
  • 51
  • 5
1
vote
1 answer

Azure Synapse Analytics Dedicated SQL Pool: How can I create an external table as select ( CETAS ) and partition by CustomerID?

I can create CETAS tables in Azure Blob storage fairly easily ( link ), however, is there a simple way to automatically partition by CustomerID so that the rows associated with each customer will end up in their own folder? Essentially I'm looking…
Colorado Techie
  • 1,302
  • 1
  • 13
  • 21
1
vote
1 answer

Get a list of all Synapse notebook names in Azure Synapse Analytics

In Azure Synapse Analytics, I'm trying to create a code that returns a list with all notebook names. In databricks I verified that it is possible by these two solutions: https://docs.databricks.com/dev-tools/api/latest/workspace.html#example Get…
coding
  • 135
  • 2
  • 9
1
vote
1 answer

@@ Variables not working in Synapse Dedicated SQL pool (@@error, @@rowcount, etc.,)

We are working on DW to Synapse migration. In that I am migrating an Stored Procedure which having @@error, @@rowcount variables. @@ Variables not working in Synapse Dedicated SQL pool (@@error, @@rowcount, etc.,) Any other ways to replace these…
Sankar M
  • 71
  • 6
1
2 3 4 5 6