Stack Overflow
Stack Overflow

Questions tagged [spark-dotnet]

Questions pertaining to usage of Apache Spark (and related distributions) in the context of Microsoft's .NET runtime and associated languages such as C# and F#. Feel free to add platform-specific and language-specific tags as well.

Tag Definition

Tag is used for questions pertaining to usage of Apache Spark (and related distributions) in the context of Microsoft's .NET runtime and associated languages such as C# and F#.

Related platform/code offerings

.NET for Apache Spark is currently an open-source offering at the .NET Foundation. See https://github.com/dotnet/spark and https://dot.net/spark for details.

Refinement Usage of Tag

You can refine the tag's usage by adding tags narrowing down the relevant Apache Spark related distribution and services and the specific language(s) relevant to the question.

10 questions
6
votes
4 answers

Submit a Spark job from C# and get results

As per title, I would like to request a calculation to a Spark cluster (local/HDInsight in Azure) and get the results back from a C# application. I acknowledged the existence of Livy which I understand is a REST API application sitting on top of…
Stefano d'Antonio
  • 5,874
  • 3
  • 32
  • 45
6
votes
1 answer

Apache spark queries through C#

I was wondering if there is a way I can use C# to write queries to run on Apache spark. I know spark SQL queries can be written in java/scala/python. Is there any interface for c#?
Groot
  • 311
  • 4
  • 15
2
votes
2 answers

Create dataframe from C# List - Spark for .NET

I am currently new to .NET for Spark and need to append a C# list to a delta table. I assume I first need to create a Spark DataFrame to do this. In the sample code how would I go about appending "names" to the dataframe "df"? It seems now this has…
ow123
  • 21
  • 1
  • 2
0
votes
1 answer

.Net for Apache Spark error from DataFrame.Show after UDF called

I'm a long time C# programmer but just getting my feet wet with .Net for Apache Spark. Following many "getting started" instructions and videos, I installed: 7-Zip Java 8 I downloaded Apache Spark from https://spark.apache.org/downloads.html .NET…
Dogulas
  • 25
  • 5
0
votes
1 answer

convert spark-submit command (dotnet for spark app) to spark-submit command for python app

If the following (working) spark-submit command (for a dotnet for Spark app) was executing a python script, would it still use the same --conf settings? Given a python script name of myapp.py that has no defined function (other than main), what…
Mike Williams
  • 1
  • 3
0
votes
0 answers

Apache spark dataframe to csv export in c#

Apache spark newbie here. I am querying a very large data set from ADLS using Apache Spark for .net. After querying my data, i want to convert the dataframe to a csv file and send it to an api that consumes the csv file. I have the…
Bonaii
  • 75
  • 6
0
votes
0 answers

C# Apache Spark orc file path exist on adls

Spark newbie here. I've got a large set of data that is collected and stored in a folder respective to the date it occurred on on ADLS. Each folder is named according to the date (example: <2020-12-04>). I am trying to query the most recent data…
Bonaii
  • 75
  • 6
0
votes
1 answer

iterate over spark dataframe column in C#

I am using the microsoft.spark version 1.0.0 to process a parquet file that is received from the network. I have mapped the parquet file into a Dataframe and i am trying to strip the alias from the userid column which is stored as an email address…
Bonaii
  • 75
  • 6
0
votes
1 answer

.Net for Apache Spark authentication against ADLS (Azure datalake store) gen 1

I am new to apache spark. I am trying to use Microsoft apache nuget library to read data from ADLS. I cant seem to figure out how i can authenticate using spark. There seems to be no documentation around this at all. Is this even possible? I am…
Bonaii
  • 75
  • 6
0
votes
1 answer

Method not implemented exception on Take method in Microsoft.Spark

I am trying to setup spark with the new Microsoft.Spark library. The method DataFrame.PrintSchema works fine, however the method DataFrame.Take() gives an System.NotImplementedException. Allot of other methods also give this exception. I took a look…
Jan-Wiebe
  • 61
  • 2
  • 11