Stack Overflow
Stack Overflow

Questions tagged [delta-rs]

delta-rs provides low level access to Delta tables in Rust, which can be used with data processing frameworks like datafusion, ballista, polars, vega, etc. It also provides bindings to other higher level language Python.

delta-rs provides low level access to Delta tables in Rust, which can be used with data processing frameworks like datafusion, ballista, polars, vega, etc. It also provides bindings to other higher level language Python.

Repo: https://github.com/delta-io/delta-rs

9 questions
2
votes
4 answers

How to read from Azure Blob Storage with Python delta-rs

I'd like to use the Python bindings to delta-rs to read from my blob storage. Currently I am kind of lost, since I cannot figure out how to configure the filesystem on my local machine. Where do I have to put my credentials? Can I use adlfs for…
user7454972
  • 218
  • 3
  • 13
1
vote
2 answers

How to add a new column when writing to a Delta table?

I am using delta-rs to write to a Delta table in the Delta Lake. Here is my code: import time import numpy as np import pandas as pd import pyarrow as pa from deltalake.writer import write_deltalake num_rows = 10 timestamp = np.array([time.time() +…
Hongbo Miao
  • 45,290
  • 60
  • 174
  • 267
1
vote
1 answer

connection error when accessing deltatable in s3 using delta-rs python deltalake pkg

I am trying to read a delta table in s3 using delta-rs deltalake lib , but to access the s3 I need to pass aws_access_key_id , aws_secret_access_key and a ssl certificate. with below code(without using a certificate) from deltalake import…
Scarface
  • 359
  • 2
  • 13
1
vote
0 answers

Reading a Delta Lake table from a S3 Bucket

I'm trying to use the library delta-rs to read some delta tables from a S3 bucket, but I'm not able to get them from the bucket. Here is the snippet of my code: #[tokio::main] async fn main() -> Result<(), Box> { let credentials =…
Evandro Lippert
  • 336
  • 2
  • 11
0
votes
0 answers

how to build delta-rs python for hdfs support

i would build delta-rs v0.10.1 with HDFS support. for this purpose i have costomized Cargo.toml file and add hdfs feature then building and installing it in environment using make install command. features = ["azure", "gcs", "python", "datafusion",…
Mohammad Esmaeili
  • 81
  • 6
0
votes
0 answers

How to append to delta table using Rust?

I have this Python code that executes 3 insert transactions into a delta table import pandas as pd from deltalake.writer import write_deltalake from deltalake import DeltaTable if __name__ == '__main__': # First transaction id_list = [] …
Finlay Weber
  • 2,989
  • 3
  • 17
  • 37
0
votes
0 answers

Connect to delta table in hdfs using python without using Pyspark

I have a delta table in hdfs stored as a hive table. I need to connect to the table and load the latest version of the table. I was able to connect to hdfs using pyarrow library. But it is loading entire versions on the hdfs. Here is my code import…
Josin Mathew
  • 45
  • 2
  • 9
0
votes
0 answers

Possible bug in using Pyarrow is_null function with delta tables

I've noticed an issue while trying to apply filters on Pyarrow datasets initialised from delta tables. Specifically, the is_null expression predicate only seems to return rows if all the rows in the particular partition/parquet file have null values…
Rohith Paul
  • 3
  • 1
0
votes
0 answers

Error when trying to generate a Delta table from a Parquet file using delta-rs library

I'm attempting to write a Delta table without employing Spark, and I've chosen to use the delta-rs library. I've encountered an issue when trying to generate a Delta table using a Parquet file. Here is the error message I get: thread 'main' panicked…
Evandro Lippert
  • 336
  • 2
  • 11