Stack Overflow
Stack Overflow

Questions tagged [amazon-s3-select]

Amazon S3 Select enables applications to retrieve only a subset of data from an Amazon S3 object by using simple SQL expressions.

See: SQL Functions - Amazon Simple Storage Service

91 questions
66
votes
7 answers

What is difference between AWS S3 Select and AWS Athena?

I am trying to understand what is difference between AWS Athena service and the newly released S3 select (still in preview). How are use cases different for both of those? It seems both help in selecting partial data from S3.
user3444718
  • 1,485
  • 6
  • 22
  • 32
28
votes
2 answers

Can S3 Select search multiple objects?

I'm testing out S3 Select and as far as I understand from the examples, you can treat a single object (CSV or JSON) as a data store. I wanted to have a single JSON document per S3 object and search the entire bucket as a 'database'. I'm saving each…
Nic Cottrell
  • 9,401
  • 7
  • 53
  • 76
13
votes
3 answers

Streaming in / chunking csv's from S3 to Python

I intend to perform some memory intensive operations on a very large csv file stored in S3 using Python with the intention of moving the script to AWS Lambda. I know I can read in the whole csv nto memory, but I will definitely run into Lambda's…
Ajjit Narayanan
  • 632
  • 2
  • 8
  • 18
11
votes
0 answers

SKIP/OFFSET/ScanRange of rows in AWS S3 Select

I wonder if anyone figured out how to skip rows in S3 Select? SELECT S.* FROM s3object S SKIP 100 LIMIT 200 --or SELECT * from s3object s LIMIT 5, 10 --or SELECT * from s3object s limit 5 OFFSET 10 It looks like you can limit number of records…
Alex B
  • 2,165
  • 2
  • 27
  • 37
11
votes
1 answer

does aws s3 select work with multiple files?

I'm using S3 select to query a object from a file in S3 bucket but wondering if I can do it with multiple files or if its limited to just one. I wrote one query from a file using aws lambda but when I tried writing another one for a different file…
runner05
  • 123
  • 2
  • 5
10
votes
1 answer

How S3 select pricing works? What is data returned and scanned in s3 select means

I have a 1M rows of CSV data. select 10 rows, Will I be billed for 10 rows. What is data returned and data scanned means in S3 Select? There is less documentation on these terms of S3 select
bharath reddy
  • 101
  • 2
  • 4
9
votes
1 answer

AWS S3 SELECT WebUI error: "Quoted record delimiter found in the file"

I am trying to the Select From for CSV in AWS S3 SELECT Web UI, but I get the following error msg: Quoted record delimiter found in the file. To allow quoted record delimiters, please set AllowQuotedRecordDelimiter to 'TRUE'. But there is no option…
Saqib Ali
  • 3,953
  • 10
  • 55
  • 100
9
votes
3 answers

AWS CLI for S3 Select

I have the following code, which is used to run a SQL query on a keyfile, located in a S3 bucket. This runs perfectly. My question is, I do not wish to have the output written over to an output file. Could I see the output on the screen (my…
rogerwhite
  • 335
  • 4
  • 16
8
votes
3 answers

S3 Select CSV Headers

I am using S3 Select to read csv file from S3 Bucket and outputting as CSV. In the output I only see rows, but not headers. How do I get output with headers included. import boto3 s3 = boto3.client('s3') r = s3.select_object_content( …
thotam
  • 941
  • 2
  • 16
  • 31
6
votes
2 answers

How to get list of all columns from a parquet file using s3 select?

I have a parquet file stored in S3 bucket. I want to get the list of all columns of the parquet file. I am using s3 select but it just give me list of all rows wihtout any column headers. Is there anyway to get all column names from this parquet…
CodeHunter
  • 2,017
  • 2
  • 21
  • 47
5
votes
0 answers

S3 Select with Presto

I am trying out S3 Select from Presto using hive connector and Minio Object store. I am able to create an external table and run all the SQL queries. But, S3 Select does not seem to be working, even with the hive.s3select-pushdown.enabled=true set…
Shreyas Bhagavath
  • 51
  • 2
5
votes
1 answer

Parse expression with binary and unary operators, reserved words, and without parentheses

I'm trying to parse expressions made of the binary operator +, the unary operator not and identifiers that can be any alphabetical string that isn't not from pyparsing import ( CaselessKeyword, Combine, Word, alphas, opAssoc, …
Michal Charemza
  • 25,940
  • 14
  • 98
  • 165
5
votes
3 answers

Amazon S3 Select From not working

Amazon S3 has a new feature called select from which allows one to run simple SQL queries against simple data files - like CSV or JSON. So I thought I'd try it. I created and uploaded the following CSV to my S3 bucket in Oregon (I consider this…
Ken Krueger
  • 1,005
  • 14
  • 26
5
votes
3 answers

s3-select querying data on field name

I am trying to query data from json data in s3-select. { person = [ { "Id": 1, "Name": "Anshu", "Address": "Templestowe", "Car": "Jeep" } { "Id": 2, "Name":…
yogas
  • 189
  • 1
  • 3
  • 17
5
votes
1 answer

Aggregate functions in S3 Select

Amazon's S3 Select allows a user to write SQL against S3 objects, but there's insufficient documentation around what standard SQL functionality is supported. According to the documentation, Amazon S3 Select supports AVG, COUNT, MAX, MIN, and SUM.…
Kirk Broadhurst
  • 27,836
  • 16
  • 104
  • 169
1
2 3 4 5 6 7