sqoop vs Spark for data ingestion

Asked Oct 15 '21 at 17:09

Active Oct 15 '21 at 17:27

Viewed 64 times

If I run a sqoop query over RDBMS, say "select id,name from customer where country='US'". In this case sqoop will run the query in RDBMS and fetch only the filtered record which will then be split across to number of mappers we defined.

I want to understand how it will work if run the same query in spark over RDBMS? Filtration (country=US) will happen in RDBMS side or in spark memory? My understanding is filtration should happen in RDBMS but some articles say that filtration will happen in spark memory after fetching all records. Please clarify.

edited Oct 15 '21 at 17:27

mazaneicha

8,794
4
33
52

asked Oct 15 '21 at 17:09

user3625945

1

How about trying it out and look at your database logs to see if the `WHERE` query is included or not? – OneCricketeer Oct 15 '21 at 17:11
1

Or, search SO for an answer? This is rather old but still a good start : https://stackoverflow.com/questions/32573991/does-spark-predicate-pushdown-work-with-jdbc – mazaneicha Oct 15 '21 at 17:26
Quality is going down – thebluephantom Oct 16 '21 at 08:29

sqoop vs Spark for data ingestion

0 Answers0