Why spark sql is adding where 1=0 during load?

Question

I am pretty new with spark. I have a task to fetch 3M record from a sql server through denodo data platform and write into s3. In sql server side it is a view on join of two tables. The view is time consuming.

Now I am trying to run a spark command as:

val resultDf = sqlContext.read.format("jdbc").option("driver","com.denodo.vdp.jdbc.Driver").option("url", url).option("dbtable", "myview").option("user", user).option("password", password)

I can see that spark is sending query like:

SELECT * FROM myview WHERE 1=0

And this portion is taking more than an hour.

Can anyone please tell me why the where clause is appending here?

Thanks.

Were you able to find something for this? – pri Sep 16 '21 at 11:08 — pri, Sep 16 '21 at 11:08

score 0 · Answer 1 · edited Dec 09 '16 at 06:16

If I'm understanding your issue correctly, Spark is sending SELECT * FROM myview WHERE 1=0 to the Denodo Server.

If that is the case, that query should be detected by Denodo as a query with no results due to incompatible conditions in the WHERE clause and the execution should be instantaneous. You can try to execute the same query in Denodo's VQL Shell (available in version 6), Denodo's Administration Tool or any other ODBC/JDBC client to validate that the query is not even sent to the data source. Maybe Spark is executing that query in order to obtain the output schema first?

What version of Denodo are you using?

score 0 · Answer 2 · edited Jan 23 '20 at 17:50

I see this is an old thread -- however we are experiencing the same issue -- however it does not occur all of the time nor on all connections/queries -

SQOOP command is sent -- the AND (1=0) context ('i18n' = 'us_est') is added somewhere --we are using Denodo 7 -- jdbc driver com.denodo.vdp.jdbc.Driver

select
   BaseCurrencyCode,BaseCurrencyName,TermCurrencyCode,TermCurrencyName,
   ExchangeAmount,AskRate,BidRate,MidMarketRate,ExchangeRateStartDate,
   ExchangeRateEndDate,RecCreateDate ,LastChangeDate
from 
   CurrencyExchange
WHERE 
  LastChangeDate > '2020-01-21 23:20:15' 
  And LastChangeDate <= '2020-01-22 03:06:19' 
  And (1 = 0) context ('i18n' = 'us_est' )

Why spark sql is adding where 1=0 during load?

2 Answers2