0

The incorrect operation has detected the possible error and throwed the following warning:

UnsupportedOperationChecker: Detected pattern of possible 'correctness' issue due to global watermark. The query contains stateful operation which can emit rows older than the current watermark plus allowed late record delay, which are "late rows" in downstream stateful operations and these rows can be discarded. Please refer the programming guide doc for more details.;

Since I am pretty sure the join operation is correct, how can I suppress the warning?

Oli
  • 9,766
  • 5
  • 25
  • 46
Eljah
  • 4,188
  • 4
  • 41
  • 85
  • I believe this is being tried on spark-shell , one way to achieve there is using `sc.setLogLevel('ERROR')` where sc is the sparkcontext object. this will prevent any warn messages – SHASHI SHEKHAR Barnwal Nov 23 '21 at 16:26
  • @SHASHISHEKHARBarnwal thank you. What if I need to suppres only this warning, but not others? – Eljah Nov 23 '21 at 16:36
  • 1
    Not pretty sure as I have not been able to find any scenario where we filter only one type of warning, but many other aplication level logs setting there are answers which might be useful [link](https://stackoverflow.com/questions/27781187/how-to-stop-info-messages-displaying-on-spark-console) – SHASHI SHEKHAR Barnwal Nov 24 '21 at 04:37

0 Answers0