Apache Spark RDD and Java 8: Exception handling

Question

I need to skip the record, if i get any exception while iterating the file content using Java 8 and Spark.

I do not want to throw exception, i just need to skip that record and continue with other records.

Code Example is :

JavaRDD<Model> fileRDD = sc.textFile("filePath")
                .map(line -> {
                    try {
                    String[] parts = line.split("\\|");
                    Long key = Long.parseLong(parts[0];
                    return line;
                    } catch (NumberFormatException nfe) {
                        //if i throw RuntimeException, its working file
                        //but i dont want to throw exception, i want to just skip the line,
                        // how do i do it using java 8 stream methods
                    }
                });

score 7 · Accepted Answer · answered Oct 06 '15 at 11:36

7

You can use filter instead of map:

JavaRDD<Model> fileRDD = sc.textFile("filePath")
            .filter(line -> {
                try {
                    String[] parts = line.split("\\|");
                    Long key = Long.parseLong(parts[0];
                    return true;
                } catch (NumberFormatException nfe) {
                    return false;
                }
            });

answered Oct 06 '15 at 11:36

Tagir Valeev

97,161
19
222
334

inside map i have to do lot of other works, so before doing map i need to filter then start process the records using map. – Shankar Oct 06 '15 at 11:44
1

@Shankar, sure, you can first `filter`, then `map`. – Tagir Valeev Oct 06 '15 at 11:49

score 1 · Answer 2 · edited May 23 '17 at 11:53

1

String[] parts = line.split("|");

The pipe character should be escaped.

String[] parts = line.split("\\|");

See: https://stackoverflow.com/a/9808719/3662739

edited May 23 '17 at 11:53

Community

1
1

answered Oct 06 '15 at 11:39

zeroTAG

65
8

actually i have escaped the pipe character..anyway thanks for that. – Shankar Oct 06 '15 at 11:43
oh, nvm then xD, I thought that would make the array break and that's why you couldn't skip. – zeroTAG Oct 06 '15 at 11:46
3

While this is a reasonable comment to the question, it's a *comment*, not an answer. Please avoid posting comments as answers. When you'll earn reputation 50, you will be able to post comments as well. – Tagir Valeev Oct 06 '15 at 11:46

Apache Spark RDD and Java 8: Exception handling

2 Answers2