0

I have a job that applies some regex to a large number of strings. This I have achieved by creating a cross join job with List of strings and List of regex as the input.

Normally this runs fine, but once in a while for a certain input and regex pair, the task execution never terminates - mostly due to input being too big / regex not being efficient.

In this case I would prefer the task to get 'timed out', or the job as a whole to get 'timed out' so I know something is wrong and skip processing.

I went through the flink config docs but wasn't able to find out.

I did a workaround creating a future async thread inside task and cancelling it after a certain time to apply the regex, but it seems like an overkill. Hence looking for a better solution.

binit
  • 476
  • 1
  • 5
  • 15

1 Answers1

1

There is no job limit in Flink that can do what you look for. Since that sounds very batch-oriented, I'd also don't think that this is a feature that's actively worked on.

Nevertheless, your solution is actually quite good already. Other solutions depend on your infrastructure. If you trigger the job by using airflow or any other workflow system, I'd imagine that they can cancel tasks after some time. If you run it on K8s or YARN, you may be able to limit the total resource usage. But if you don't use any of it, then your solution is a good safe guard.

Some more ideas: do you really need the slow java Regexes or could you use Re2 or other automaton libraries? Could you add some sanity checks on very large input strings and skip them? Could you simply stop applying the CrossFunction after your time ran out (graceful termination)?

Arvid Heise
  • 3,524
  • 5
  • 11
  • Thanks for the pointers, as of now its a standalone cluster but I will also explore K8s. I don't want to use YARN because it requires installing hadoop, and I want to keep it lightweight. For now I am comfortable with java regexes because as long as they produce output it is fine, it shouldn't take infinite time (like now). I would prefer a generic task termination strategy because it can happen due to other unknown causes (eg bug in code). Hence for now will explore K8s and see if I can kill task upon timeout. I'm already going through its site, any other links would be helpful! – binit Aug 25 '20 at 20:17
  • Here is some related thread that may already solve things for you. There is a GNU command `timeout` https://stackoverflow.com/questions/44955786/how-to-set-a-time-limit-for-a-kubernetes-job – Arvid Heise Aug 26 '20 at 08:53