Quartz job getting triggered at wrong times

Question

Issue Description: We are having a quartz job which is scheduled to run at different times during the day say 1PM and 4PM and 7PM. Issue is that the job is getting triggered and executed on times other than the scheduled times also.

What we did: We have already tried shutting down our servers completely (JBoss) and clearing quartz tables but that hasn't worked at all.

We are using quartz 1.6 and I want to know, if there is any bug in the version and if quartz upgrade can resolve the issue OR If there is any problem with setting up quartz properties and if properties can be tweaked to resolve this.

Edit - more details below:

I have corrected the job timings now. Also, job is getting triggered at any random times other than these we mentioned in its schedule. There is no pattern in when it is getting triggered apart from the scheduled times.

Below are the job and trigger details in DB properties file. Based on these job details and trigger details will be set up in DB in Quartz_Triggers and Quartz_CronTrigger tables:

<job>
        <job-detail>
            <name>Match Job</name>
            <group>JB_QUARTZ</group>
            <job-class>com.qd.qehadmin.common.scheduler.MatchJob</job-class>
            <volatility>false</volatility>
            <durability>true</durability>
            <recover>true</recover>
        </job-detail>
        <trigger>
            <cron>
                <name>Match Job Trigger</name>
                <group>JB_QUARTZ</group>
                <job-name>match Job</job-name>
                <job-group>JB_QUARTZ</job-group>
                <cron-expression>0 0 13,16,19 * * ?</cron-expression>
            </cron>
        </trigger>
    </job>

Below are the quartz properties details in DB:

# Configure Main Scheduler Properties  
#============================================================================
org.quartz.scheduler.instanceName = JB_QUARTZ
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool  
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore  
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = true
org.quartz.jobStore.dataSource = jobSchedulerDS
org.quartz.jobStore.tablePrefix = JB_QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.jobStore.clusterCheckinInterval = 10000
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.jobSchedulerDS.jndiURL=java:JBAPI
org.quartz.dataSource.jobSchedulerDS.java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
#java.naming.provider.url=jnp://localhost:3099
java.naming.provider.url=jnp://166.20.337.12:8441,166.20.337.14:8441,166.20.337.16:8441,166.20.337.19:8441
#============================================================================

Below is the Java code for job:

public void execute(JobExecutionContext ctx) throws JobExecutionException
    {

        try{        
            //Scheduler scheduler = new StdSchedulerFactory().getScheduler();           
            SendEmail sm = new SendEmail();     
            boolean running = false;            

                if (ctx.getJobDetail().getKey().getName().equalsIgnoreCase("match Job") ) { 
                    LogFile.MATCH_JOB.logInfo("Match jobs size  : "+ctx.getScheduler().getCurrentlyExecutingJobs().size(), this.getClass().getName());  
                    if(ctx.getScheduler().getCurrentlyExecutingJobs().size()==1)
                    {
                    initClient(); //This method will read properties from DB                    
                    startTime=System.currentTimeMillis();                   
                    Match(); //This method will execute job level code  
                    endTime=System.currentTimeMillis();

                    LogFile.MATCH_JOB.logInfo("***Match job ends*** Loadtest: "+Constants.loadTest+" in time: "+(endTime-startTime)/1000 + "secs", this.getClass().getName());
                }
                    else
                    {
                        running=true;  
                    } 
                }


            if(running)  
            { 
                LogFile.MATCH_JOB.logInfo("The Match job is already running – sending email",this.getClass().getName());
                sm.sendEmail();
            }       

        }catch(Exception e){
            e.printStackTrace();
            LogFile.MATCH_JOB.logError("***Match Job Error*** " +e.getStackTrace(), this.getClass().getName());
        }
    }

It shouldn't be a bug of Quartz. More likely that you are doing something wrong. More than that is very hard to tell, since you didn't supply any of your relevant code and/or your configurations. — yishaiz, Sep 14 '17 at 08:58
Hi Yishaiz, Thanks for your reply. Please let me know what all details you would require on this. This issue is kind of baffling us for couple of months now and we are not able to get a hold around this at all. Thanks! — FindYourself, Sep 15 '17 at 13:28
Your Quartz properties file, and the code that is responsible for scheduling your job. Also you could add the [quartz-scheduler] tag which is much more popular. — yishaiz, Sep 16 '17 at 20:20
Also, you say that the job is " executed on times other than the scheduled times" - are these regular times? what are they? — yishaiz, Sep 16 '17 at 20:21
You can add all this info to the question itself, it will be easier to read — yishaiz, Sep 18 '17 at 14:30
Also, please add the [quartz-scheduler] tag, and answer my question about what are the times that your job is triggered — yishaiz, Sep 18 '17 at 14:31
Also, if I check your cron `0 0 7,13,19 * * ?`, e.g. at http://www.cronmaker.com/, then the times are 1:00 PM, 7:00 PM and 7:00 AM, which is not what you want according to what you wrote in your question. — yishaiz, Sep 18 '17 at 14:35
Hi Yishaiz, details have been updated in question. Please check and let me know, in case of further queries. Thank you much! — FindYourself, Sep 21 '17 at 11:04
Anyone having any pointers? Help would be greatly appreciated. Thanks! — FindYourself, Sep 27 '17 at 09:17
Just to be sure that I understand correctly, the job is triggered in all the required times, but **in addition** it also triggered at some other times? Can you write an example of times that the job was triggered of any particular day? Also, how do you determine what where the times that the job was triggered? logging by java/looking at the quartz tables/something else? — yishaiz, Sep 27 '17 at 18:46
Also, I see you are using `getCurrentlyExecutingJobs()`. See this answer https://stackoverflow.com/a/27395623/3883957 and that it says "*Also notice that this method is not cluster aware*" and also "*remember not to use it in Job class*", so maybe its causing problems. — yishaiz, Sep 27 '17 at 18:49
Also, try to look at the quartz tables after 1PM/4PM/7PM and see what is supposed to be the next time that the job will be triggered, whether its a correct value or not. — yishaiz, Sep 27 '17 at 18:52
Hi Yishaiz, sorry for coming back so late as I was on leave during this period. I looked at your suggestions and found them helpful but I am not able to understand the solution in the link you mentioned for not using the getCurrentlyExecutingJobs() method especially the below part && !jobCtx.getFireTime().equals(ctx.getFireTime())) .The jobCtx and ctx objects look same to me and how is it making difference in getting the firetime.Can you please explain. Thanks much in advance! — FindYourself, Oct 09 '17 at 15:30
Well it seems that ```ctx``` is the current job, and ```jobCtx``` is in a loop, of all the CurrentlyExecutingJobs, each time he compares another one to the current job. But that's not so important, the important thing is what I quoted - that the usage of ```getCurrentlyExecutingJobs``` is not cluster aware and *remember not to use it in Job class*. You can try, for example, to remove the usage of that and just see if that fixes the trigger times. Also, You didn't answer my previous questions, so helping more than that would be difficult for me. — yishaiz, Oct 09 '17 at 18:03
Hi Yishaiz, job timings are 0 0 13,16,19 * * ? i.e. 1PM, 4PM and 7PM only(see DB properties file details above). Yes, in addition to these times job is also getting executed at some random times for no apparent reason.I looked at the trigger and next fire time values are correctly getting updated in QUARTZ TRIGGER tables in DB for the job.Even if job gets to run at some other time than the regular ones I mentioned above, value is getting updated in table.But that is what we need to find out why it gets triggered at those wrong times.Also, there is no common pattern in this wrong triggering. — FindYourself, Oct 12 '17 at 09:24
Please note that this is a CRON TRIGGER set up in DB for the job. Also, when you say do not use getCurrentlyExecutingJobs() in Job class, do you suggest any alternative to check if a particular job is already running or not because as per requirement we have to put a check and dont run the job if it is already running. I can see in your provided link that they are also using the getCurrentlyExecutingJobs() method so how is it making difference and solving the issue. I think I did not miss answering anything now. Apologies and Thank you for bearing with me and being with me till now on this..:) — FindYourself, Oct 12 '17 at 09:35
Hi Yishaiz, others, any help/comments on the above. Thank you! — FindYourself, Oct 23 '17 at 08:43
Hi, 1. could you answer the question of *"Also, how do you determine what where the times that the job was triggered? logging by java/looking at the quartz tables/something else?"*. 2. You said that *"Even if job gets to run at some other time than the regular ones I mentioned above, value is getting updated in table"* - which value is getting updated, the wrong one? if so, then Quartz is the one that is scheduling the wrong time. Is that the case? 3. Just to be sure - you are working in cluster mode, right? — yishaiz, Oct 23 '17 at 17:43
4. Regarding "any alternative to check if a particular job is already running or not" - because you are in cluster mode, you have to check it with an API that reads from the db. Because any API that reads from the heap in your current JVM won't be effective because it doesn't know if some other instance is running the same job. So: [1] maybe this answer https://stackoverflow.com/a/31479434/3883957 will be good for you. [2] Or maybe there is another table in Quartz with an API that writes this information. [3] Or using something like Redis to distribute the information. — yishaiz, Oct 23 '17 at 17:49
[4] Or writing this into your own db which every job will read\write from. But anyway you are using it **inside** the job itself so maybe option [1] I gave you won't help. 5. Which leads me to the question - why do you need to check if the job isn't already running? As far as I know, Quartz should be the one responsible to invoke the job only once in the cluster, so why do you need that inside the job? — yishaiz, Oct 23 '17 at 17:53
6. Another question - which 1.6 Quartz are you using? 1.6.6 or something older? — yishaiz, Oct 24 '17 at 20:16
Hi Yishaiz, yes we are checking the timestamp values in quartz tables and in logs- logging by java both and this is in cluster mode only. We need to check if job is already running or not because in case the load is more and it is running till its next scheduled time, it should actually not kick off again but inform the team that it is already in progress. We are using quartz 1.6.0. — FindYourself, Oct 27 '17 at 13:51
If I understand correctly, you want to prevent overlapping job execution. If that is the case, then you can use `StatefulJob`, and Quartz will take care of that - see this answer https://stackoverflow.com/a/1638573/3883957. And then you won't need `getCurrentlyExecutingJobs`. Regarding Quartz 1.6 version, you may want to upgrade to at least 1.6.6 (if not to a newer version...), because it seems that 1.6.0 has many bugs. For example, see here what they say in the comments https://jira.terracotta.org/jira/browse/QTZ-311 — yishaiz, Oct 29 '17 at 06:30
Hi Yishaiz, Thanks! We have decided to go for below things in order to avoid this issue: 1) Upgrade the quartz version to at least 1.6.6 or newer 2) Use OracleDriver delegate for Oracle DB instead of StandardJDBCTemplate 3) Remove getCurrentlyExexutingJobs() method from code .Now I have two questions:Q1 How to prevent overlapping of jobs- We are using FIRED_TRIGGERS table to achieve this which keeps an entry of the job in database while it is executing. Would this be an OK approach? Q2 While we decide on version to upgrade, could you suggest us a version and what to take care while upgrading? — FindYourself, Nov 15 '17 at 17:08
1. I think that you should avoid as much as possible from direct access to Quartz tables, and to try to use Quartz APIs as much as you can. But if that's necessary and it solves the problem, it may be ok. But you should try to be sure you are not missing something in this table usage. But why won't you use the `StatefulJob`, anyway? — yishaiz, Nov 15 '17 at 17:40
2. I'm sorry but I don't know enough what are the differences between the versions and which version is best for you (of course the newest should be the best, but its more risky). There are some migration guides like http://www.quartz-scheduler.org/documentation/quartz-2.x/migration-guide.html — yishaiz, Nov 15 '17 at 17:40
Hi Yishaiz, We are migrating from version 1.6.0 to 2.0.2. Everything seems to be working fine like data is going into newly created tables in database but when jobs have to kick off actually, it is not able to do so and failing with below errosr while it is trying to fetch properties data from Oracle database: — FindYourself, Jan 22 '18 at 08:50
Hi, I recommend you to open a new question for that issue It's a different issue than in current question, and it will be easier to understand the issue, currently it's hard to understand the stacktrace. Also some other people might see the question and help you. after you post the new question it, you can write here a link to that question and I will be able to take a look as well. — yishaiz, Jan 22 '18 at 11:50
Hi Yishaiz, Thanks!. Here is the link for new question.https://stackoverflow.com/questions/48396851/error-in-lookup-property-in-quartz-2-0-2-with-oracle-db — FindYourself, Jan 23 '18 at 08:19
Hi Yishaiz, we recently upgraded to version 2.0.2 from 1.6.0 but still issue persists. I have opened a new question here https://stackoverflow.com/questions/51400442/quartz-job-getting-triggered-multiple-times-in-quartz-version-2-0-2. It would be great if it is possible to provide any insight. Thanks! — FindYourself, Jul 18 '18 at 11:09
Hi, sorry to hear that its still an issue for you. I think it would be great if you will add code samples (properties file, java, xml if needed) in the new question as you added in this question, so the new one would be clearer (also for other people, otherwise they won't read all these comments). — yishaiz, Jul 18 '18 at 11:52

Quartz job getting triggered at wrong times

0 Answers0