Spring 4 Quartz 2 Weblogic 12c Oracle 12c - Jobs occasionally triggered on both nodes in cluster

Question

We are using Spring 4.2.8, Quartz 2.2.3, Weblogic 12c as application server and Oracle12c as RDBMS. We have a clustered environment with 2 nodes part of the weblogic cluster. We are seeing that occasionally the quartz job is being triggered on both the nodes in cluster.

The spring quartz configuration is given below. Please advice if there is any configuration which may need to be tweaked to prevent the job from firing from both the servers at the same time -

    <beans:bean id="sendJobId"
        class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
        <beans:property name="jobClass" value="GenericSchedulerJob" />
        <beans:property name="jobDataAsMap">
            <beans:map>
                <beans:entry key="batchProcessorName" value="sendJob" />
            </beans:map>
        </beans:property>
    </beans:bean>

    <beans:bean id="sendJobTrigger"
        class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
        <beans:property name="jobDetail" ref="sendJobId" />
        <beans:property name="cronExpression" value="0 0 9 * * ? *" />
        <beans:property name="misfireInstructionName" value="MISFIRE_INSTRUCTION_DO_NOTHING" />
    </beans:bean>

    <beans:bean id="schedulerFactoryBean"
            class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
            <beans:property name="autoStartup" value="true" />
            <beans:property name="applicationContextSchedulerContextKey"
                value="applicationContext" />
            <beans:property name="dataSource" ref="testDB" />
            <beans:property name="waitForJobsToCompleteOnShutdown"
                value="true" />
            <beans:property name="quartzProperties">
                <beans:props>
                    <beans:prop 
     key="org.quartz.scheduler.skipUpdateCheck">true</beans:prop>
                    <beans:prop key="org.quartz.scheduler.instanceName">EFF-Scheduler</beans:prop>
                    <beans:prop key="org.quartz.scheduler.instanceId">AUTO</beans:prop>

                    <beans:prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool
                    </beans:prop>
                    <beans:prop key="org.quartz.threadPool.threadCount">20</beans:prop>
                    <beans:prop key="org.quartz.threadPool.threadPriority">5</beans:prop>
                    <beans:prop key="org.quartz.jobStore.misfireThreshold">60000</beans:prop>
                    <beans:prop key="org.quartz.jobStore.tablePrefix">QRTZ_</beans:prop>
                    <beans:prop key="org.quartz.jobStore.isClustered">true</beans:prop>
                    <beans:prop key="org.quartz.jobStore.clusterCheckinInterval">20000</beans:prop>
                    <beans:prop key="org.quartz.jobStore.selectWithLockSQL">SELECT * FROM {0}LOCKS UPDLOCK WHERE
                        LOCK_NAME = ?
                    </beans:prop>
                    <beans:prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate
                    </beans:prop>
                    </beans:props>
            </beans:property>  

  <beans:property name="triggers">
        <beans:list>
            <beans:ref bean="sendJobTrigger" />
        </beans:list>
    </beans:property>

The Datasource configuration is given below -

<jee:jndi-lookup id="testDB"
        jndi-name="java:/comp/env/jdbc/testDB" />

Following quartz related tables are available in Oracle DB -

QRTZ_CRON_TRIGGERS
QRTZ_FIRED_TRIGGERS
QRTZ_JOB_DETAILS
QRTZ_LOCKS
QRTZ_SCHEDULER_STATE
QRTZ_SIMPLE_TRIGGERS
QRTZ_TRIGGERS

Below are the entries in QRTZ_LOCKS table -

schedulerFactoryBean    TRIGGER_ACCESS
schedulerFactoryBean    STATE_ACCESS

The contents of GenericSchedulerJob.java are provided below -

@Component
public class GenericSchedulerJob extends QuartzJobBean {

    private String batchProcessorName;

    private static final Logger LOG = LoggerFactory.getLogger(GenericSchedulerJob.class);

    @Override
    protected void executeInternal(JobExecutionContext jobContext) throws JobExecutionException {
        ApplicationContext context;
        try {
            context = (ApplicationContext) jobContext.getScheduler().getContext().get("applicationContext");
            BaseSchedulerJob job = (BaseSchedulerJob) context.getBean(batchProcessorName);
            job.execute(Integer.toHexString(hashCode()));

        } catch (SchedulerException e) {
            LOG.error("ERROR :" + e.getMessage(), e);
        }

    }
    public String getBatchProcessorName() {
        return batchProcessorName;
    }
    public void setBatchProcessorName(String batchProcessorName) {
        this.batchProcessorName = batchProcessorName;
    }

}

    @Component
    public class SendJob implements BaseSchedulerJob{
@Autowired
    private SendService sendService;
public void execute(String processId) {
sendService.sendPendingPickRequests(job);
}
}

The actual quartz job is fetching data from DB , creating an excel file with the data retrieved from DB and emailing the same to recipient.

Based on the logs it appears like while the job on one server is in processing of writing data to the excel file, the job on other server is starting.