Internalizing the Batch Scheduler: The OnQuartz Development Journey

To reduce the operational cost and customization limitations of an external Batch Scheduler, we built an in-house Batch scheduler called OnQuartz. We designed it to keep the existing service flow and Batch project code as unchanged as possible, mounting the Scheduler on the existing servers while ensuring stability.

As Scheduler candidates, we reviewed Spring Scheduler and Quartz Scheduler, but chose Quartz because it officially supports clustering and HA. We adopted MISFIRE_INSTRUCTION_DO_NOTHING to prevent duplicate execution, and attached Slack notifications so that the person in charge could decide whether to rerun a job. We also tuned clusterCheckinInterval to 5 seconds and misfireThreshold to 60 seconds, so that delays of up to 60 seconds during a leader change would be handled normally by the new Pod.

The final structure breaks down into three stages.

• Scheduler → Batch execution request: Quartz calls the Batch execution API, and any job whose status is RUNNING is SKIPped to prevent duplicate execution
• Common library → Batch execution: Responds immediately to the Scheduler, calls the Batch API asynchronously, and periodically sends alive messages and results via Kafka
• Scheduler → Result processing: Consumes the Topic, and if no message arrives for a certain period, marks the job as FAIL to prevent it from staying RUNNING indefinitely

Deployment was built around a BLUE-GREEN environment, and we solved leader election accordingly. We referenced the current instance information in the Quartz table and used a DB Lock to look up the leader, ensuring the server with the latest Tag became the leader. The leader check runs every N seconds, so the leader moves to Standby if it drops out and resumes when it comes back.

During operation, we also ran into a number of exceptions. Using @Async without a dedicated Executor made tracing difficult in the common library, so we had to use an explicit Executor and a TaskDecorator to copy the MDC of the calling thread. We also introduced the concept of TTL to prevent duplicate execution when jobs piled up in the Queue during a server outage and then ran again after recovery — we limited the Queue size and compared the request time with the current time so that any job past its TTL would be skipped immediately.

Finally, there was also a problem where a Batch job failed but its status remained stuck as RUNNING. This happened because, in some MSAs, if the AOP pre-logic failed, the library's afterJob would never run, so only alive messages kept being sent. This case remains an open task requiring stronger guarantees for afterJob through exception handling and structural changes.

Internalization started in March, went through about 2 months of testing, and by September reached 100% conversion across the entire company. Along with reducing external solution costs, we also built a dashboard for batch management and monitoring, creating a more flexible Batch environment.