The payment batch that stopped at 10 of 42

BBPortal runs a batch job that processes payments for a list of students in one go. When it works, nobody thinks about it.

One night the job processed 10 students out of 42 and then simply stopped. There was no crash and no error to point at. It didn't fail loudly; it went quiet.

A job that stops without an error is usually not failing; it is waiting. So the question changed from "what threw?" to "what is it waiting for?"

I traced the job's path for each student. Most of the work is local: reading records and preparing the charge. One step is not: the call out to the payment gateway.

The call to the payment gateway was blocking and had no timeout. When the gateway did not answer, the job waited for it indefinitely, and every student after the 11th was never reached.

The fix follows from the cause: a call to a system you don't control needs a timeout, and a batch needs to keep going when one item fails. Each payment can then fail on its own and be reported, instead of holding up the rest of the list.

A silent stop with no error became a named, explainable cause the team could act on.

Every external call needs a timeout and a failure path. Silent failures cost more than loud ones, because nobody starts looking.