Exstream output queue stalls when another process copies PDFs from the output directory

Symptom

The engine writes PDFs normally until a separate script starts copying finished files to another location. From that point most jobs sit queued in the output queue and throughput collapses. Stop the copy script and the queue drains again.

The copier does not have to be a script you wrote. Backup agents, antivirus scanners, and file indexers produce the same pattern, and those are harder to notice because nobody thinks of them as part of the output chain.

Cause

The engine and the copier contend for the same files while the engine is still writing or closing them. A copy process that grabs a file the moment it appears in the directory can hold a lock, or hold the file open long enough, that the engine's own writes and renames start blocking behind it.

Field reports from this situation suggested serializing the relevant processing with the -sync switch in the argument file. Be aware that this mechanism is not fully confirmed: the original case was resolved without establishing whether the root contention was filesystem locking, connector behavior, or the external watcher itself. Treat -sync as a candidate fix to test, not a documented guarantee.

Fix

The reliable fix is to stop copying files based only on their appearance in the directory. Give the handoff an explicit completion signal:

1. Let the engine write to a staging directory, or under a temporary filename.
2. Move or rename the file into the watched directory only after it is closed.
3. Point the copier at the watched directory, never at the staging area.

A rename on the same filesystem is atomic, so the copier can never see a half-written file.

If you want to test -sync instead, do it in a controlled copy of the job and measure queue latency, throughput, and file completeness before and after. If synchronous processing removes the stall, that is evidence the contention is real, but check the watcher and the filesystem before accepting the throughput cost of running synchronously in production.

Reproduce on the production filesystem type

When you diagnose this, record the output connector, the filesystem type (local versus network share matters a lot here), the argument file, and the observed lock behavior. Concurrent-access stalls behave differently on NFS and SMB than on local disk, and a fix that works in a test environment on local disk can fail in production on a share.