The File That Copied Itself but Still Crashed

Problem

A production OCR worker started failing every batch with PermissionError: [Errno 1] Operation not permitted — on a directory where every file was rwxrwxrwx. Stranger still: the file it "failed" to copy was sitting right there on disk, fully written.

app.engine - ERROR - Batch processing failed: [Errno 1] Operation not permitted:
'/batches/TB20260806.00016/tm000001.pdf'
 
Traceback (most recent call last):
  File "/app/app/routes.py", line 45, in create_batch
    ocr_engine.create_batch(
  File "/app/app/engine.py", line 641, in create_batch
    shutil.copy(pdf_path, new_pdf_path)
  File "/usr/lib64/python3.11/shutil.py", line 432, in copy
    copymode(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.11/shutil.py", line 313, in copymode
    chmod_func(dst, stat.S_IMODE(st.st_mode))
PermissionError: [Errno 1] Operation not permitted: '/batches/TB20260806.00016/tm000001.pdf'

The offending line looked completely innocent:

shutil.copy(pdf_path, new_pdf_path)

Context

The document-processing platform runs as a set of services via docker-compose on a Linux VM. The OCR engine's worker consumes jobs from RabbitMQ, stages the input PDFs into a batch folder on a shared volume, and processes them. The staging copy is pure plumbing — generated filenames in a working directory that nobody downstream inspects for anything but bytes.

Investigation

First check: permissions on the target directory. From the host, everything was wide open:

drwxrwxrwx  2 ubuntu ubuntu     10 Aug  6 08:42 .
-rwxrwxrwx  1 ubuntu ubuntu 287013 Aug  6 11:18 tm000001.pdf
-rwxrwxrwx  1 ubuntu ubuntu 361186 Aug  6 08:42 tm000001.tif

Every file 777, owned by ubuntu (uid 1000). A permission error in a world-writable directory made no sense at first glance.

Second check: who is the container? Inside the worker container, the same directory listed the owner as a bare 1000 1000 — the container had no ubuntu user, and more importantly, the worker process itself was running as a different user than the one owning the files.

Third clue — and the one that cracked it: the timestamp on tm000001.pdf says 11:18, which is exactly when the error fired. The file was successfully written by the very operation that "failed." The copy worked. Something after the copy blew up.

That's when the traceback stops looking like noise and starts telling the story. The error isn't raised in a write — it's raised in copymode, inside chmod_func:

File ".../shutil.py", line 432, in copy
    copymode(src, dst, follow_symlinks=follow_symlinks)
File ".../shutil.py", line 313, in copymode
    chmod_func(dst, stat.S_IMODE(st.st_mode))

Root Cause

Two Unix facts collided here.

Fact 1: shutil.copy is two operations, not one. The shutil copy family differs only in how much metadata each variant drags along:

So shutil.copy(src, dst) writes the bytes, then calls os.chmod(dst, mode_of_src).

Fact 2: chmod requires ownership, not write permission. These are different privileges. A 777 file lets anyone write to it, but only the file's owner (or a process with CAP_FOWNER, i.e. root) may change its mode. The worker could create and write the file just fine — and then failed to chmod the file it had just created.

Why would a process fail to chmod its own freshly created file? Because /batches is an NFS mount with root squash. The container ran as root, and on an NFS export with root_squash (the default), the server remaps root to an unprivileged anonymous user (nobody). From the NFS server's point of view:

• the write succeeded — the directory is 777, anyone can create files there;
• the chmod failed — the file is owned by uid 1000 on the server, and the squashed client identity doesn't own it and has no CAP_FOWNER.

Root inside the container. Nobody on the wire. EPERM on the chmod.

The failure had been latent in the code forever. It only surfaced when the deployment moved to a topology where the worker's effective uid stopped matching the owner of the share — the classic "worked on my machine, worked on the last three environments" bug.

Solution

The staging copy only ever needed the file contents — nobody downstream reads the staged file's permission bits. So the fix is to stop asking for the metadata copy:

# Not shutil.copy — its chmod on the new file raises
# PermissionError when /batches is a network share we don't own.
shutil.copyfile(pdf_path, new_pdf_path)

One line. The comment matters almost as much as the change: shutil.copy is what everyone reaches for by default, and without the comment, a future refactor would "normalize" it right back and silently reintroduce the crash.

Technical Decisions

There was a second valid fix at the infrastructure layer: run the container as the uid that owns the share (user: "1000:1000" in docker-compose). Then the chmod succeeds because the worker is the owner. I prototyped it — it works — but it drags along its own baggage:

• the base image's directories (/app, model caches) are laid out for a different uid, so it needs supplementary groups and a writable HOME;
• it fixes this one deployment, while the same code also runs on Kubernetes and OpenShift, where pods get arbitrary uids and the same latent bug would still be waiting.

The code fix is strictly smaller and covers every deployment at once. When a bug can be fixed either in code or in infrastructure, prefer the layer where the fix is smallest and travels with the code.

Lessons Learned

• shutil.copy ≠ "copy a file." It's copyfile + chmod. If you only need the bytes — and you usually do — use copyfile. Reach for copy/copy2 only when preserving permissions or timestamps is an actual requirement.
• Write permission and chmod permission are different things. 777 means anyone can write; only the owner can chmod. Any code that copies into storage it doesn't own (NFS, CIFS/SMB, mounted volumes, uploaded-files directories) should treat metadata operations as optional.
• Root inside a container is not root on a network share. NFS root_squash demotes root to nobody at the server. If a privileged process gets EPERM on a filesystem operation, check whether the path is a network mount before questioning your sanity.
• Read the traceback frame by frame, not just the last line. The error message pointed at a file path; the frames pointed at copymode → chmod. That distinction was the entire diagnosis.
• Check the timestamps. A "failed" operation that leaves a fresh artifact behind failed after the part you assumed. The mtime on the "uncopied" file was the single most informative clue in this incident.