How to Implement Atomic File Writing in Python (No Partial Writes)

Apr 5, 2026

Python programming

My Python service crashed mid-write and corrupted the config file. The original data was gone, and the new data was incomplete. I had to restore from backup.

The Problem

I was writing a configuration file like this:

with open('config.json', 'w') as f:
    json.dump(data, f)

This is fine until something goes wrong:

Process crashes mid-write
Disk runs out of space
Power fails
System reboots

Any of these leaves you with a partial file. The original content is overwritten, and the new content is truncated.

Before:  {"database": "postgres", "port": 5432, "host": "localhost"}
After:   {"database": "postgres", "por
                                              <- TRUNCATED!

Why This Happens

When you open(file, 'w'), Python truncates the file immediately. Then it writes content incrementally. Between truncation and completion, your file is in an inconsistent state.

Time    File State           Risk Level
─────────────────────────────────────────────
T0      Original content     Safe
T1      TRUNCATED!           DANGER - original gone
T2      Partial new data     DANGER - incomplete
T3      Complete new data    Safe again

The window between T1 and T3 is your vulnerability window.

The Solution: Write-Then-Rename Pattern

The fix is to never write directly to your target file:

┌─────────────────┐
│  Write to temp  │  ← If crash here: temp file left behind
│     file        │     Original file is UNTOUCHED
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Sync to disk  │  ← Ensure data is actually written
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Atomic rename   │  ← POSIX: all-or-nothing operation
│  temp → target  │     Either succeeds completely or fails
└─────────────────┘

On POSIX systems (Linux, macOS), os.rename() is atomic. The file either has the old name or the new name - never something in between.

My First Attempt

I implemented this pattern myself:

import os
import tempfile
from contextlib import contextmanager

@contextmanager
def atomic_write(filepath, mode='w', encoding='utf-8'):
    """Context manager for atomic file writes."""
    directory = os.path.dirname(filepath) or '.'
    fd, temppath = tempfile.mkstemp(
        dir=directory,
        prefix='.atomic_write_',
        suffix='.tmp'
    )

    try:
        with os.fdopen(fd, mode, encoding=encoding) as f:
            yield f
            f.flush()
            os.fsync(f.fileno())  # Critical: ensure data hits disk

        # Atomic rename
        os.rename(temppath, filepath)

    except Exception:
        # Clean up temp file on failure
        if os.path.exists(temppath):
            os.unlink(temppath)
        raise


# Usage
with atomic_write('config.json') as f:
    json.dump(data, f)

This worked on Linux. But I made several mistakes.

Mistake #1: Wrong Temp Directory

Initially I used /tmp:

# WRONG - different filesystem!
fd, temppath = tempfile.mkstemp(suffix='.tmp')

os.rename() only works atomically within the same filesystem. Across filesystems, it falls back to copy-then-delete, which is NOT atomic.

Fix: Always create temp file in the same directory as target:

directory = os.path.dirname(filepath) or '.'
fd, temppath = tempfile.mkstemp(dir=directory, suffix='.tmp')

Mistake #2: Forgetting fsync

I initially skipped os.fsync():

# WRONG - data may still be in OS cache!
with os.fdopen(fd, 'w') as f:
    f.write(content)
    # Missing: f.flush() and os.fsync()
os.rename(temppath, filepath)

Without fsync, the data might live only in the OS page cache. A power failure before cache flush means data loss.

Fix: Always sync before rename:

with os.fdopen(fd, 'w') as f:
    f.write(content)
    f.flush()
    os.fsync(f.fileno())  # Force data to disk
os.rename(temppath, filepath)

Mistake #3: Windows Compatibility

On Windows, os.rename() fails if the target exists:

Linux/macOS: os.rename() replaces existing file (atomic)
Windows:     os.rename() raises FileExistsError if target exists

Fix: Use os.replace() on Windows:

if os.name == 'nt':  # Windows
    os.replace(temppath, filepath)
else:  # POSIX
    os.rename(temppath, filepath)

The Complete DIY Solution

Here’s my refined implementation:

import os
import tempfile
from contextlib import contextmanager
from typing import Generator, IO

@contextmanager
def atomic_write(
    filepath: str,
    mode: str = 'w',
    encoding: str = 'utf-8'
) -> Generator[IO, None, None]:
    """
    Context manager for atomic file writes.

    Writes to a temporary file, then atomically renames on success.
    Guarantees that filepath is never in a partially-written state.

    Args:
        filepath: Target file path
        mode: File mode ('w' for text, 'wb' for binary)
        encoding: Text encoding (ignored for binary mode)

    Yields:
        File object for writing

    Example:
        with atomic_write('config.json') as f:
            json.dump(config, f)
    """
    # Create temp file in same directory (required for atomic rename)
    directory = os.path.dirname(os.path.abspath(filepath))
    fd, temppath = tempfile.mkstemp(
        dir=directory,
        prefix='.atomic_write_',
        suffix='.tmp'
    )

    try:
        # Open temp file for writing
        if 'b' in mode:
            f = os.fdopen(fd, mode)
        else:
            f = os.fdopen(fd, mode, encoding=encoding)

        with f:
            yield f
            f.flush()
            os.fsync(f.fileno())

        # Atomic rename (cross-platform)
        os.replace(temppath, filepath)

    except Exception:
        # Clean up temp file on any failure
        try:
            os.unlink(temppath)
        except OSError:
            pass
        raise

Using the `safer` Library

For production code, I eventually switched to the safer library. It handles edge cases I would miss:

import safer

# Simple usage - caches in memory by default
with safer.open('config.json', 'w') as f:
    json.dump(data, f)
    # File only written on successful context exit

# For large files - uses temp file instead of memory
with safer.open('large_data.bin', 'wb', temp_file=True) as f:
    f.write(large_data)

The library provides features I didn’t implement:

Feature	My DIY	safer Library
Lines of code	~50	1
Memory caching	No	Yes
Temp file mode	Always	Optional
Windows support	Manual	Automatic
Permission handling	Basic	Comprehensive
Error handling	Manual	Tested

Install with:

pip install safer

When to Use Which Approach

Use DIY when:

You want zero dependencies
You’re learning how atomic writes work
Your requirements are simple

Use safer when:

Writing production code
Handling large files (memory caching option)
Running on multiple platforms
You don’t want to debug file operations

POSIX Atomicity Guarantee: The POSIX standard guarantees that rename() is atomic with respect to other filesystem operations. This is why the write-then-rename pattern works.

Database WAL: SQLite and PostgreSQL use a similar pattern (Write-Ahead Logging). They write changes to a separate log file first, then checkpoint to the main database.

Journaling Filesystems: Ext4, XFS, and APFS use journaling to provide similar guarantees at the filesystem level. But your application still needs atomic writes for consistency.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 safer - Python package for safer file writing
👨‍💻 Reddit discussion: Atomic write context manager
👨‍💻 Python tempfile documentation
👨‍💻 Linux rename() system call - atomic behavior

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!