Python CSV KeyError: Fix a Hidden UTF-8 BOM in Headers

Your CSV visibly starts with a column called id. Yet the first csv.DictReader row raises KeyError: 'id', or row.get("id") returns None. If the file starts with the three bytes EF BB BF, the first field name can actually be '\ufeffid': it looks like id, but Python sees an extra Unicode character.

For a CSV known to be UTF-8, open it with `encoding="utf-8-sig"` and `newline=""`. Python's utf-8-sig decoder skips an optional leading UTF-8 byte-order mark (BOM). It also reads an ordinary UTF-8 file without that prefix. Check the first bytes and the parsed headers before changing your importer, because a different encoding or delimiter needs a different fix.

This guide is for Python developers importing CSV reports, spreadsheet exports and API extracts on Windows, macOS or Linux. The examples use Python's standard library, with a small synthetic fixture verified under Python 3.13.5 on 2 November 2026. No Excel installation, third-party library or real customer data was involved.

Where the invisible character comes from

A UTF-8 BOM is the three-byte signature EF BB BF at the *start* of a text file. It represents Unicode U+FEFF. UTF-8 itself has no byte-order ambiguity, so the marker is not required to decode its characters. Some tools nevertheless write it as a signature. The Unicode Consortium's BOM FAQ explains its role and why a leading marker can interfere with consumers that expect specific characters at the beginning of a file.

The Python codecs documentation makes the decoding distinction explicit:

• utf-8 decodes the leading bytes as the character U+FEFF; it does not silently delete that character.
• utf-8-sig skips the three-byte signature when it occurs at the beginning of the stream, and otherwise handles UTF-8 normally.

CSV adds a second step. csv.DictReader treats the first row as field names when you have not supplied your own. Those names are ordinary Python strings; it does not strip hidden Unicode markers or normalize column names. See the `DictReader` reference.

With a file that begins EF BB BF 69 64 2C, the 69 64 bytes spell id and 2C is the comma. Decode with utf-8 and the first key is '\ufeffid'. Decode the same bytes with utf-8-sig and it becomes 'id'. The payload bytes have not been edited; only the decoder's treatment of the initial signature changed.

Reproduce the bug without touching a real CSV

The following program writes a disposable two-row CSV with a deliberately inserted BOM, reads it twice, and removes the fixture when finished. Save it as bom_demo.py and run it using Python 3.10 or later.

import csv

from pathlib import Path

from tempfile import TemporaryDirectory

from codecs import BOM_UTF8

with TemporaryDirectory() as work:

    sample = Path(work) / "example.csv"

    sample.write_bytes(BOM_UTF8 + b"id,name\r\n17,Ana\r\n")

    for encoding in ("utf-8", "utf-8-sig"):

        with sample.open("r", encoding=encoding, newline="") as f:

            row = next(csv.DictReader(f))

            print(encoding, list(row), row.get("id"))

In the Python 3.13.5 run, the output was:

utf-8 ['\ufeffid', 'name'] None

utf-8-sig ['id', 'name'] 17

The first line is the trap: None is not evidence that the CSV's first data row contains an empty ID. The dictionary does contain 17, but under a different key. If the code had used row["id"] instead, the first read would have raised KeyError.

The second line shows the targeted fix. The CSV still contains the same three signature bytes; Python simply treats them as a file prefix rather than part of the first column name.

This demonstrates a specific UTF-8-with-BOM input. It does not establish how every spreadsheet exporter behaves or prove that all KeyError problems are encoding bugs.

Diagnose your file before choosing the decoder

Start with a copy of the supplied CSV. Keep the original, especially when an ingestion job also moves or deletes files after processing. Inspect only a few raw bytes, not the entire file:

from pathlib import Path

from codecs import BOM_UTF8

path = Path("report.csv")

with path.open("rb") as f:

    prefix = f.read(4)

print("First bytes:", prefix.hex(" "))

print("UTF-8 BOM:", prefix.startswith(BOM_UTF8))

For the synthetic fixture, the first line would begin with ef bb bf 69. The final byte, 69, is the letter i after the marker. For an ordinary BOM-free id,name header, the first bytes instead begin 69 64 2c 6e.

If the file has no BOM, look at the *actual header strings* with repr(reader.fieldnames). That helps reveal leading spaces, unexpected punctuation and a delimiter mismatch. A csv.DictReader opened with the wrong delimiter may treat an entire header line as a single field. A BOM fix would not repair that.

The following table keeps the diagnoses separate.

The Unicode Consortium's encoding signature table documents the different UTF-8, UTF-16 and UTF-32 prefixes. A prefix is helpful evidence, but a missing UTF-8 BOM does *not* identify the encoding of arbitrary text.

Make the importer reject an unexpected schema

Once you have confirmed that this feed is UTF-8 with or without a BOM, make its decoding and expected fields explicit. This prevents a silently misparsed header from turning into a long series of missing values.

import csv

from pathlib import Path

path = Path("report.csv")

required = {"id", "name"}

with path.open("r", encoding="utf-8-sig", newline="") as f:

    reader = csv.DictReader(f)

    headers = reader.fieldnames or []

    if len(headers) != len(set(headers)):

        raise ValueError(f"Duplicate CSV headers: {headers!r}")

    missing = required.difference(headers)

    if missing:

        raise ValueError(

            f"Missing required columns {sorted(missing)}; "

            f"received {headers!r}"

        )

    for row_number, row in enumerate(reader, start=2):

        if None in row:

            raise ValueError(

                f"Row {row_number} has more values than headers"

            )

        identifier = row["id"]

        name = row["name"]

        # Validate and process these values for your application.

This intentionally does not turn a decoding failure into success, does not coerce IDs into integers, and does not write anything back to the input file. It checks two common problems: duplicate header names and missing columns. It also rejects extra fields that DictReader places under the None key when there are more values than columns. The CSV library reference explains how extra and missing fields are represented.

For real production data, decide separately whether empty IDs, duplicate records, blank rows, extra columns and newline-containing fields are acceptable. A successful BOM decode is only the first layer of validation. If you have a stated data contract, check column order and data types against it as well.

Why newline="" belongs in these examples

Python recommends opening CSV file handles with newline="" so the CSV module can handle line endings correctly, including newlines inside quoted fields. This choice is independent of the BOM; it is part of the documented CSV reader and writer contract. Python's text I/O documentation explains that decoding and newline handling are separate text-stream operations.

When utf-8-sig is the wrong fix

A `UnicodeDecodeError` is not the same symptom as a hidden header key. If the byte stream is not valid UTF-8, selecting utf-8-sig will not make arbitrary bytes decode correctly. The codec only handles the optional leading UTF-8 signature; it is not a universal encoding detector.

Avoid treating errors="ignore" or errors="replace" as a data-import repair. They can discard or substitute characters, which may corrupt names, identifiers or values while making the script appear to finish. Use them only for a deliberate, documented loss-tolerant workflow, not as a blind fix for records that matter.

Likewise, do not routinely call lstrip() on every header to remove an unknown number of characters. It can erase meaningful leading spaces and can hide a different producer-side error. If your input contract permits trimming spaces, implement and test that rule explicitly. If the contract permits a UTF-8 BOM, utf-8-sig is narrower and clearer.

A BOM-like U+FEFF *inside* a field or halfway through the file is another issue. The UTF-8 signature codec does not promise to remove that character wherever it appears. Inspect the producer or concatenation process rather than stripping all occurrences and hoping the resulting data remains equivalent.

Writing CSV: choose a consumer contract

For a new machine-to-machine CSV feed, a straightforward choice is UTF-8 without a BOM, with the encoding and delimiter documented. Use the normal utf-8 codec for writing and newline="" for the CSV writer. Some spreadsheet workflows may specifically request a UTF-8 BOM for interoperability; in that case utf-8-sig on output writes the three-byte prefix. The Python codec reference documents that output behaviour, rather than assuming the CSV format itself requires the marker.

import csv

with open("export.csv", "w", encoding="utf-8", newline="") as f:

    writer = csv.writer(f)

    writer.writerow(["id", "name"])

    writer.writerow(["17", "Ana"])

If the receiving application *requires* a BOM, change the encoding to utf-8-sig and test that specific consumer. Do not switch every export globally based on one file that happened to open successfully.

Also separate encoding from spreadsheet formula interpretation. A BOM does not make untrusted cells safe to open in Excel. Jitsumi's published guide on preventing formula-like CSV values from becoming spreadsheet formulas addresses that distinct export boundary.

Verify the actual result

For the affected import, the pass condition is concrete: the file's beginning has been identified, the expected header is exactly 'id', row["id"] reads the intended value, and an unexpected schema still fails visibly. Repeat this with a BOM-free UTF-8 file too; utf-8-sig should read both formats.

If the same file opens differently in another tool, compare the bytes, not only the visual display. A spreadsheet can hide an initial signature while a program exposes it as part of the first key. In a reliable pipeline, the producer and consumer agree on encoding, delimiters, header spelling and permitted values.

Sources and further reading

Source documentation and the synthetic Python experiment were checked on 2 November 2026.

• Python standard library: `utf-8`, `utf-8-sig` and BOM handling.
• Python standard library: `csv.DictReader`, headers and newline handling.
• Python standard library: text I/O, binary I/O and explicit encodings.
• Unicode Consortium: UTF byte-order-mark signatures and limitations.