Why does my DAT file import with garbled characters?

Almost always an encoding mismatch. The þ text qualifier is a single byte (0xFE) in Windows-1252 but two bytes (C3 BE) in UTF-8. A reader assuming the wrong encoding looks for the wrong byte sequence, so field boundaries disappear and records shred or mojibake.

How do I tell which encoding a DAT file uses?

Check the first bytes for a BOM (EF BB BF = UTF-8, FF FE = UTF-16 LE), then try a strict UTF-8 decode of the first 64 KB, then look for 0xFE appearing at regular field boundaries (CP1252) versus C3 BE (UTF-8). Relativity's RDC only detects encoding from a BOM — without one it falls back to whatever default encoding was selected.

What encoding does Relativity expect for load files?

The Relativity Desktop Client auto-detects encoding only from a byte order mark and otherwise uses the encoding chosen in the import settings. Western European (Windows), UTF-8 and the UTF-16 variants are supported; the Concordance defaults are quote þ (254), newline ® (174), multi-value ; (59).

← LoadfileKit / Guides / DAT encoding trap

The DAT encoding trap

Why the þ delimiter breaks silently · a byte-level diagnosis guide

The most common "my DAT won't parse" ticket is not a delimiter problem at all. The delimiters are fine — the encoding is wrong, and the failure is silent: no error message, just shredded records or garbled custodian names after the load.

The same character, two different byte strings

The Concordance qualifier þ and line-break marker ® are single bytes in Windows-1252 and two bytes in UTF-8:

Character	Windows-1252 (CP1252)	UTF-8
þ (text qualifier)	0xFE — one byte	C3 BE — two bytes
® (in-value line break)	0xAE — one byte	C2 AE — two bytes

A parser that assumes the wrong encoding looks for the wrong byte sequence and misses every field boundary:

UTF-8 file read as CP1252. Every qualifier arrives as the two characters Ã¾ (C3 → Ã, BE → ¾). The parser finds no 0xFE, no delimiters, and the whole file looks like one unparseable line.
CP1252 file read as UTF-8. A lone 0xFE is not a valid UTF-8 start byte. Depending on the tool you get a decode error, a replacement character, or a silent skip — in every case the boundaries are gone.

Why it happens so often

A DAT has no encoding declaration. There is no BOM requirement, no header field, nothing in-band that says which codec produced the file. Older tools — Concordance itself and many vendor export pipelines built on Windows — default to CP1252. Newer cloud platforms increasingly emit UTF-8. When a file passes through production on one platform, QC on a second and import on a third, the encoding can be silently changed at any step.

The most dangerous variant is a mixed-encoding file: someone opens a CP1252 DAT in an editor that assumes UTF-8, makes a one-line edit, and saves. Now most of the file is CP1252 and a few re-saved lines are UTF-8. Per-record repair is the only cure, which is why it is worth catching before it ships.

Diagnosis, in order

Step	Check	Reading
1	First bytes of the file	EF BB BF → UTF-8 with BOM · FF FE → UTF-16 LE · none → keep going
2	Strict UTF-8 decode of the first ~64 KB	Succeeds with zero errors → almost certainly UTF-8
3	Look for the qualifier in the bytes	Lone FE appearing at regular intervals (field boundaries) → CP1252 with þ delimiters. C3 BE pairs at boundaries → UTF-8
4	Fallback scoring	Try remaining candidates (Windows-1252, GB18030 when custodians are Chinese) and keep the one where the delimiters survive and the text decodes cleanly

Any hex editor shows you steps 1 and 3 on the first line of the file. If you see FE 44 4F 43 ... it is CP1252; if you see C3 BE 44 4F 43 ... C3 BE it is UTF-8.

What the platforms expect

Relativity's desktop client detects encoding only from a byte order mark. No BOM means it cannot determine the encoding and falls back to the default encoding selected in the import settings — so a BOM-less UTF-8 file can load as Western European without any warning. Relativity's default delimiter set matches the Concordance convention: quote = Unicode 254 (þ), newline = Unicode 174 (®), multi-value = 059 (;). Other platforms differ; some image-load pipelines expect ANSI/Windows-1252 outright. A file with no BOM and no declared encoding is a guess waiting to happen — which is exactly why encoding is checked before anything else on load.

Fixing it

Re-encode the whole file in one pass into a single family (all-CP1252 or all-UTF-8), never patch records piecemeal.
Verify the delimiters survive after re-encoding: field count per record against the header, and þ/® present in the expected byte form.
Re-decode and spot-check names and paths — custodian names and file paths are where mojibake is visible, and a mis-decoded path silently stops resolving on import.
Test-load before shipping. A five-minute import on the target platform is cheaper than a rejected production.

The converter runs this exact detection on every load — BOM honor, strict UTF-8 verification, Windows-1252/GB18030 scoring by delimiter survival — and writes the output back in the same family. The DAT format guide covers the delimiter grammar itself, and the DAT viewer guide shows how to inspect a suspect file without importing it anywhere.

FAQ

Why does my DAT import with Ã before every þ?

The file is UTF-8 being read as Windows-1252. Re-encode it, or change the import encoding to UTF-8.

Should I just add a BOM?

If the target is Relativity or another BOM-driven detector, yes — a UTF-8 BOM makes detection deterministic. Legacy Concordance Desktop expects CP1252 and may misread the þ if the file is saved as plain UTF-8, so confirm the target platform's expectation first.

Can a DAT be half CP1252 and half UTF-8?

Yes, and it parses as neither cleanly. It happens when an editor re-saves a few lines. Split at the encoding boundary, convert each part, and rejoin — or re-export the whole file from source.