Format-preserving encryption

Overview

Format-preserving encryption, usually shortened to FPE, is a way of encrypting data so that the ciphertext has the same format as the plaintext. A 16-digit number encrypts to another 16-digit number. A nine-character code made of uppercase letters encrypts to another nine-character code of uppercase letters. The output looks like valid data to databases, file formats and applications that expect a fixed shape, yet it is protected by a cryptographic key.

This property solves a practical problem. Many older systems have rigid column definitions, validation rules and report layouts. They cannot accept a long random binary string or a base64 blob in a field that was designed for digits. FPE lets teams protect such fields without redesigning the systems around them. It is closely related to data tokenization, and some tokenization products use FPE as their engine. For tabular data and models that learn from tables, FPE keeps columns typed, keeps lengths stable and, when used deterministically, keeps values consistent, so joins and group counts still work. This article explains how FPE works, where it fits and where it does not.

What it is

FPE is a family of block-cipher-based modes that encrypt a string over a chosen alphabet and return a string of the same length over the same alphabet. The alphabet is described by its radix. For decimal digits the radix is 10. For hexadecimal it is 16. For lowercase letters it is 26. For letters and digits together it is 62.

A quick analogy

Imagine a combination lock with four wheels, each showing digits from 0 to 9. A normal cipher would replace your four-digit code with a long string of unrelated symbols that no lock could display. FPE is like a secret shuffling procedure that always turns one valid four-digit setting into another valid four-digit setting, and that can be undone only if you know the secret. Every valid setting still fits the lock.

Key characteristics

What a tweak is

A tweak is like a salt that is public but affects the result. If you use the account type as the tweak, the same card number encrypts differently in different account types. This limits the damage of pattern analysis, because identical values in different contexts no longer give identical ciphertexts. The tweak does not need to be secret, but it must be available at decryption time.

Where it is typically applied

• Payment card numbers in legacy applications.
• National and tax identification numbers.
• Phone numbers, account numbers and policy numbers.
• Short codes such as postal codes or employee numbers.
• Test data creation, where realistic-looking but protected values are needed.

Where it came from

The idea of encrypting small domains without expanding them goes back to cryptographic research in the 1990s on ciphers over arbitrary sets. Papers in the early 2000s analyzed how to build ciphers for domains such as the set of numbers up to a given limit. A 2002 paper on ciphers with arbitrary finite domains is often cited, and later academic work in the 2000s formalized security notions and constructions based on Feistel networks and on card-shuffling methods. The name format-preserving encryption was popularized in that period, notably in work on how to encrypt credit card numbers inside existing database systems.

Standardization followed. A United States national standards publication, NIST Special Publication 800-38G, specifies two modes named FF1 and FF3. Later analysis found weaknesses in the original FF3 for some parameters, and the standards body revised the recommendation, leading to the mode known as FF3-1. Readers should consult the current version of the standard and any later guidance before choosing a mode, because the status of modes can change as cryptanalysis advances.

Brief timeline

How it works

Step 1: define the alphabet and length

You decide the radix and the length of the string. A nine-digit identifier has radix 10 and length 9. The standard modes need a minimum domain size so the cipher is not trivially small. A common rule requires that the number of possible values, radix to the power of length, be at least one million. Very short strings, such as a two-digit code, are therefore not good candidates.

Step 2: convert the string to numbers

Each symbol is mapped to a number between 0 and radix minus 1. For digits this is the digit itself. The string becomes a sequence of small integers.

Step 3: split into two halves

The sequence is divided into a left half and a right half. If the length is odd, the halves differ by one symbol. Think of tearing a ticket into two pieces.

Step 4: run a Feistel network

The core of FF1 and FF3-1 is a Feistel network. In each round:

1. Take the right half and the key, tweak and round number.
2. Feed them to a round function built from a standard block cipher, usually AES.
3. Convert the round function output into a number modulo a power of the radix.
4. Add that number to the left half, symbol by symbol, modulo the radix.
5. Swap the halves.

After several rounds the halves are recombined. Because every step works modulo the radix, the output stays in the same alphabet and length. Because each round can be undone by subtracting instead of adding, the whole process is reversible given the key and tweak.

An everyday analogy for the Feistel idea

Picture two people at a table with a deck of numbered cards, each holding half a secret. In each round, one person uses a private scrambling recipe on their own cards and adds the result to the other person's cards. Then they swap seats. After enough rounds, no single half reveals the original. To undo it, run the rounds backward with the same recipe.

Step 5: convert back to a string

The final numbers are mapped back to symbols. The result is a string of the same length and alphabet as the input.

FF1 and FF3-1 in brief

The details of the specifications are precise and should be implemented by well-reviewed libraries, not hand-written code.

Handling formats that are not pure alphabets

Real fields often contain separators. A card number may have spaces. A phone number may have a plus sign and dashes. Systems handle this by:

1. Extracting only the characters to encrypt.
2. Encrypting those characters with FPE.
3. Reinserting the separators at their original positions.

Some policies also leave a prefix or suffix in clear, for example the first six and last four digits of a card number, and encrypt only the middle.

Handling checksum digits

Many identifiers end in a check digit computed from the other digits. After encrypting, the check digit may no longer match. Options include:

• Encrypt all digits and accept an invalid checksum.
• Encrypt all but the last digit and recompute the checksum.
• Encrypt and then adjust the checksum so validation passes.

The right choice depends on whether downstream systems validate the checksum and whether you want encrypted values to be distinguishable from real ones.

Key concepts and terms

A worked example

A numeric walk-through of the idea

Consider a four-digit code, radix 10, length 4. Split it into two halves, left and right, of two digits each. Take the code 1234, so left is 12 and right is 34.

Suppose a toy round function turns the right half and a key into a two-digit number. Let the result for round 1 be 57. Add it to the left half modulo 100: 12 plus 57 is 69. Swap the halves, so left is 34 and right is 69.

In round 2, suppose the round function of 69 gives 21. Add to the left half modulo 100: 34 plus 21 is 55. Swap: left is 69, right is 55. After the final round, join them to get a four-digit result such as 6955.

To decrypt, run the rounds in reverse order and subtract instead of add. The same round function outputs are recomputed from the same key, so the original 1234 is recovered. Real FPE uses many more rounds and a strong cipher for the round function, and it also refuses domains as small as this toy example.

Python with a vetted library

The sketch below shows the usage pattern with a placeholder library call. Always check the actual API of the library you choose, and fetch the key from a secrets manager.

from fpe_library import FF1  # placeholder name for a reviewed implementation

def get_key() -> bytes:
    # In real use, retrieve from a managed key service, not from source code
    return bytes.fromhex("00112233445566778899aabbccddeeff")

cipher = FF1(key=get_key(), radix=10)

def protect_card(card: str, tweak: bytes) -> str:
    digits = [int(c) for c in card if c.isdigit()]
    head, middle, tail = digits[:6], digits[6:-4], digits[-4:]
    enc = cipher.encrypt(middle, tweak=tweak)
    return "".join(map(str, head + enc + tail))

def reveal_card(card: str, tweak: bytes) -> str:
    digits = [int(c) for c in card if c.isdigit()]
    head, middle, tail = digits[:6], digits[6:-4], digits[-4:]
    dec = cipher.decrypt(middle, tweak=tweak)
    return "".join(map(str, head + dec + tail))

protected = protect_card("4111111111111111", tweak=b"retail")
print(protected)
print(reveal_card(protected, tweak=b"retail"))

The example keeps the first six and last four digits in clear and encrypts the six middle digits. That is a domain of one million values, the minimum allowed, which is a good reminder that short middles are weak. A larger encrypted portion is safer.

Applying FPE to a table column

import pandas as pd

df = pd.DataFrame({
    "order_id": [1001, 1002, 1003],
    "account": ["123456789", "123456789", "987654321"],
    "amount": [25.0, 40.0, 18.5],
})

df["account_fpe"] = df["account"].map(lambda a: protect_digits(a, tweak=b"accounts"))
safe = df.drop(columns=["account"])
print(safe)

Because encryption is deterministic for a given key and tweak, the first two rows get the same protected account and remain linkable. The protected value is still nine digits, so a database column defined as nine digits accepts it.

Choosing parameters

Domain size alone is not security. An attacker who knows that only a small subset of values is in use, such as a list of valid postal codes, can exploit that structure.

Strengths and limitations

Strengths

• No change to schemas, column types or lengths.
• Works with legacy systems and fixed-width files.
• Deterministic mode keeps joins, grouping and deduplication possible.
• No vault or mapping table is needed. Storage cost is nil beyond the data itself.
• Easy to scale, since encryption is a local computation.
• Reversible by authorized systems that hold the key and tweak.
• Based on a standard block cipher and standardized modes.
• Fast enough for high-volume batch and real-time use.

Limitations

• Deterministic output leaks equality. Equal inputs give equal outputs under the same key and tweak, which allows frequency analysis.
• Small domains are weak. Few possible values means few possible ciphertexts, so guessing is easy.
• No authentication or integrity. FPE does not detect tampering. A modified ciphertext still decrypts to some valid-looking value.
• Strict dependence on correct tweak management. Losing the tweak means losing the data.
• Key rotation is harder than with tokenization because every stored ciphertext must be re-encrypted.
• Format preservation means ciphertext can be mistaken for real data. Mark protected fields clearly in metadata.
• Cryptanalysis has found issues with earlier modes, so implementations must follow current guidance.
• Not a good fit for long free text, large documents or unstructured fields. Use standard authenticated encryption for those.

Security notes in plain terms

FPE trades some security margin for compatibility. Standard modern encryption uses random initialization values so the same message never looks the same twice and adds authentication. FPE cannot add extra length, so it cannot carry a random value or an authentication tag. That is the cost of keeping the format. Choose it when the format constraint is real, not out of convenience.

How it compares with alternatives

FPE versus tokenization

Both keep data in shape. The main difference is where trust lives. Vaulted tokenization trusts a database of mappings and its access controls. FPE trusts a key and a cipher. FPE scales without lookups, but key rotation and data recovery require care. Tokens from a vault can be random and unrelated to the data, which removes any mathematical link. Enterprise platforms such as Protegrity offer both approaches, and a key service like HashiCorp Vault can supply the keys. Distributed jobs that apply either approach are described in Spark UDF de-identification.

When to prefer which

• Choose FPE when you cannot change the schema, want no extra infrastructure and can protect keys well.
• Choose vaulted tokenization when you need random tokens, easy revocation and central audit of every reveal.
• Choose standard authenticated encryption when format does not matter.
• Choose masking or synthetic data when the real value is never needed again.

Role in tabular foundation models

FPE is not part of a model. It is a way of protecting columns before they reach systems where models learn from or predict on raw tables. Its behavior has direct effects on how such models see the data.

How table models read columns

A tabular foundation model treats a table as rows and columns with mixed types: numeric, categorical, text and date. It copes with missing values, may not rely on column names and is pre-trained on many tables, often synthetic ones, so it can predict on a new table by in-context learning from example rows without weight updates. See the model architecture and research. Its input pipeline usually infers a type for each column from the values.

Effects of FPE on model inputs

• Type inference. An FPE-protected digit string still looks like a number. A loader may infer a numeric column and the model may treat it as a quantity, reading order and magnitude into what is really an identifier. Declare identifier columns as categorical or as raw identifiers so they are not treated as numbers.
• Consistency as a feature. With deterministic FPE and a fixed tweak, repeated identifiers stay repeated. A model can still learn that the same account appears many times, which supports behavior patterns, repeat detection and anomaly detection.
• Meaningless magnitude and order. Encrypted digits have no numeric relation to the originals. Do not encrypt fields where magnitude matters, such as amounts, ages or counts, unless the model does not need to use them.
• Embedded structure destroyed. If an identifier carries structure, such as a region prefix, encrypting it removes that signal. If the model needs the structure, keep the structured part in clear, derive a separate safe feature before encryption or choose a different method.
• Missing values. Do not encrypt null or empty values. Keep them as missing so that imputation and missingness patterns stay visible. See imputation networks and masked cell modeling.
• Tweaks and joins. Different tweaks give different ciphertexts for the same input. Use the same tweak across tables that must join and a different tweak for domains that must not link.
• Column semantics. Renaming columns for privacy can reduce hints that help semantic or sector classification of columns. Evaluate accuracy on the protected schema.

Concrete task effects

Privacy and risk when serving models

Putting sensitive identifiers into a context window or a log is a risk. Encrypting them first with FPE reduces exposure, and decrypting after the prediction returns restores usability. Remember that deterministic encryption lets an observer learn which rows share the same identifier, so combine it with access control and minimal retention. For measured performance on tabular tasks, see benchmarks.

Practical pattern

1. Identify identifier columns and mark them categorical in the model's input schema.
2. Encrypt them with FPE at the edge of the trusted zone, with a stable tweak per domain.
3. Send protected rows to the model service.
4. Return predictions keyed by a row identifier.
5. Decrypt or join back inside the trusted zone when a real value is needed.

Practical guidance and best practices

Choose modes and libraries carefully

• Use a well-reviewed library that implements the current standard.
• Prefer FF1 unless you have a specific reason and current guidance for FF3-1.
• Never write your own FPE.
• Confirm that the library enforces minimum domain sizes.
• Test against known-answer vectors published with the standard.

Manage keys properly

• Store keys in a managed key service or hardware module. See HashiCorp Vault.
• Use separate keys for separate data domains.
• Version keys and store the key version with the data or in metadata.
• Plan for rotation: decrypt with the old key and encrypt with the new key in controlled batches.
• Restrict who can call decryption, and log every call.

Use tweaks wisely

• Use tweaks to separate contexts, such as table, field or tenant.
• Keep tweak values documented and stable.
• Do not rely on tweaks for secrecy.

Protect against small-domain weaknesses

• Encrypt as many characters as practical, not just a few.
• Avoid fields with a tiny set of valid values.
• Be aware of known value lists, such as valid postal codes, that shrink the effective domain.
• Combine with access controls and monitoring because guessing attacks work best when many ciphertexts are available.

Integrate with data platforms

• Apply FPE at ingestion or in a gateway, not in every application.
• In distributed engines, use vectorized or batch calls. See Apache Spark.
• Keep data classification in a data catalog and record flows in data lineage.
• Test every downstream validator, report and index with protected values.

Align with privacy rules

Regulations such as GDPR treat encrypted personal data as still being personal data when the key exists. FPE is a safeguard that lowers risk, not a way to make data anonymous. Get legal and security review for specific obligations.

Test checklist

Migration plan for an existing column

Moving a live column to FPE needs a plan, because the data and every reader change at once.

1. Inventory every system that reads or writes the column, including reports, exports and backups.
2. Decide which characters are encrypted, which stay in clear and which tweak applies.
3. Add a new column for the protected value, or a shadow table, instead of overwriting at first.
4. Backfill in batches, with a round-trip check on a sample of each batch.
5. Switch readers one by one to the protected column.
6. Remove the original column only after a monitored period with no reads.
7. Delete or lock down old backups that still contain originals.

Thinking about attack models

The lesson is that FPE protects well against casual exposure and poorly against a determined attacker who can query the system or who knows that few values are possible. Layer it with access control and monitoring.

Performance expectations

FPE does a number of AES operations per value, so it is slower than a simple hash but far faster than a network lookup in most designs. In batch pipelines, throughput is usually limited by input and output rather than by the cipher. In real-time paths, the cost is small compared with a database call. Measure on your own hardware, use vectorized or batched calls where the library offers them, and avoid creating a new cipher object for every value.

Documentation to keep

• A list of protected fields with radix, length, tweak and key identifier.
• The rule for separators and clear-text prefixes or suffixes.
• The checksum policy.
• The key rotation schedule and the procedure.
• Who may call decryption and for what purposes.

Examples of field policies

When not to use FPE

• The field is long free text or a document.
• The field has only a handful of valid values.
• Analysts need ranges, ordering or arithmetic on the value.
• You need protection against an attacker who can submit values and read outputs without limits.
• You can change the schema at will, so standard authenticated encryption is simpler and stronger.

Reading the standard as a practitioner

You do not need to implement the standard to use it well. It helps to know four things: the allowed radix range, the minimum domain size, the tweak rules of the chosen mode and the requirement that the underlying block cipher be a approved one such as AES. A good library hides the rest. When reviewing a vendor or an open library, ask whether it passes the official test vectors, how it reports invalid input, and how it documents the status of each mode.

Common mistakes

1. Encrypting tiny domains. A two-digit or four-digit field offers almost no protection.
2. Treating deterministic output as random. Equal inputs still give equal outputs.
3. Using the same tweak everywhere. This links values across contexts that should stay separate.
4. Losing the tweak. If the tweak is not stored or derivable, decryption fails.
5. Hard-coding keys. Keys in source code or notebooks leak easily.
6. Skipping authentication. FPE does not detect tampering. Add integrity checks elsewhere if needed.
7. Encrypting numeric measures. The model or report loses magnitude.
8. Letting loaders treat protected digits as numbers. Declare identifier columns as categories.
9. Encrypting nulls. This hides the missing-value pattern and breaks imputation.
10. Rolling your own implementation. Subtle errors break security. Use a vetted library.
11. Ignoring checksum rules. Downstream validators may reject or wrongly accept protected values.
12. Ignoring mode status. Standards change after cryptanalysis. Review current guidance regularly.
13. Assuming compliance. A cipher is one control among many. Governance, access and monitoring still matter.

Frequently asked questions

What is format-preserving encryption?

Format-preserving encryption is a method that encrypts data so that the result has the same length and character set as the input. A number stays a number of the same length, and a code of letters stays a code of letters. This lets systems with fixed column definitions or validation rules store and process encrypted values without being redesigned.

How does FPE differ from regular encryption?

Regular encryption such as AES in an authenticated mode outputs binary data that is longer than the input and looks random. FPE is built so the output fits the input's format. To do that it gives up random per-message values and authentication tags, which means it is deterministic and offers less protection against some attacks than standard modes.

What are FF1 and FF3-1?

They are standardized FPE modes defined in a national standards publication and built on a Feistel network that uses AES as its round function. FF1 is flexible about lengths and radixes. FF3-1 is a revised version of FF3 that addresses weaknesses found in the original. Current guidance should be checked before choosing a mode.

Is FPE the same as tokenization?

No, though they overlap. Both can produce values that look like the original. FPE computes the protected value from the original with a key, so there is nothing to look up. Vaulted tokenization stores a mapping in a protected database. Some vaultless tokenization products use FPE internally. The choice affects key management, scaling and audit.

Is format-preserving encryption secure?

It can be secure when used with a well-reviewed implementation, a strong key, a large enough domain and sensible tweaks. Its limits are real: it is deterministic, it does not authenticate data and it is weak on tiny domains. It is a good fit when format constraints rule out standard encryption and a poor fit when they do not.

What is a tweak in FPE?

A tweak is a public input that changes the mapping without changing the key. It acts like a context label. Using different tweaks for different fields or tenants stops equal values in different places from producing equal ciphertexts. The tweak must be available when decrypting.

Can I run analytics on FPE-protected data?

Yes for many tasks. Deterministic FPE keeps equal values equal, so you can count distinct items, group rows and join tables that share a key and tweak. You cannot do arithmetic, ordering or range queries on encrypted identifiers, because the encryption destroys numeric relationships.

How does FPE affect machine learning on tables?

Protected identifiers behave like categories, which is usually acceptable, but a loader that sees digits may treat them as numbers. Mark them categorical. Do not encrypt measures that a model needs. Keep nulls as nulls, and keep keys, tweaks and versions stable over time so training and prediction see consistent values.

What are the minimum requirements for the data domain?

The standard modes require the number of possible values to be at least a set minimum, one million in the standard's rules, and many practitioners advise much more. Small domains are easy to attack by guessing or by building a table of all possible values, so encrypt as many characters as practical.

How do I rotate FPE keys?

Rotation means decrypting stored values with the old key and re-encrypting them with the new key, usually in controlled batches, while tracking the key version for each record. Because there is no mapping table to update, every protected value must be processed. Plan for the time, storage and testing this requires.

Glossary

• AES: A widely used block cipher.
• Check digit: A digit that helps catch errors in an identifier.
• Ciphertext: Encrypted data.
• Deterministic: Producing the same output for the same input and settings.
• Domain: The set of all valid inputs for a field.
• Feistel network: A structure that splits data into two halves and mixes them over rounds.
• FF1, FF3-1: Standardized FPE modes.
• Key rotation: Replacing a key and re-protecting data under the new key.
• Plaintext: The original, unencrypted data.
• Radix: The number of symbols in the alphabet used.
• Round function: The keyed function applied in each Feistel round.
• Tweak: A public value that changes the mapping under one key.
• Vaultless: Protection that does not rely on a stored mapping table.

Summary

Format-preserving encryption protects a value while keeping its length and character set, so digits stay digits and codes stay codes. Standardized modes such as FF1 and the revised FF3-1 use a Feistel network with AES, a secret key and an optional public tweak. FPE is useful when schemas, validators and legacy systems cannot accept ordinary ciphertext, and it needs no vault. Its trade-offs are determinism, weakness on small domains and lack of built-in integrity checks. For tabular foundation models, it keeps identifier columns consistent for joins, imputation, anomaly detection and in-context learning, as long as those columns are declared categorical, nulls stay null, numeric measures are not encrypted and keys and tweaks stay stable. Compare it with data tokenization and Protegrity, and manage keys with care.

Related

It belongs to Data protection in my stack, under Data engineering and lakehouse. Others in the same group: