Spark UDF de-identification

Overview

Spark UDF de-identification is the practice of using user-defined functions (UDFs) in Apache Spark to mask, tokenize, hash or encrypt sensitive fields while data is being processed. Instead of loading raw personal data into a lake and cleaning it later, the pipeline removes or transforms identifying values on the way in. Spark distributes the work across many machines, so the same function runs over billions of rows.

For teams that build models on tables, this is a practical gate. A table that has passed through a de-identification job can be shared with analysts, used for training and kept for longer, with much lower risk. This page explains how the pattern works, shows code, compares options and describes how it supports tabular foundation models. Related pages: Apache Spark, Delta Lake, HashiCorp Vault, Thales and Databricks.

What it is

It is a design pattern rather than a single product. The ingredients are simple.

• A Spark job reads source data from files, a database or a stream.
• One or more UDFs (or built-in column functions) transform the sensitive columns.
• The output is written to a lake or warehouse table that holds only protected values.

What counts as sensitive

Common targets include:

• direct identifiers: names, national ids, email addresses, phone numbers, account numbers
• quasi-identifiers: birth date, postal code, job title, rare category values
• free text that may contain names or addresses
• device and network identifiers such as IP addresses and cookie ids
• sensitive attributes: health codes, income, religion, union membership

The main transformations

Where the UDF fits

Spark has many built-in functions such as sha2, regexp_replace, substring and concat. A UDF is needed when the logic is more specific: calling a tokenization library, applying a format-preserving method, detecting entities in free text, or fetching a key from a service. A UDF is custom code that Spark calls on each row or on batches of rows.

Where it came from

Spark began as a research project for in-memory cluster computing and grew into a general data engine. Early on, custom logic in Spark meant writing Scala or Java functions. Later, PySpark brought Python users in, and with it Python UDFs. Row-at-a-time Python UDFs were slow because every row crossed the boundary between the JVM and Python.

Vectorized UDFs

The arrival of Apache Arrow based data exchange led to pandas UDFs, also called vectorized UDFs. They pass whole batches as pandas Series or DataFrames, which is far faster. Today the recommended pattern for heavy Python work in Spark is a pandas UDF or a built-in function, and a plain Python UDF only for small jobs.

The privacy side

The need grew from privacy rules and security practice.

• Laws about personal data pushed firms to minimize what they store.
• Breach costs showed that raw copies in lakes were a large risk.
• Data science teams wanted broad access to data, which made safe defaults necessary.
• Techniques such as k-anonymity, pseudonymization and differential privacy were studied in research and then adopted in simplified forms in industry.

The combination of a scalable engine and mature privacy ideas gave rise to the "de-identify during ingestion" pattern. It is now a common stage in data platform design, often between a raw landing zone and a curated zone.

How it works

Think of a mailroom in a large office. Letters arrive, and before they go to desks, a clerk covers the names on the envelopes with stickers showing a code. The code book stays locked in a drawer. The desks get useful mail but cannot tell who it came from. Spark UDF de-identification is that clerk, working on millions of letters at once.

Step 1: Read the data in partitions

Spark splits the input into partitions, small pieces that can be processed in parallel. Each executor, which is a worker process, handles some partitions.

Step 2: Define the protection rules

A rule maps each column to a method. For example: email to tokenization, birth_date to year only, postal_code to the first three characters, notes to a text scrubber.

Step 3: Register or use the functions

For built-in functions, you simply call them in a select. For custom logic, you wrap a Python or Scala function as a UDF and apply it to the column.

Step 4: Get keys safely

The UDF needs a secret, such as a key for keyed hashing. The key should be fetched from a secrets service (HashiCorp Vault or a hardware-backed manager from a vendor like Thales) and shipped to executors in a safe way, for example as a broadcast variable or retrieved once per partition. It should not be written in code or logged.

Step 5: Apply the function in batches

With a pandas UDF, Spark hands each executor a batch of values as a pandas Series. The function transforms the whole batch at once. This reduces overhead and allows efficient vector operations.

Step 6: Write only protected columns

The output table contains the new columns. The raw columns are dropped before the write. A common mistake is to keep both and rely on permissions; it is safer to not store the raw values in the curated zone at all.

Step 7: Record lineage and metrics

The job logs counts: rows read, rows written, values redacted per column and failures. It records the version of the rules. Later, an auditor can ask which rule version produced a table.

An everyday analogy for each method

• Redaction is a black marker pen on a document.
• Masking is showing only the last four digits on a receipt.
• Keyed hashing is a stamp that always makes the same pattern for the same name, but only if you own the stamp.
• Tokenization is a coat check ticket.
• Generalization is saying "in their thirties" instead of giving an age.
• Noise is blurring a photo slightly so faces are harder to match.

Deterministic versus random

A deterministic method gives the same output for the same input. That keeps joins and group-bys working. It also means that anyone with a guess list and the method can test guesses, unless the method uses a secret key. A random method gives a different output each time, which is safer but kills joins. Choose per column.

Handling structured streaming

De-identification is not only for batch files. Spark Structured Streaming can read from a message queue and apply the same functions to each micro-batch.

• The functions must be stateless or carry only small broadcast state.
• Pseudonym keys must be identical in batch and streaming, or the same person will look like two people.
• Late or malformed records should go to a quarantine location with restricted access, not to the curated table.

stream = (
    spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "broker.example.internal:9092")
    .option("subscribe", "tickets")
    .load()
)

The code above only shows the source. After parsing the payload into columns, the same withColumn calls from the batch job apply.

Choosing the level of protection by risk

Dealing with rare values

Rare values are dangerous because they stand out. A job title held by one person in a large table identifies that person. Two common fixes are:

• Top-N plus other. Keep the most frequent categories and group the rest as other.
• Minimum count suppression. Replace any category with fewer than a set number of rows by other.

counts = clean.groupBy("job_title").count()
common = counts.filter("count >= 20").select("job_title")
clean = (
    clean.join(common.withColumn("keep", F.lit(1)), "job_title", "left")
         .withColumn("job_title", F.when(F.col("keep") == 1, F.col("job_title")).otherwise("other"))
         .drop("keep")
)

Performance tuning in practice

• Select only needed columns early, so less data moves through the UDF.
• Avoid UDFs on columns used in filters when a built-in function would work, so the optimizer can push filters down.
• Tune the Arrow batch size if memory is tight.
• Cache the output of expensive steps only if reused.
• Watch for skew: a hot key can make one task much slower than the rest.

Governance checklist

• [ ] A written list of sensitive columns, with an owner.
• [ ] Rules stored in version control and reviewed.
• [ ] Unit tests for each protection function.
• [ ] Post-job checks for raw columns and patterns.
• [ ] Access to the raw landing zone limited and logged.
• [ ] Retention period set for raw data.
• [ ] Key rotation plan that keeps pseudonyms consistent.
• [ ] Documented measure of model quality on the protected data.

When not to use this pattern

If a table is tiny and used once, a manual script may be enough. If the data must never leave the source system, apply masking there. If you need formal privacy guarantees against strong attackers, consider differential privacy or synthetic data with proven properties, and involve a privacy specialist.

Key concepts and terms

A worked example

This example uses PySpark. Paths and hosts are fake.

The input

A table of support tickets with columns ticket_id, customer_email, phone, birth_date, postal_code, notes, priority, resolution_hours. We want to build a model of resolution time. The model needs priority, resolution_hours, an age band and a coarse region. It does not need the email or phone.

Code

import hmac
import hashlib
import re
import pandas as pd
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import pandas_udf

spark = SparkSession.builder.appName("deid-tickets").getOrCreate()

# Placeholder: fetch from a secrets service at run time.
# The host below is fake and shown only for illustration.
def get_secret() -> bytes:
    return b"value-from-secrets-service"

SECRET = spark.sparkContext.broadcast(get_secret())

@pandas_udf("string")
def keyed_pseudonym(values: pd.Series) -> pd.Series:
    key = SECRET.value
    def one(v):
        if v is None:
            return None
        mac = hmac.new(key, v.strip().lower().encode("utf-8"), hashlib.sha256)
        return mac.hexdigest()[:24]
    return values.map(one)

EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
PHONE_RE = re.compile(r"\+?\d[\d\s().-]{7,}\d")

@pandas_udf("string")
def scrub_text(texts: pd.Series) -> pd.Series:
    def one(t):
        if t is None:
            return None
        t = EMAIL_RE.sub("[EMAIL]", t)
        t = PHONE_RE.sub("[PHONE]", t)
        return t
    return texts.map(one)

raw = spark.read.parquet("s3://example-bucket/landing/tickets/")

clean = (
    raw
    .withColumn("customer_key", keyed_pseudonym(F.col("customer_email")))
    .withColumn("age_band",
        F.when(F.col("birth_date").isNull(), None)
         .otherwise(F.concat(
             (F.floor((F.year(F.current_date()) - F.year("birth_date")) / 10) * 10).cast("string"),
             F.lit("s"))))
    .withColumn("region", F.substring("postal_code", 1, 3))
    .withColumn("notes_clean", scrub_text(F.col("notes")))
    .drop("customer_email", "phone", "birth_date", "postal_code", "notes")
)

clean.write.mode("overwrite").parquet("s3://example-bucket/curated/tickets/")

What each line does

• The secret is fetched once on the driver and broadcast, so executors do not call the secrets service per row.
• keyed_pseudonym normalizes the email (trim, lower case) before hashing, so the same person gets the same key even if the case differs.
• scrub_text removes emails and phone numbers from free text with regular expressions. This is a simple first pass, not a complete solution for names.
• Age becomes a decade band. Postal code becomes a three-character region.
• Raw columns are dropped before the write.

A numeric check of generalization

Suppose 1,000,000 tickets have an exact birth date. Many combinations of birth date and full postal code are unique. After converting to a decade band and a three-character region, suppose each combination now has at least 50 rows. That means the table meets a k-anonymity level of at least 50 for those two fields, which is a strong improvement. You should compute this, not assume it:

combo = clean.groupBy("age_band", "region").count()
combo.agg(F.min("count").alias("k_min")).show()

If k_min is below your threshold, widen the bands or suppress rare groups.

A SQL variant for simple cases

SELECT
  sha2(concat(lower(trim(customer_email)), 'example-salt'), 256) AS customer_key,
  priority,
  resolution_hours,
  substring(postal_code, 1, 3) AS region
FROM landing.tickets;

This uses only built-in functions. It is fast and needs no UDF, but a fixed salt in SQL text is weaker than a secret key held in a service. Use it only for low-risk fields.

Testing the job

def test_pseudonym_is_stable():
    a = one("User@Example.com ")
    b = one("user@example.com")
    assert a == b

def test_no_raw_columns_left(df):
    forbidden = {"customer_email", "phone", "birth_date", "postal_code", "notes"}
    assert forbidden.isdisjoint(set(df.columns))

Tests that assert the absence of raw columns catch the most common regression: someone adds a column to the select and forgets to protect it.

Strengths and limitations

Strengths

• Scale. The same logic runs over gigabytes or petabytes by adding machines.
• Early protection. Raw values never reach the curated zone.
• Flexibility. You can write any logic in Python or Scala.
• Fits existing pipelines. It is one more stage in a Spark job, orchestrated like others.
• Testable. Functions can be unit tested on small frames.
• Reusable. One library of protection functions can serve many pipelines.

Limitations

• Performance traps. Row-at-a-time Python UDFs are slow. They also block some query optimizations because Spark treats a UDF as a black box.
• Key handling is on you. Shipping secrets to executors safely needs care.
• Not a full privacy guarantee. Pseudonymization alone does not prevent re-identification through quasi-identifiers.
• Free text is hard. Regular expressions miss names, addresses and unusual formats. Entity detection models help but are not perfect.
• Utility loss. Strong protection reduces signal for modelling.
• Drift. Source schemas change. A new column may carry personal data and slip through if rules are written as a deny list.
• Operational cost. More stages mean more to monitor.

How it compares with alternatives

A mature platform usually combines several: a governance layer for access, Spark for bulk de-identification at ingestion and a key service behind it.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed numeric and categorical types, missing values and often no helpful column names. They are pre-trained on synthetic or real tables and use in-context learning, where labelled example rows are given at inference and the model predicts without gradient updates. See /model/architecture and /model/research for details. Spark de-identification connects to this world at several points.

Preparing data for pre-training and fine-tuning

If real tables are used for pre-training, continual learning or fine-tuning, they must first be made safe. A Spark job at ingestion can strip identifiers, bin sensitive numbers and scrub text. The output is a set of tables that a training system can read widely.

Why these models tolerate de-identified tables

Because the models read structure and values rather than depending on column names, a table with neutral or renamed columns still works. A pseudonymized id column is just another categorical column. Generalized numbers become ordinal categories or coarse numerics. Missing values are part of the learning signal, so redacted fields that become null are handled as missing, though you should consider whether the pattern of nulls leaks information.

Imputation after masking

Masked or suppressed cells create gaps. Tabular models can impute these gaps, which is useful for completing partially protected data. Treat the imputed values with care. If a sensitive attribute is predicted accurately from other columns, you may have recreated the information you removed.

Anomaly detection on de-identified data

Anomaly detection works well on protected tables when the signal is in numeric patterns and category frequencies. Keyed pseudonyms keep entity-level patterns, such as one account with unusual activity, without revealing who the account belongs to. This is a good fit for fraud and operations monitoring.

Sector and semantic classification

Classification of columns or entities by sector or meaning depends on content. If text is scrubbed too aggressively, semantic signal is lost. A middle path is to keep category-like text and remove only entity-level details such as personal names and contact data. For columns that are fully pseudonymized, only structure and co-occurrence can be learned.

Continual learning

When a model is updated with new batches, the de-identification rules must stay consistent. If the secret key changes, the same entity gets a new pseudonym, and the model sees old and new rows as different entities. Keep keys stable, or version the scheme and plan a mapping.

Leakage and memorization

Even with in-context learning, a model sees the rows you put in the prompt. If those rows contain direct identifiers, they pass into the model's working memory. De-identification before inference is as important as before training. For fine-tuned models, memorization of rare rows is a risk, so remove unique identifiers and consider suppressing rare combinations.

A typical flow

1. Raw files arrive in Amazon S3 or similar storage.
2. A Spark job runs the de-identification UDFs.
3. The output is written as Delta Lake tables with schema checks.
4. Training and evaluation jobs read the curated tables.
5. Experiments are tracked in a system such as MLflow.
6. Benchmarks on protected tables are reported against reference datasets, as discussed in /model/benchmarks.

This is a general pattern. It does not assume any specific built-in integration between a product and a model.

Practical guidance and best practices

Prefer built-in functions first

Spark's own column functions are optimized and can be pushed down by the query planner. Use them when they cover the need. Reach for a UDF only when you must.

If you use a Python UDF, use a pandas UDF

Vectorized UDFs are much faster than row-at-a-time functions. Keep the function pure: no external calls per row and no global mutable state.

Fetch secrets once

Retrieve keys on the driver and broadcast them, or retrieve once per partition. Never call a secrets service for each row. Never print the key in logs.

Use allow lists, not deny lists

Define which columns may pass through unchanged, and treat everything else as sensitive by default. Then a new column in the source does not leak by accident.

Normalize before you hash

Trim spaces, lower-case emails and standardize phone formats before computing a pseudonym. Otherwise the same person gets several keys.

Use keyed methods

A plain hash of an email can be reversed by guessing from a list. A keyed hash cannot, unless the key leaks.

Protect quasi-identifiers

Hiding names is not enough. Generalize dates and locations, suppress rare categories and measure k-anonymity on the output.

Handle nulls deliberately

Decide what null input should produce. Usually it should stay null. Do not hash the string "None".

Make the job idempotent

Running the job twice should give the same result. Deterministic methods and fixed rule versions make that possible.

Store rule versions

Write the rule version into the table properties or a lineage record. When a rule changes, you can find tables that need reprocessing.

Add data quality checks

After the job, assert that no raw column remains, that formats are valid and that row counts match expectations.

Control the landing zone

The raw landing zone is the riskiest place. Limit access, set a short retention period and encrypt it. Delete raw data once curated data is verified.

Test utility

Train a baseline model on the protected table and on a permitted reference, and compare. If there is a large gap, find which transformation removed the signal.

Common mistakes

• Leaving raw columns in the output. Always drop them in code, not just by permission.
• Using plain unsalted hashes. These can be reversed through guessing.
• Hard-coding a salt or key. It ends up in repositories and logs.
• Calling a remote service per row. This causes slow jobs and rate-limit failures.
• Row-at-a-time Python UDFs on large tables. Use pandas UDFs or built-ins.
• Ignoring quasi-identifiers. Birth date, postal code and gender together can identify many people.
• Trusting regular expressions for free text. They miss names and unusual formats.
• Changing keys without a plan. Pseudonyms change and joins break.
• Different pseudonyms in different tables. Use the same function and key so joins work.
• Skipping tests. Silent failures leave raw data in place.
• Logging samples. Debug output with real rows defeats the purpose.
• Forgetting null handling. Hashing nulls produces a common value that looks like a real entity.
• Assuming de-identified means anonymous. Legal and technical standards differ.
• Not measuring model impact. Over-protected data can ruin modelling value.

Frequently asked questions

What is a Spark UDF?

A Spark UDF is a user-defined function that you write in Python, Scala or Java and apply to columns in a Spark DataFrame. It lets you run custom logic that built-in functions do not provide. Spark treats it as a black box, so it can be slower than built-in functions, and vectorized pandas UDFs are the preferred form for Python.

Why use a UDF for de-identification?

Many protection steps need custom logic, such as calling a tokenization library, applying format-preserving encryption, scrubbing entities from text or using a secret key. A UDF packages that logic so it can run in parallel across a cluster as part of the ingestion job.

Is hashing an email enough to de-identify it?

Usually not. A plain hash of an email can be reversed by hashing a list of likely emails and comparing. A keyed hash with a secret held in a separate service is much stronger. Even then, the result is pseudonymized data, not anonymous data, because the key holder can recompute it.

Should I use a pandas UDF or a plain Python UDF?

Use a pandas UDF for anything beyond tiny datasets. It processes batches using Arrow transfer and vectorized operations, which avoids the heavy cost of moving each row between the JVM and Python. A plain Python UDF is fine for small test jobs.

How do I give a UDF a secret key safely?

Fetch the key from a secrets service on the driver, then broadcast it to executors, or fetch it once per partition. Do not embed it in code, do not log it and do not call the secrets service for every row. Short-lived credentials and least-privilege identities reduce the risk further.

Does de-identification reduce model accuracy?

It can, depending on what is removed. Dropping identifiers the model never needed costs nothing. Generalizing a numeric field can lose signal. The way to know is to train on the protected table and compare with a reference run inside a secure environment.

How do I handle free text columns?

Start with regular expressions for emails, phone numbers and ids, then add entity recognition for names and addresses if the risk justifies it. No method is perfect, so consider dropping free text if the model does not need it, or restricting access to the scrubbed version.

Can tabular foundation models work on pseudonymized data?

Yes. These models learn from the structure and values of tables, not from the meaning of identifier strings. A pseudonymized id acts as a category, and generalized numbers still carry order. Column names are not required, so renaming columns to neutral labels is often harmless.

How do I check that the output is safe?

Check that no raw columns remain, scan for patterns such as emails in text columns, compute k-anonymity over quasi-identifiers and test for rare unique rows. Add these checks as automated tests in the pipeline so they run on every job.

Where should this stage sit in the pipeline?

Place it between the raw landing zone and the curated zone, as early as possible. That way only protected data spreads to analysts, notebooks and training systems.

Glossary

• Arrow: A columnar in-memory format used to move data efficiently between systems.
• Broadcast: Sending a read-only value to all workers once.
• Curated zone: Storage holding cleaned, protected data.
• DataFrame: A table-like structure in Spark.
• De-identification: Removing or altering identifying details.
• Driver: The Spark process that coordinates a job.
• Executor: A worker process that runs tasks.
• Generalization: Replacing precise values with broader ones.
• HMAC: A hash-based message authentication code, a keyed hash.
• Idempotent: Giving the same result when run more than once.
• k-anonymity: A property where each quasi-identifier combination appears at least k times.
• Landing zone: Where raw data first arrives.
• Lineage: Record of how data was produced.
• Partition: A slice of data processed as a unit.
• Pseudonymization: Replacing identifiers with consistent surrogates.
• Quasi-identifier: A field that can identify when combined with others.
• Redaction: Removing a value.
• Salt: Extra data added to a hash input.
• Tokenization: Replacing a value with a surrogate token.
• UDF: User-defined function.

Summary

Spark UDF de-identification applies custom or built-in functions inside Apache Spark jobs to mask, tokenize, hash, generalize or encrypt sensitive fields as data enters a lake. It scales to very large tables, keeps raw values out of curated storage and can be tested like any code. Success depends on good key handling, normalization before hashing, treatment of quasi-identifiers, measured privacy checks and attention to speed through pandas UDFs and built-in functions. For tabular foundation models, the pattern prepares real tables for training, fine-tuning and inference while keeping the structure that these models learn from: rows, columns, mixed types and missing values. Pair it with a key service, a governed lakehouse and clear measurement of both privacy and model quality. For the model side, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Data protection in my stack, under Data engineering and lakehouse. Others in the same group: