Every researcher training generative models on faces knows the drill: download FFHQ (Karras et al., 2019), write a preprocessing pipeline, run inference with a segmentation model, compute depth maps, extract DINO embeddings, encode captions through T5, serialize everything, and pray your disk doesn’t fill up before the actual training begins. By the time your DataLoader is ready, you’ve burned a week on infrastructure that has nothing to do with your research question.

Stratum-FFHQ eliminates that week. It is a complete, pre-computed multi-modal enrichment of the entire FFHQ dataset — 70,000 images, each paired with dense captions, DINOv3 semantic embeddings, T5 text encodings, DWPose keypoints, depth maps, surface normals, and body-part segmentation masks. The dataset is freely available on Hugging Face at timlawrenz/stratum-ffhq.

Why Pre-Computed Enrichment Matters

The prx-tg portrait generation project — covered extensively in our ablation study, Asymmetric Flow Matching experiments, and FP8 optimization work — is a multi-conditioned Diffusion Transformer that ingests identity embeddings (DINOv3), spatial maps (depth, normals, segmentation), pose keypoints, and text captions simultaneously. Every training step requires all four modalities to be aligned, decoded, and normalized before a single gradient is computed.

The preprocessing pipeline behind prx-tg — the Stratum-HQ enrichment system — became the bottleneck. Running inference across five separate models (DINOv3, T5-Large, DWPose, Sapiens-1B for depth, Sapiens-1B for segmentation) on 70,000 images consumed days of GPU time. Every ablation, every hyperparameter sweep, every architecture tweak was gated on this preprocessing step.

Stratum-FFHQ is the solution: compute once, train infinitely.

What’s Inside the Dataset

Every image in the original FFHQ dataset maps to a rich set of aligned artifacts:

Artifact Modality Format Dimensions Description
Captions Parquet — Dense, objective descriptions of image content
T5 Hidden States Tarred .npy (512, 1024) T5-Large encoder output per caption token
DINOv3 CLS Token Tarred .npy (1024,) Global style and composition embedding
DINOv3 Patch Tokens Tarred .npy (257, 1024) Fine-grained semantic features at patch level
DWPose Keypoints Tarred .npy (133, 3) Facial landmarks and upper body posture [x, y, confidence]
Segmentation Tarred .npy (H, W) 28-class body-part and facial labels via Sapiens-1B
Depth Maps Tarred .npy (H, W) Sapiens relative depth, foreground-masked, normalized to [0, 1]
Surface Normals Tarred .npy (H, W, 3) Sapiens per-pixel normals (XYZ), L2-normalized

Model Provenance

Each modality is produced by a specific, well-documented model:

  • DINOv3-ViT-L/16 — Self-supervised vision transformer providing semantic patch and CLS embeddings used for identity conditioning in prx-tg.
  • T5-Large — Text encoder producing dense hidden states from generated captions, enabling text-conditioned generation and manipulation.
  • DWPose — Whole-body pose estimation yielding 133 keypoints for spatial conditioning on posture and facial landmark placement.
  • Sapiens-1B — A family of vision foundation models fine-tuned for human-centric tasks, providing depth estimation, surface normal prediction, and 28-class body-part segmentation.

Architecture for High-Throughput Streaming

The dataset is designed for training loops that saturate GPU compute, not for one-off downloads. Every modality is packaged into WebDataset shards — synchronized tarballs of 1,000 items each — enabling lazy streaming over HTTP with zero local storage requirements.

The key design decisions:

  1. Modality-isolated shards. Depth maps, pose keypoints, and DINOv3 embeddings live in separate tarball directories. Your DataLoader pulls only the modalities your model actually needs.
  2. Perfectly aligned keys. Shard 00000-00999.tar in the depth directory contains exactly the same image indices as shard 00000-00999.tar in the pose directory. This means you can zip() streams across modalities without key-matching logic.
  3. Lightweight metadata. Captions live in queryable Parquet files rather than inside the heavy binary shards, keeping metadata access fast and independent of training I/O.

Streaming depth and pose together requires only a few lines:

import webdataset as wds

# Generate aligned URLs for the modalities you need
urls_depth = [f"https://huggingface.co/datasets/timlawrenz/stratum-ffhq/resolve/main/depth/{i:02d}000-{i:02d}999.tar" for i in range(70)]
urls_pose = [f"https://huggingface.co/datasets/timlawrenz/stratum-ffhq/resolve/main/pose/{i:02d}000-{i:02d}999.tar" for i in range(70)]

# Initialize streaming pipelines with auto-decoding
ds_depth = wds.WebDataset(urls_depth, shardshuffle=False).decode().to_tuple("__key__", "npy")
ds_pose = wds.WebDataset(urls_pose, shardshuffle=False).decode().to_tuple("__key__", "npy")

# Zip the perfectly aligned streams — no key matching required
for (key_d, depth_map), (key_p, pose_data) in zip(ds_depth, ds_pose):
    print(f"Depth: {depth_map.shape} | Pose: {pose_data.shape}")
    break

For local development, download only the modalities you need:

hf download timlawrenz/stratum-ffhq --include "depth/*" "pose/*" --repo-type dataset

The Broader Vision: Dataset-Agnostic Enrichment

Stratum-FFHQ is not a one-off dataset release. It is the first public output of the Stratum-HQ enrichment pipeline, a dataset-agnostic system that can apply this same multi-modal processing to any image collection.

The pipeline architecture is intentionally modular: each enrichment stage (captioning, embedding, depth estimation, segmentation, pose detection) runs as an isolated step that reads from and writes to a standardized artifact directory. Swap out the input images, and the same pipeline produces the same modalities for your domain — medical imaging, satellite photography, product catalogs, or wildlife monitoring.

This matters because the current generation of multi-modal generative models — Diffusion Transformers with cross-attention conditioning, ControlNets with spatial guidance, identity-preserving portrait generators — all demand the same set of aligned modalities. Building these models should not require rebuilding the preprocessing infrastructure from scratch every time.

In the Spirit of FFHQ

When the original Flickr-Faces-HQ dataset was released, it set a gold standard for high-quality, openly accessible data that propelled the entire generative AI community forward. With Stratum-FFHQ, we want to continue that legacy.

We believe that the next generation of AI breakthroughs shouldn’t be locked behind corporate compute clusters or proprietary APIs. By freely sharing these computationally expensive, heavily processed multi-modal embeddings, we hope to lower the barrier to entry and keep high-quality research data open, accessible, and free for everyone.

What’s Next for prx-tg

With Stratum-FFHQ eliminating the preprocessing bottleneck, the prx-tg project can finally scale. Our next milestone is a 50,000-step training run on the full 70,000-image dataset using the winning configuration from our ablation series: Muon optimizer, Asymmetric Flow Matching, and FP8 native precision — all on a single high-VRAM GPU.

The ablation phase proved the architecture. Now the data is ready. It’s time to train.

Update (July 2026): The face vector representations that condition prx-tg — a 50-d pose & expression vector from DWPose keypoints and a 64-d identity vector from AuraFace, both built on top of Stratum-FFHQ — are now documented in Face Vectors: 50-d Pose from Keypoints, 64-d Identity from AuraFace.

References

  1. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4401–4410.