The goal is to describe a person’s face from a photo. Given an image, we want two things: who they are, and how they’re posed. The deliverable is two orthogonal vectors — a 50-dimensional pose & expression encoding that captures everything transient about a photograph (head angle, mouth opening, gaze direction), and a 64-dimensional identity encoding compact enough for a text-to-identity Prior yet lossless for cross-shoot face verification at R@1 = 0.842.

These two vectors are empirically orthogonal (R² ≈ 0). Changing the head tilt doesn’t shift ethnicity. Changing the identity doesn’t alter the expression. The firewall had to be measured and proven, not assumed.

This post documents the experimental journey that settled both vectors: what we built, what we killed, and why.

The Problem

The deliverable is two orthogonal vectors:

  1. A pose & expression vector (z_g), 50-dimensional, interpretable, and identity-blind — it captures yaw, pitch, mouth opening, and expression: everything that varies between photos of the same person.
  2. An identity vector (AuraFace-LDA), 64-dimensional, compact enough for a text-to-identity Prior, yet lossless for face verification — it captures who the person is, independent of how they’re photographed.

The orthogonality between these two streams is an empirical result, not a design assumption. We measured it with ridge regression (R² = −0.033 — worse than predicting the mean) and tracked it across every experimental phase. The firewall is real, but it emerged from inductive bias, not architectural enforcement.

Datasets

We work with two image corpora throughout this project.

FFHQ (Flickr-Faces-HQ)

A public dataset of 70,000 high-resolution face images (1024×1024), one image per Flickr identity, created by NVIDIA Corporation (Karras et al., 2019) and released under CC BY-NC-SA 4.0. We use stratum-ffhq (see our dataset announcement), our own preprocessed variant built with Stratum HQ, which layers DWPose keypoints, DINOv3 embeddings, T5 captions, and Sapiens segmentation/depth/normals onto every FFHQ image. This is the workhorse: the z_g encoder is fit on 69,851 FFHQ faces, and the text→identity Prior is trained on FFHQ’s single-image-per-identity captions.

Limitation: one image per identity means we can’t test cross-shoot generalization on FFHQ. A model that memorizes background lighting or clothing can score well on FFHQ’s held-out split without actually recognizing identity.

Proprietary Multi-Shoot Dataset

A commercial face dataset with ~70,000 approved images across ~323 identities, each identity photographed across multiple shoots (different days, lighting, clothing, hair styling, camera setups). Median 209 images per identity, 321 identities with ≥10 images. This is the only substrate that can answer “does this representation survive a different photo session?”

We use it exclusively for cross-shoot validation — Fisher J identity-blindness gates, verification AUC, and the GT-LDA kNN ceiling test. All gates that report “cross-shoot” or “multi-identity” numbers in this post are run on this dataset. It is never used for training.

All images in the proprietary dataset are commercial-licensed and cannot be reproduced. Figures in this post that show faces or landmarks are drawn from FFHQ only.

Part 1: zg — The 50-d Pose & Expression Vector

1.1 The Raw Material: DWPose Keypoints

zg starts not with pixels but with 68 facial landmark coordinates from DWPose (Yang et al., 2023), a whole-body pose estimator. DWPose produces 133 COCO-WholeBody keypoints per image. We slice out indices 23–90 — the 68 dense iBUG/300W facial landmarks (Sagonas et al., 2013) (jawline, eyebrows, nose bridge, lip contours) — and drop the confidence channel, leaving a flat 136-dimensional vector:

(133, 3)  →  slice [23:91]  →  (68, 3)  →  drop confidence  →  (68, 2)  →  flatten  →  ℝ¹³⁶

Indices 0–16 (body), 17–22 (feet), and 91–132 (hands) are discarded. Only the face matters — we want keypoints that move with expression and pose, not body language.

The raw data is noisy. DWPose jitter, expression asymmetry, and focal distortion all live in those 136 coordinates. The job of the encoder is to factor out identity-relevant morphology (which lives in AuraFace) and retain only the transient pose and expression — what varies between photos of the same person: head angle, mouth opening, gaze direction, expression asymmetry.

1.2 Why Plain 2D GPA Failed

The first attempt was straightforward: run Generalized Procrustes Analysis (GPA) to strip translation, uniform scale, and in-plane rotation (roll), then fit PCA.

It produced beautiful components. C₁ was yaw. C₂ was pitch.

That’s fatal. The guiding principle is that the identity vector must describe the invariant person. Yaw and pitch are transient camera state. Encoding them in the identity vector makes it partly a description of a camera event — a semantic category error.

The math was correct; the objective was wrong. 2D GPA is blind to out-of-plane 3D rotation. A head turning left↔right produces huge variance in 2D coordinate space, and PCA — doing exactly what it is optimized to do — shoves that variance into the top components.

The PASS verdict for Phase 1 was revoked within 24 hours.

1.3 The Fix: 3D Frontalization via EPnP

The solution is algebraic, not data-driven: estimate the head’s 3D rotation from the 68 keypoints, rotate to a canonical frontal frame, then run PCA on the pose-normalized shapes.

Concretely:

  1. EPnP (Efficient Perspective-n-Point) (Lepetit et al., 2009) estimates head rotation R by fitting the 68 2D keypoints against a canonical 3D mean-face template (the 300W/iBUG 3D reference), using an orthographic camera model.
  2. The keypoints are lifted to 3D using a hand-built neutral radial depth prior, then rotated by R⁻¹ to face forward.
  3. The frontalized points are reprojected to 2D.
  4. A light 2D GPA (center, uniform scale, in-plane roll) is applied to the now-pose-normalized shapes.
  5. PCA is fit on the result. The remaining variance captures transient facial configuration — yaw, pitch, mouth opening, expression, and keypoint-level asymmetry that reflects how the face is posed, not whose face it is.

The EPnP solver is deterministic, CPU-only, and scales to 70k images in under two minutes. No GPU, no learned model, no 3DMM needed.

The depth bonus: rotating a profile to frontal doesn’t discard the profile-only signal (nose projection, brow ridge, chin protrusion). It preserves it — the reprojected X-spread of the nose keypoints encodes the projection depth that a frontal-only data filter would delete.

1.4 Proving It Worked: The Pose-Invariance Probe

We tested the encoder with a controlled probe: take one identity, synthesize keypoint configurations at ±30° yaw, encode each, and measure how much z_g varies.

Encoder Mean per-component std of zg across yaw
Raw Phase 1 (2D GPA only) 1.45
Pose-normalized (3D EPnP) 0.29
Improvement 4.9×

The raw encoder leaks ~1σ of yaw variance into the sliders. The pose-normalized encoder suppresses it by nearly 5×. Variance retention stayed unchanged at 99.94% — we didn’t lose information; we just stopped encoding the wrong thing.

Visually, the ±3σ traversals confirm the decoupling. After frontalization, C₁ and C₂ stay frontal and bilaterally symmetric across the full ±3σ range — the lateral nose-vs-jaw shear (yaw) and vertical whole-face compression (pitch) that dominated the raw encoder are gone.

±3σ traversal of PCA Component 1 after 3D frontalization — faces remain frontal and bilaterally symmetric across the full range

±3σ traversal of PCA Component 2 — vertical compression (pitch artifact) eliminated

1.5 Why 50 Dimensions?

The PCA was fit on 69,851 FFHQ faces from the stratum-ffhq dataset. The dimension k = 50 was chosen through three lines of evidence:

1. Scree / retained variance. At k = 50, the encoder retains 99.987% of the variance in the 136-dimensional keypoint space — essentially lossless.

Scree plot: variance explained vs. number of PCA components. Elbow at PC 5–8; 99.987% retained at k = 50

2. Reconstruction error vs. k.

k (components) Reconstruction RMSE Interpretation
1 0.0082 Coarse face shape
10 0.0018 The “elbow” — captures ~90% of detail
50 0.00037 Pixel-level asymmetry resolution

At 50 PCs, the reconstruction error is ~0.00037 in normalized coordinate space — barely above numerical precision. The scree plot’s elbow is at PC 5–8, but 50 PCs resolve the fine-grained pose and expression detail (subtle mouth asymmetry, eyebrow arching, eyelid position) that a 5-PC approximation would erase. Going higher would capture DWPose detector noise, not biology.

3. Architectural consistency. The original E design envisioned a 3-partition vector [z_g | z_d | z_a] ∈ ℝ¹⁵⁰, with each partition at 50 dimensions. zd (monocular depth) and za (surface normals) were both conclusively killed by the verification-AUC gate (ΔAUC −0.023 to −0.039 — they subtract identity signal). But the 50-d convention for zg was settled before those deaths, and it held up on its own merits.

The shipped production encoder lives at experiments/geometry_pca/output/encoder_production.npz. Contents: PCA components (50×136), canonical 3D template (68×3), whitening statistics (µ and σ per component), GPA reference mean, and explained variance ratios. The canonical template is persisted inside the encoder so inference frontalization is fully reproducible. Full fit time: ~107 seconds on CPU.

1.6 Identity-Blindness: zg Carries Almost No Identity Signal

This is the most important measurement in the project — and the one that corrected a months-long premise error.

The original premise was that the 50 zg values would serve as identity sliders. We’d sweep an axis and watch the person’s face structure change. Empirically, this turned out to be wrong.

We measured identity separability via Fisher’s criterion J = tr(S_W⁻¹ S_B) on our proprietary multi-shoot corpus: 69,110 images across 323 identities.

Corpus N images N identities Global Fisher J Axes with J > 0.15
Legacy (small-N) 1,448 101 ~0.08 27
Full (2026-06-27) 69,110 323 0.059 6

When we scaled from 1,448 images to 69,110, Fisher J collapsed from 0.08 to 0.059. The number of axes with meaningful identity separability (J > 0.15) dropped from 27 to 6. Twenty-two legacy “morphology” axes fell into noise — they were small-N optimism. Even the surviving 6 are modest: at J ≈ 0.15–0.20, they carry orders of magnitude less identity signal than AuraFace.

The verification AUC confirms this. zg’s standalone identity verification AUC is 0.67–0.69 on face-crop DWPose — above chance (0.5) but far below AuraFace’s 0.797–0.969. zg is a weak identity carrier, and framing it as an identity space was a category error.

Text→zg is a definitive FAIL. When we trained a Prior model to predict zg from T5 text embeddings, the per-dim MSE ratio was 1.75 — 75% worse than predicting the global mean (ratio 1.0). Text does not predict zg. This makes sense: visual-language model captions describe identity attributes (skin tone, hair, face shape at the semantic level), not the fine-grained keypoint configuration produced by DWPose under a specific camera, lens, and expression.

zg is a pose/expression control space. Its axes govern head angle, mouth opening, gaze direction, and expression asymmetry. It is not an identity space. Identity lives in AuraFace.

1.7 Orthogonality: zg ⟂ AuraFace

We verified that the two streams don’t share a linear subspace. Ridge regression from zg (50-d) → AuraFace (512-d) on 8,000 FFHQ pairs:

Held-out R² = −0.033 — worse than predicting the mean.

The 50-d keypoint basis explains essentially none of the 512-d identity embedding. The two streams are genuinely complementary. A planned “project zg out of AuraFace” orthogonalization step was dropped as pointless — there’s nothing linear to remove. The empirical firewall is a natural property of the representations.

Important caveat: this is measured orthogonality, not designed. It’s a happy accident of the different inductive biases of DWPose keypoints vs. AuraFace’s deep embedding space. We track it; we don’t assume it.

Part 2: AuraFace-LDA — The 64-d Identity Space

2.1 Why Not Raw AuraFace 512-d?

AuraFace produces a 512-dimensional unit-norm embedding from face crops. Built on the ArcFace additive angular margin loss (Deng et al., 2019), it’s a strong identity carrier — raw dinov3_cls (AI, 2025) face-crop AUC is 0.766, and flesh-masked DINOv3 patch tokens hit AUC 0.797 (fully cross-shoot verified).

But AuraFace has a critical structural property: no low-rank PCA structure.

Pooled PCA on 140,217 FFHQ + proprietary dataset vectors:

  • PC1 explains 2.08% variance — an order of magnitude less than typical (PCA on faces usually sees 20–40% in PC1)
  • Flat spectrum, participation ratio ≈ 217/512
  • PC1 is a domain artifact (FFHQ vs proprietary dataset capture style, separation 2.05), not identity

Unsupervised GANSpace-style semantic sliders therefore don’t exist in AuraFace. You can’t just take the top-K PCA components and get interpretable identity axes. The identity signal is distributed across the full 512-d hypersphere.

2.2 Enter Supervised LDA

The alternative is supervised: Linear Discriminant Analysis finds the directions that maximize between-persona separation relative to within-persona scatter — using identity labels.

The LDA computation:

  1. Gather: 140,217 pooled AuraFace vectors across FFHQ and our proprietary multi-shoot dataset
  2. Clean: Remove PC1 (domain artifact) and the yaw direction (R² = 0.41 with head pose, costs zero identity: ΔAUC < 0.0001)
  3. Fit LDA: on 259 train personas (PC1-removed, centered), producing k discriminative directions
  4. Test: project 64 held-out identities onto the basis and verify generalization

LDA dimension sweep (Phase 5-prep, 259 train / 64 held-out personas):

LDA dimensions Verification AUC % of full 512-d
512 (raw, cleaned) 0.969 100%
80 0.965 99.6%
64 ~0.964 99.5%
40 0.956 98.7%
20 0.934 96.4%

The ceiling test (reconstruct 512-d from LDA → verify vs raw AuraFace) tells a complementary story about pure representational loss. The top-80 LDA dimensions recover 99.6% of full-512 verification power. The representation is highly compressible — the discriminative identity information lives in a low-dimensional subspace.

2.3 The Ceiling Test: Why Exactly 64-d?

We ran an exhaustive ceiling test: take the ground-truth (GT) LDA coordinates → reconstruct to 512-d → verify against the raw AuraFace vector. This measures how much identity information is lost purely by the LDA projection.

Representation AUC vs raw AuraFace
raw vs raw (sanity check) 0.9989
cleaned (PC1+yaw removed) vs raw 0.9989
GT LDA-64 reconstruction vs raw 0.9998
GT LDA-32 0.9765
GT LDA-16 0.8658

The LDA-64 reconstruction scores marginally above the raw baseline (0.9998 > 0.9989). This is real, not a measurement error — the LDA projection acts as a denoising step. Raw AuraFace 512-d vectors include ~448 dimensions of within-persona noise (head pose, lighting, expression variation) that make same-identity pairs slightly less similar than they could be. Projecting onto the 64 discriminative directions zeroes out those noise dimensions, then reconstructing back to 512-d produces a cleaner vector where same-persona pairs are more tightly clustered while between-persona separation is preserved. The +0.0009 lift is small but consistent — it’s the same mechanism by which PCA denoising can improve downstream classifier performance.

The 64-d LDA projection is essentially lossless. AUC 0.9998 is near-perfect — virtually all verification-relevant identity survives the compression. 32-d is still strong (0.9765) but measurably degraded. 16-d loses significant identity signal (0.8658).

64 was the settled choice because:

  • Near-lossless (ceiling 0.9998) — no information-theoretic regret
  • Compact enough for the Prior to predict from text (512 → 64 is an 8× compression)
  • Power-of-2 dimension — friendly to neural network architectures (layer widths, attention heads)
  • Matched the k = 50 convention of zg — both vectors are in the same order of magnitude, keeping any downstream model’s weights balanced

2.4 Cross-Shoot Validation: The Proof AuraFace-LDA Actually Works

All the above measurements were on FFHQ (1 image per identity). The real test is cross-shoot: can the LDA representation recognize the same person across different photo sessions with different lighting, clothing, hair styling, and camera setups?

Phase 5b GT-LDA Ceiling Test on our proprietary multi-shoot corpus (2,999 query images, 30,000 index images, 242 personas, held-out shoots):

Query source R@1 R@5 R@10
Prior (text→LDA, stochastic FM) 0.010 0.037 0.072
Random null 0.006 0.030 0.053
GT-LDA (real AuraFace) 0.842 0.922 0.941

R@1 = 0.842 cross-shoot. When the query is the real held-out-shoot AuraFace vector (no Prior, no text), the LDA-64 representation retrieves the correct identity 84.2% of the time from a different photo shoot. This is the first empirical proof that AuraFace-LDA is a genuine cross-shoot identity carrier — something no FFHQ-only gate could establish.

The text Prior (R@1 = 0.010) fails at exact-person retrieval. But the GT-LDA ceiling proves the problem is in the Prior, not in the representation. The LDA space is sound. Text is many-to-one with identity — “blonde woman with blue eyes” describes millions of people, and no text encoder can disambiguate them.

2.5 LDA Axes: A Visual Check

The top LDA axes carry real identity signal — the attribute AUC measurements (skin 0.889, hair 0.712) confirm that stepping along them produces faces sharing semantic attributes. But what do the axes look like?

Below is a contact sheet of the ±3σ extremes for LDA axes 1–6. Each face is the nearest FFHQ recall at that coordinate — the FFHQ image whose LDA projection lands farthest along that axis in either direction. The LDA basis itself was fit on the pooled FFHQ + proprietary dataset, but we show FFHQ extremes here for reproducibility.

LDA axes 1–6 contact sheet: ±3σ nearest-recall extremes from FFHQ. Axes separate on skin tone, age, gender, ethnicity — but these are discriminative directions, not validated attribute maps

We leave the interpretation open — these are extreme projections on the LDA basis, not a validated attribute map. A few things to keep in mind when reading the sheet:

  • These are nearest-recall extremes from FFHQ only. The faces at ±3σ are not generated to maximize the attribute; they’re the real FFHQ images that happen to score highest/lowest on that axis. Some matches are strong (LDA1 cleanly separates skin tone), others are noisy (LDA5 mixes age, gender, and expression).
  • The LDA basis was fit on a different corpus than the one shown. FFHQ extremes highlight demographic axes (age, gender, ethnicity) that both datasets share; axes dominated by proprietary-dataset-specific confounds (occlusion, styling) are attenuated here.
  • No attribute labels have been validated against ground-truth annotations. The axes are discriminative directions in AuraFace embedding space — they maximize identity separability, not semantic purity. Nuisance variables that correlate with identity (makeup, lighting, age) leak into the basis.

FFHQ images used under CC BY-NC-SA 4.0 by NVIDIA Corporation. Image IDs: 55145, 34801, 68584, 68788, 56527, 01970, 35887, 17509, 28380, 02280, 54895, 66223, 13242, 40524, 11530, 06247, 33506, 44375, 63815, 30081, 46426, 23213, 07599, 12699, 66304, 65163, 28005, 04736, 57196, 19382, 17192, 28236, 42368, 11460, 31870. See FFHQ license terms.

Part 3: What We Learned About Methodology

This project generated as many methodological lessons as it did vector representations.

The Keypoint Resolution Trap

We initially reported zg’s identity verification AUC as 0.541 — “very weak identity carrier.” This was wrong. The same frozen encoder, same 1,429 images, only the DWPose source differed:

Pose source zg AUC
Editorial-frame DWPose (face ≈140px) 0.541
Face-crop DWPose (face ≈695px) 0.671

A +0.13 AUC swing purely from keypoint measurement precision. The trap: editorial keypoint confidence was HIGHER (0.944 vs 0.865). Confidence ≠ precision. DWPose is confident about imprecise keypoints. Always benchmark your encoder at the resolution your downstream pipeline uses.

The Trace-J Blindness

Trace-J tr(S_W⁻¹ S_B) / K averages across components, so it’s blind to complementarity. We caught this when a partition ×1.7’d trace-J but verification AUC fell −0.004. Multivariate-J tr(S_W⁻¹ S_B) inflates with dimensionality at K ≈ 100. The canonical gate metric is now verification AUC — scale-invariant, threshold-independent, and immune to dimensionality mirages.

Every Transfer Gate Needs a Random-Projection Null

Phase 3b’s DINO→slider bridge technically passed (AUC 0.704 > 0.51 bar) — but random 50-d DINO projections scored 0.712 ± 0.007. The bridge was worse than random — it destroyed identity. Without the null, we’d have shipped a broken “fast path.” Every gate that claims transfer of identity or semantic information must include the random-projection null of its input representation.

The Do-Nothing Null

Before computing any PASS metric, compute the null: predict-mean for MSE, AUC = 0.5 for verification, random-projection for retrieval. A PASS that doesn’t beat its named null is not a PASS. Two verdicts were revoked this session alone for missing this discipline.

Negative Results Are Product

zd (monocular depth) and za (surface normals) were each built, fit, cached, evaluated, and killed — at a cost of roughly a week each. The scientific conclusion — “monocular volumetric models hallucinate plausible generic geometry” — saved us from building downstream systems on Sapiens maps, which would have been an expensive dead end. The documents stay in the ledger permanently. Negative results aren’t failure; they’re the guardrails that keep the project on the viable path.

Part 4: The Numbers at a Glance

Property zg AuraFace-LDA
Dimension 50 64
Source DWPose 68-point iBUG keypoints AuraFace 512-d embeddings
Pipeline EPnP 3D frontalize → GPA → PCA → whiten PC1+yaw removal → LDA projection
Retained variance / ceiling 99.987% AUC 0.9998 (ceiling)
Fit corpus 69,851 FFHQ faces 140,217 FFHQ + proprietary dataset vectors
Fit time ~107s CPU ~minutes CPU
Identity Fisher J 0.059 (very weak carrier) — (the carrier itself)
Verification AUC (standalone) 0.67–0.69 0.964 (64-d) / 0.969 (512-d)
Cross-shoot R@1 — (not identity) 0.842 (GT-LDA ceiling)
Orthogonality to other R² = −0.033 vs AuraFace R² = −0.033 vs zg
Text predictability FAIL (ratio 1.75, worse than null) 0.687 AUC (coarse attribute region)
Role Pose/expression control Sole identity carrier

References

  1. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4401–4410.
  2. Yang, Z., Zeng, A., Yuan, C., & Li, Y. (2023). Effective Whole-body Pose Estimation with Two-stages Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.
  3. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., & Pantic, M. (2013). 300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge. Proceedings of the IEEE International Conference on Computer Vision Workshops.
  4. Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An Accurate O(n) Solution to the PnP Problem. International Journal of Computer Vision, 81(2), 155–166.
  5. Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4690–4699.
  6. AI, M. (2025). DINOv3: Self-supervised learning for vision at unprecedented scale. ArXiv Preprint.