Synthetic Eye Image Generation using Foundation Models

Gaze estimation means working out from an image where a person is looking. Training such a model takes a lot of labelled images, and recording real ones is slow and expensive. Tools like UnityEyes can render a million labelled eye images within 24 hours on ordinary hardware, but they draw only the eye region on a flat grey background. Models trained on that struggle with real photos, which is known as the sim-to-real gap.

The research question

How can repainting synthetic images with foundation models improve gaze estimation model performance by closing the domain gap?

I kept every pixel of the rendered eye and let a diffusion model paint only the grey background, a technique called outpainting. Because the eye itself never changes, the gaze labels stay correct without any new annotation. I called the resulting dataset FluxEyes.

UnityEyes

FluxEyes

UnityEyes

FluxEyes

UnityEyes

FluxEyes

I wanted to stay close to the low hardware needs that make UnityEyes popular, so everything ran on a single RTX 4080 with 16 GB of VRAM. With all its components, Flux.1 needs 33 GB, so parts of it spill into far slower CPU memory. Reducing model parameters from 32 to 8 bit and moving to flex2, a smaller community model, cut generation from 42 seconds to about 7 seconds per image. That is still more than two months for a million images, so I set a budget of 48 hours and 20,000 images.

I compared three datasets, each with 20,000 training and 5,000 test images: UnityEyes, FluxEyes, and real photos from ETH-XGaze, filtered to the same head poses and with different people in training and test. On each one I trained a ResNet50, pretrained on ImageNet, for 25 epochs, and then tested every model on every test set.

• On real photos, the error fell from 19.47° to 17.87°, about 8%, just by repainting the background.
• The model trained only on UnityEyes did far worse on repainted images: 7.24° instead of 1.65°. I read that as a sign it had leaned on the grey background.
• The model trained on real photos scored almost the same on both synthetic sets, 22° and 21.8°. Repainting did not disturb the parts of the image that matter for gaze.