Projects / 02 / AutoBG

Computer vision + generative imaging

AutoBG

Everyday car photos.
Studio-ready results.

See the demo Explore the system Read the technical study

Upload a car photo → generate a studio composite → download the PNG.

01 / The idea

Change the setting.
Keep the car.

AutoBG turns a raw vehicle photograph into a composed studio image. High-resolution matting separates the car; a geometry-aware compositor places it, grounds the wheels, and integrates the lighting. Diffusion is applied where it adds value—not automatically across the entire image.

Problem
Inconsistent backgrounds, contaminated edges, and vehicles that look pasted into a scene.
Approach
Shared segmentation + deterministic compositing + controlled generative refinement.
Demo workflow
Upload → before / after → downloadable PNG. No manual masking.
Interface
Next.js / React frontend; FastAPI image-processing backend.

02 / System architecture

One matte.
Three possibilities.

The browser sends a multipart image request to FastAPI. Input validation and segmentation are shared; the rendering mode determines how much of the scene is generated. Results return directly as PNG bytes.

One upload. Three rendering paths.AutoBG / System map
BrowserShared backendMode dispatch
Next.js interfaceupload · before / after · download
POST /processFormData: file + template + mode
PNG responseblob → preview + local download
FastAPI gatewaydecode RGB · validate · downscale ≤ 2048px
BiRefNet HRPyTorch inference → continuous RGBA matte
Mode routershared cutout → selected renderer
templatecompositing · no diffusion
reflect_aireflect_gen · floor-only inpainting
aiaigen · full background inpainting
reflect_ai + ai reuse one cached SDXL / ControlNet pipeline.
Every request starts with the same matte. The UI requests reflect_ai; the API defaults to template when no mode is supplied.
01 / Template

Consistency first.

A fixed studio, geometric reflection, contour-following shadows, light wrap, and seeded grain. No diffusion after matting.

02 / Reflect AI

Generate selectively.

The current UI uses White Studio + reflect_ai. SDXL refines a masked floor reflection while retaining the prepared car and surrounding studio.

03 / AI background

A new environment.

Generate the wider background with SDXL, then harmonize and re-paste the car cutout with contact shadows and edge-aware integration.

03 / Matting & compositing

The small details
make it believable.

Mirrors, tires, and fine silhouette edges need a continuous alpha matte. Foreground-color recovery removes contamination from the old background. Shadows and reflections then follow the vehicle's actual contact contour, not just its bounding box.

A matte, not a binary cutoutAutoBG / System map
PrepareInferRestoreRefine
RGB photographresize to 2048² · ImageNet normalization
BiRefNet HRforward pass → sigmoid alpha
RGBA cutoutLANCZOS alpha resize to source dimensions
Edge refinementforeground recovery · tighten · anti-alias
Foreground recovery: PyMatting ML estimation when available → color-decontamination fallback otherwise.
The continuous alpha returns to the input resolution. Template and reflection paths then recover foreground colors before tightening the edge.
The deterministic studio compositorAutoBG / System map
Prepare the sceneGround the vehicleIntegrate the result
Studio backgroundsaved PNG plate or procedural wall + floor
Prepare foregroundclean edges · white balance · floor bounce
Scale & place70% canvas width · height capped at 56%
Contact contourlowest opaque pixel in each column
Reflection + shadowmirror per column · ambient band + dark core
Composite the carstudio → reflection → shadow → foreground
Realism finishscreen-blended light wrap · seeded film grain
Studio PNG1200 × 900 · RGB
This is the template path after segmentation. Geometry, contact shadows, reflection, lighting, and seeded grain are handled with NumPy, OpenCV, and Pillow.

04 / Controlled generation

Give AI a boundary.

A geometric mirror provides the reflection's initial color and shape. A contour-following mask limits the edit; Canny conditioning guides SDXL. The output is blended with the geometric result and pasted back only through the softened floor mask.

Generate the reflection. Preserve the scene.AutoBG / System map
Build an initializationConstrain diffusionBlend back locally
Base compositestudio + contact shadow + prepared car
Geometric mirrorper-column contact · fade · distance blur
Initialization imagebase + geometric floor reflection
Floor-only maskcontact contour · 50% of car height · feather
Canny conditioningedges from the initialization image
SDXL + ControlNet1024² · 22 steps · strength 0.32 · seed 7
Blend AI + geometry50 / 50 by default · soften with blur
Masked paste-backtake only the refined floor region
Final composite1200 × 900 · PNG response
Only the feathered floor band is pasted back into the base composite. The vehicle geometry is not regenerated, and the studio outside that mask is retained.
Full-background generationAutoBG / System map
ConditionGenerateGroundMatch
Place the cutoutgray canvas · background mask · Canny edges
SDXL inpaintingtemplate prompt · 30 steps · guidance 7
Ground the scenedeglow · ambient shadow · wheel cores
Match illuminationLAB color + exposure harmonization
Finish → guided-filter alpha → paste original cutout → light wrap → film grain.
The ai path generates a new scene, then harmonizes and re-pastes the original cutout. Default generation is 1024 × 1024; it is a separate path from floor-only reflection.

Model roster

Matting, generation,
and conditioning.

Matting
ZhengPeng7/BiRefNet_HRContinuous alpha inference at a default internal resolution of 2048 × 2048.
Generation
stabilityai/stable-diffusion-xl-base-1.0Shared inpainting pipeline for full backgrounds and floor-only reflections.
Conditioning
diffusers/controlnet-canny-sdxl-1.0Canny edge guidance constrains generated content to the initialized vehicle structure.
VAE
madebyollin/sdxl-vae-fp16-fixFP16 VAE used by the CUDA-based SDXL pipeline.
Optional relight
Realistic Vision V5.1 + IC-Light FBCIndependent SD1.5 module with foreground/background conditioning; not exposed by the current API.

05 / Supporting tools

Reusable studios.
Room to experiment.

An offline plate builder reconstructs an empty studio from a reference photograph. A separate IC-Light experiment conditions relighting on both foreground and background. These are supporting modules, not extra steps in every request.

Build a reusable studio plateAutoBG / System map
ReferenceRemoveReconstructReuse
Reference photoplates/reference.png or .jpg
Mask the vehicleBiRefNet alpha → 35 × 35 dilation
Empty-studio inpaintSDXL · 30 steps · ControlNet disabled
Save the plateresize back → white_studio.png
make_plate is an offline preparation utility. The compositor loads plates/white_studio.png when it exists, otherwise it builds the background procedurally.
Optional IC-Light relighting moduleAutoBG / System map
Conditioning imagesPatched SD1.5Relit output
Foreground on graycar cutout normalized on a 512² canvas
Studio backgroundtemplate image resized to 512²
VAE encodingforeground + background → 8 conditioning channels
UNet: 4 → 12 channelsnoisy latent + foreground + background
IC-Light FBC offsetsadditive weight merge · hooked forward pass
SD1.5 relight25 steps · foreground + background conditioning
Upscale the result512² → 1024² · LANCZOS
iclight is present as an independent experiment; main does not expose it as a processing mode. It is not part of the demo's reflect_ai path.

06 / Runtime tradeoffs

Choose the work
the image needs.

Template composition prioritizes consistency and speed. Reflection inpainting spends GPU time on a narrow edit; full-background generation spends more on the wider scene. These are approximate NVIDIA L4 timings reported in the project README, not new portfolio benchmarks.

Template composite

~0.7sMatting + deterministic studio compositing.

AI reflection

~13sMatting + fixed studio + SDXL floor reflection.

AI background

~18sMatting + full SDXL background generation.

Model loading and warmup are separate from request timings. Actual runtime depends on image size, configuration, and GPU. SDXL is cached and optionally preloaded at startup.