ComfyUI Experiments - One LoRA, Eight Video Pipelines
AI/ML · ComfyUI · Generative Video · LoRA Training · Wan 2.2 · LTX 2
INTRODUCTION
Late 2025 was the explosion year for open-source video generation. Wan 2.2 dropped. LTX 2 caught up. A stack of derivative pipelines started shipping every week — InfiniteTalk for lip sync, HuMo for animated character drive, WanAnimate for motion transfer, SVT for long-form, Detailer for upscaling. Most case studies pick one model and praise it. I wanted to actually compare — and to do it honestly, that meant training a dedicated LoRA per base model, since cross-family transfer has real limits.
This work was done as part of AI R&D at Aeos Labs. I trained three custom character LoRAs — one on Wan 2.1, one on Wan 2.2, one on LTX 2 — and ran them across eight different ComfyUI pipelines, mapping each pipeline to the LoRA that matches its base architecture. Same subject, same identity prior, three trained adapters, eight different ways to get that identity onto a moving frame. ~68 final outputs. The result is a side-by-side picture of what each pipeline is genuinely best at, plus a recommendation matrix at the end.
This work was done as part of AI R&D at Aeos Labs. I trained three custom character LoRAs — one on Wan 2.1, one on Wan 2.2, one on LTX 2 — and ran them across eight different ComfyUI pipelines, mapping each pipeline to the LoRA that matches its base architecture. Same subject, same identity prior, three trained adapters, eight different ways to get that identity onto a moving frame. ~68 final outputs. The result is a side-by-side picture of what each pipeline is genuinely best at, plus a recommendation matrix at the end.
The method
One subject. Three LoRAs (Wan 2.1, Wan 2.2, LTX 2). Eight pipelines. A shared question across all of them: does the identity hold, does the motion or audio sync work, how long can it generate, where does it break? I wasn't looking for "the best model" — I was looking for which model wins which job, with the right LoRA for each base.
The LoRA
Dataset — ~50–60 mixed photos + video-clip frames per LoRA. Real-world inputs: phone-camera quality, multiple lighting setups, a mix of expressions and angles. No studio shoots — the dataset is what you'd actually have for a real-person LoRA.
Three LoRAs trained:
- Wan 2.1 LoRA — feeds the latentsync pipeline (Wan 2.1 base).
- Wan 2.2 LoRA — feeds Wan vanilla, InfiniteTalk, HuMo, WanAnimate, SVT, and Detailer (all Wan 2.2 base).
- LTX 2 LoRA — feeds the LTX 2 audio-sync pipeline. Trained separately because LTX 2's architecture differs from the Wan family enough that a dedicated LoRA outperforms cross-family transfer for production-grade output.
Training stack — AI Toolkit on RunPod. Stable, well-documented, runs cleanly on rented A100 / H100 instances. Cheaper than dedicated hardware, more reliable than any "train in the browser" service I tried.
The right LoRA per architecture matters. Cross-family transfer (using a Wan-trained LoRA on LTX 2) does technically work, but it loses fidelity that a dedicated LoRA recovers — clearest in the lip-sync test below.
1. Vanilla Wan + LoRA - does the identity hold?
Pipeline: Wan 2.2 base + LoRA applied directly. No audio, no motion drive, no extras.
Take: Identity holds in most clips, drifts in some - but close enough to validate the LoRA was real. This was the baseline check. If it didn't work, nothing downstream would.
Take: Identity holds in most clips, drifts in some - but close enough to validate the LoRA was real. This was the baseline check. If it didn't work, nothing downstream would.
Best at: Sanity-checking that your LoRA actually learned the subject before you spend cycles on downstream pipelines.
2. InfiniteTalk + LoRA - strong HeyGen alternative
Pipeline: Wan 2.2 + InfiniteTalk audio-driven lip sync + LoRA.
Take: Worked really well. I'd put this on par with HeyGen for general use — it both holds person identity and delivers convincing lip sync, which is the combination most open-source lip-sync tools fail. One real caveat: Indian English pronunciation doesn't quite land. The base model is trained heavily on Chinese audio, and you can hear the mismatch when the source audio has South Asian phonemes the model wasn't expecting. Not a dealbreaker for neutral English; a real gap if the subject has an Indian accent.
Take: Worked really well. I'd put this on par with HeyGen for general use — it both holds person identity and delivers convincing lip sync, which is the combination most open-source lip-sync tools fail. One real caveat: Indian English pronunciation doesn't quite land. The base model is trained heavily on Chinese audio, and you can hear the mismatch when the source audio has South Asian phonemes the model wasn't expecting. Not a dealbreaker for neutral English; a real gap if the subject has an Indian accent.
Best at: Audio-driven dialogue when the speaker has clean, neutral English. Production-grade lip sync, no service fees, runs locally.
3. HuMo + LoRA - animated backgrounds, weaker face
Pipeline: HuMo audio-driven character animation + LoRA.
Take: Good for scenes where you want the background or environment to be animated/dynamic along with the character - HuMo handled scene-level motion better than InfiniteTalk did. The trade-off: identity preservation, even with the LoRA, wasn't as strong as InfiniteTalk's. So you get a more dynamic world with a slightly less faithful face.
Take: Good for scenes where you want the background or environment to be animated/dynamic along with the character - HuMo handled scene-level motion better than InfiniteTalk did. The trade-off: identity preservation, even with the LoRA, wasn't as strong as InfiniteTalk's. So you get a more dynamic world with a slightly less faithful face.
Best at: Scenes where the character and the world are both meant to move. Pick this when InfiniteTalk feels too static.
4. WanAnimate + LoRA - best-in-class motion transfer
Pipeline: WanAnimate (motion-driven from a reference video) + LoRA.
Take: Best in class. With the LoRA layered on top, this takes things to another level - you can puppet the LoRA character with arbitrary reference motion. This is the pipeline I reach for when I want the LoRA character performing a specific physical action I already have on video.
Take: Best in class. With the LoRA layered on top, this takes things to another level - you can puppet the LoRA character with arbitrary reference motion. This is the pipeline I reach for when I want the LoRA character performing a specific physical action I already have on video.
Best at: Motion transfer - "puppeteer the LoRA with this reference video."
5. Wan + latentsync - alternate single-character lip sync
Pipeline: Wan 2.1 + latentsync for audio-driven single-character lip sync.
Take: Worth knowing as a fallback when InfiniteTalk doesn't cooperate, but less natural than InfiniteTalk on the same single-character lip-sync job. Tested on single character only - didn't push into multi-character scenes.
Take: Worth knowing as a fallback when InfiniteTalk doesn't cooperate, but less natural than InfiniteTalk on the same single-character lip-sync job. Tested on single character only - didn't push into multi-character scenes.
Best at: A second lip-sync option to keep in the toolbox. Reach for InfiniteTalk first; latentsync if InfiniteTalk produces artifacts on a specific clip.
6. Wan SVT + LoRA - long-form via chained nodes
Pipeline: Wan 2.2 + SVT long-video extension. Chained 5–6 nodes per generation, each ~3–5 seconds, totaling ~20-second coherent clips.
Take: Pushed Wan to ~20-second outputs that actually hold continuity. This is genuinely useful — most Wan-family setups cap out at ~5 sec; chaining through SVT gets you to short-film-clip lengths without obvious seams. The math checks out in the outputs: 5-6 nodes × 3-5 sec each.
Take: Pushed Wan to ~20-second outputs that actually hold continuity. This is genuinely useful — most Wan-family setups cap out at ~5 sec; chaining through SVT gets you to short-film-clip lengths without obvious seams. The math checks out in the outputs: 5-6 nodes × 3-5 sec each.
Best at: Long-form (>10 sec) shots where coherent identity matters. Pair with Detailer if quality drops as the chain extends.
7. Wan Detailer + LoRA - identity-preserving upscaling
Pipeline: Wan Detailer + the same LoRA used during the enhance pass.
Take: Most underrated finding of the whole project. Standard upscalers improve resolution but often change the face — the person no longer looks like themselves at higher resolution. Running Detailer with the same LoRA active during the enhance pass keeps the upscaled output recognizable as the original subject. Simple idea, big difference in practice.
Take: Most underrated finding of the whole project. Standard upscalers improve resolution but often change the face — the person no longer looks like themselves at higher resolution. Running Detailer with the same LoRA active during the enhance pass keeps the upscaled output recognizable as the original subject. Simple idea, big difference in practice.
Best at: Upscaling / finishing pass when you can't afford the face to drift. The "same LoRA twice" trick is the take-home lesson.
8. LTX 2 audio-sync + LoRA - cross-family, best lip sync
Pipeline: LTX 2 (unified image + audio + video model) + Wan-trained LoRA + audio sync.
Take: Best-in-class for the specific job of "I have a still image of the subject + an audio track + I want a lip-synced clip back." Because LTX 2 is a unified audio+video model, the lip sync feels more naturally coupled to the audio than the Wan-family pipelines, which bolt audio on top. Surprisingly, the Wan-trained LoRA still worked when applied to LTX 2 — cross-family transfer wasn't as bad as I expected.
Take: Best-in-class for the specific job of "I have a still image of the subject + an audio track + I want a lip-synced clip back." Because LTX 2 is a unified audio+video model, the lip sync feels more naturally coupled to the audio than the Wan-family pipelines, which bolt audio on top. Surprisingly, the Wan-trained LoRA still worked when applied to LTX 2 — cross-family transfer wasn't as bad as I expected.
Best at: Single image + audio → lip-synced clip. Cleanest output for that exact job. Less affected by the Indian-accent gap that hits InfiniteTalk.