Sounds and videos as prompts · EmbeddingGemma 2 → Qwen-Image 2.1

Qwen-Image 2.1 with its 8B Qwen3-VL text encoder replaced by EmbeddingGemma 2 (0.74B) and a 107M adapter (oakoliver/eg2-qwen-image-2.1-adapter). EG2 embeds text, sound and video frames through one language model, so a sound or a short video can be the prompt: write audio1, audio2, video1, video2 where Qwen-Image's edit prompts say image1. Sounds with an obvious visual (instruments, birds, bells, laughter, clapping, fireworks) work best; abstract textures (wind, zipper, typing) mostly do not. How it was built.

Research and evaluation demo. The adapter is a derivative of Qwen-Image 2.1 and ships under the Qwen Research License; no commercial use. Turbo = the 8-step Qwen-Image 2.1 Turbo transformer (fast, fixed schedule); Base = the 2.1 base transformer with the number of steps you choose (40 was used for the benchmarks). Free ZeroGPU time is limited per visitor, so Turbo is the default.

Prompt (use audio1, audio2, video1, video2 for the clips)

audio1

audio2

video1

video2

Model

Turbo (8 steps, fixed schedule) Base (choose the steps)

Size (width × height)

Seed

Random seed

Result

Run

Examples (CC0 clips from FSD50K / Freesound and Wikimedia Commons; the videos are 6-frame contact clips, exactly what the model sees):

Examples

Prompt (use audio1, audio2, video1, video2 for the clips)	audio1	audio2	video1	Model	Denoising steps (Base model)	Size (width × height)	Seed

Sounds: first 20 s of the clip, mono 16 kHz (25 soft tokens per second). Videos: 6 frames sampled uniformly. Any format ffmpeg reads (wav, mp3, flac, ogg, m4a, mp4, webm, mkv). The microphone works too: hum, clap, bark.
Engine: PyTorch + diffusers; the production path is the Rust/Metal engine described in the write-up. Attribution of the example clips is in the Space's README.