Making Qwen and ComfyUI take turns on one GPU

I wanted to chat with Qwen locally and use it as my art and movie director.

Not just ask it for a prompt and copy that prompt somewhere else. I wanted to discuss a shot, let Qwen call ComfyUI, have it inspect the image or video, talk about what worked and render the next version. A small creative loop running entirely on my machine.

Instead, I kept running out of VRAM.

Qwen ran perfectly well on the 5090. My ComfyUI workflows also ran perfectly well on the 5090. The problem appeared when the render started while Qwen was still loaded in Ollama. I would unload Qwen by hand, run the workflow, unload ComfyUI’s models and bring Qwen back for the next reply.

I was supposed to be discussing lighting and camera movement. I was doing manual VRAM housekeeping between messages.

After enough out-of-memory errors, I stopped treating this as annoying setup and built the handoff into OpenCode.

01 / Why it ran out of VRAM

Ollama and ComfyUI both keep models loaded on purpose

Neither application was doing anything wrong. Ollama uses keep_alive because unloading a model after every answer would be painfully slow. ComfyUI also keeps models loaded because reading several gigabytes of weights again for every render would normally be a terrible idea.

From OpenCode’s point of view, my workflow was sequential:

Qwen → prompt → ComfyUI → asset → Qwen → revision

The applications did not know about that sequence. Returning a response from Ollama did not mean Qwen had left the GPU, and finishing one ComfyUI tool call did not necessarily mean its models were gone.

There was a second trap. An MCP tool can return before ComfyUI has really drained its queue. If I released the GPU in OpenCode’s normal after hook, I could hand it back to Ollama while the render was still running.

So what does “finished” mean here?

For Ollama, it means the model has disappeared from /api/ps, not that an unload request returned 200. For ComfyUI, it means /queue has no running or pending work and /system_stats reports enough free memory. The APIs already exposed the truth; I needed to stop treating a polite response as proof.

The plugin therefore waits for those observable states. It does not hand the card over until the previous model has actually disappeared and the memory is available.

02 / What I built

The plugin hands the GPU over in two stages

I wrote opencode-comfy-vram-gate, a small OpenCode plugin with no runtime dependencies. It wraps the heavy ComfyUI MCP calls and gives each one an exclusive lease on the GPU.

On the way into a render, the plugin creates an atomic directory lock with a heartbeat. It checks ComfyUI’s queue first; if somebody is already rendering, it stops there without touching Ollama. Otherwise it sends keep_alive: 0 for the loaded Ollama models, polls /api/ps until they disappear, calls ComfyUI’s /free endpoint and waits for the configured amount of VRAM to become available.

Only then does the MCP call go through. The plugin forces wait: true and raises the timeout so a long video render cannot pretend to be a quick background job.

On the way back, it waits for /queue to become genuinely empty, calls /free again, checks the memory and releases the lease. Ollama is not restarted manually; Qwen reloads in the normal way when OpenCode asks for the next response.

That is the whole trick. The VRAM is never shared in parallel. Qwen gets the chair, leaves it, ComfyUI sits down, leaves it, and Qwen comes back.

There is a cost, obviously: loading model weights takes time. If both models fit together, this plugin gives you an elaborate way to make them slower. On my machine they did not fit, so slower was a considerable improvement over crashing.

03 / Failure recovery

The first version could deadlock after a failed render

The first release could hand the GPU across and back. Then a ComfyUI tool failed and OpenCode never called the normal after hook.

Now the plugin had two bad options. Delete the lease immediately and Ollama might reload while ComfyUI was still rendering. Keep it forever and the next render from the same session would deadlock against its own abandoned lock. There was also a more entertaining race: Qwen could reload before the tool-error event reached the plugin, occupying the VRAM again just as recovery began.

Version 0.1.1 came from following those failures rather than pretending the lifecycle was cleaner than it was. Terminal tool errors now trigger recovery immediately. Session idle, error and deletion events catch a missing hook. Before a new heavy call, the plugin checks whether that session left an orphan behind. Recovery also unloads Ollama a second time if the model managed to sneak back in.

The lease itself is an atomic directory with owner metadata and a heartbeat. A later process may reclaim it when the owner PID is dead on the same host or the heartbeat is stale. It sounds slightly dramatic until two OpenCode sessions both decide that the same GPU is free. Then the lock is the cheapest component in the machine.

The repository has 17 tests for these cases, using mock Ollama and ComfyUI servers. One makes the tool return before the render finishes. Another reloads Ollama during recovery. Another tries to take the lease from a second caller. Running npm test exercises the ugly paths without evicting the real Qwen from your workstation, which feels like the minimum courtesy a VRAM plugin should offer.

04 / Scope

It manages VRAM, not the creative process

It does not choose Krea, LTX, MiniMax or any other ComfyUI model. It does not select a workflow, write the prompt, pick a LoRA, inspect the output or decide whether another iteration is worthwhile. OpenCode’s instructions, skills and MCP tools already have plenty of opinions about those things.

The gate has one boring question: whose turn is it?

Version 0.1 is deliberately narrow: Linux, OpenCode 1.x, Ollama, one NVIDIA card and a local ComfyUI server. I have not validated native Windows, and unified memory on a Mac is a different problem. I would rather write that sentence than add “cross-platform” to the README and outsource the surprise.

I would also avoid unloadPolicy: all on a shared Ollama server unless making everybody else’s models disappear is part of the plan. The plugin can use an explicit allowlist instead. Neither Ollama nor ComfyUI gains authentication from this code, so both endpoints still belong on a trusted network.

And no, 17 tests against mock services do not make this a hardware compatibility lab. They make me confident about the state machine and the failures I already know. More cards, drivers and MCP implementations will find new ones. That is what version numbers are for.

Back to the experiment

Now I can continue the conversation after the render

The plugin was never the experiment I wanted to run. I wanted to sit with Qwen, talk through an image or a scene, let it render something and then continue the conversation from the result.

Before the gate, every ComfyUI call could end that conversation with an OOM or a round of manual unloading. Now Qwen writes the prompt, leaves the GPU, ComfyUI renders, leaves the GPU, and Qwen comes back to inspect what it made.

That is the part I was missing. Generating the first image is an easy demo. An art or movie director is only useful if it is still there to discuss the next one.

No manual unload. No conversation interrupted by nvidia-smi.