Multimodal and Builders

Speech, vision, multimodal tools, low-code builders, and stack selection guidance.

Speech, Vision & Multimodal

Whisper / faster-whisper / whisper.cpp

MIT · 🟢 stable

Speech-to-text: the original model, the CTranslate2 port (substantially faster), and the C++ port (runs anywhere).

• Replaces: Google Speech-to-Text, AssemblyAI
• Edge: State-of-the-art multilingual ASR for free, on your own hardware. whisper.cpp runs real-time transcription on a laptop CPU.

WhisperX

Python · BSD-2-Clause · 🟢 stable

Whisper plus word-level timestamps and speaker diarization.

• Edge: If you need to know who said what, when — subtitles, meeting notes — this is the one.

Kokoro / Piper

Apache-2.0 / GPL-3.0 · 🟢 stable

Text-to-speech. Kokoro is a tiny (~82M parameter) model with quality far above its weight class; Piper is optimized for devices as small as a Raspberry Pi.

• Replaces: ElevenLabs
• Edge: Real-time TTS on CPU. Kokoro’s small footprint makes it viable to bundle inside an app.

Pipecat

Python · Library · 🟢 stable

Framework for real-time voice and multimodal conversational agents.

• Replaces: Vapi, Retell
• Edge: pluggable STT/TTS/LLM stages over WebRTC, plus speech-to-speech model support, so you can assemble a voice agent from open parts instead of renting a platform. Known weakness: maintainers’ own issue tracker documents pipeline freezes, zombie function-call handlers after timeout, and multi-second latency in production. The linear pipeline model also fits multi-party conversation badly. Expect real engineering effort.

LiveKit Agents

Python + Node · Apache-2.0 · Framework · 🟢 stable

Realtime agent framework built on LiveKit’s WebRTC infrastructure.

• Replaces: Vapi, Retell
• Edge: the room/participant model handles multi-party and interruption natively, where a linear pipeline has to fake it. If your voice agent needs more than one human in the call, start here rather than with a pipeline framework.

ComfyUI

Python · GPL-3.0 · 🟢 stable

Node-based interface for diffusion models — image, video, and audio generation pipelines.

• Replaces: Midjourney, DALL·E
• Edge: The graph is the program — every step is inspectable and reproducible, and workflows are shareable as JSON. Supports essentially every open image/video model within days of release.

Surya

Python · GPL-3.0 (commercial exceptions) · 🟡 active

Document OCR, layout analysis, and reading-order detection in 90+ languages.

• Edge: Layout, reading order, and table structure — not just raw character recognition. Essential upstream of any document RAG.

---

Low-Code / Visual Builders

n8n

TypeScript · Sustainable Use License (fair-code, source-available) · 🟢 stable

Workflow automation with native AI agent nodes — hundreds of integrations, self-hostable.

• Replaces: Zapier, Make
• Edge: Drop to JavaScript in any node when the visual builder runs out. AI agent nodes make it a legitimate agent runtime, not just a trigger-action tool. Note: fair-code, not OSI-approved — read the license before commercial use.

Flowise

TypeScript · Apache-2.0 (with conditions) · 🟢 stable

Drag-and-drop builder for LLM flows and agents.

• Edge: Fastest way to prototype a RAG chatbot visually and expose it as an API or embeddable widget.

Langflow

Python · MIT · 🟢 stable

Visual framework for building multi-agent and RAG applications.

• Edge: Every visual component maps to real Python you can export and own. A good bridge between prototype and production code.

---

Open-Source Alternatives Cheat Sheet

---

Choosing Your Stack

Start small. Every layer below is optional until it isn’t.

Solo developer, local-first, zero API cost

Ollama → Continue (editor) + aider (terminal) → Open WebUI (chat)

Small team shipping an AI product

LiteLLM proxy → LangGraph or CrewAI → pgvector → Langfuse → promptfoo in CI

Enterprise, self-hosted, compliance-bound

vLLM (own GPUs) → LiteLLM (keys/budgets) → Qdrant → Dify or LangGraph
  → Langfuse (tracing) → Garak + NeMo Guardrails (safety)

Document-heavy RAG

Docling or RAGFlow (parsing) → LlamaIndex (retrieval) → Qdrant → Ragas (eval)

Three rules that save the most time:

1. Put a gateway in front of your models from day one. LiteLLM costs an afternoon and buys you provider switching, budgets, and fallbacks forever.
2. Use Postgres + pgvector until you have measured a reason not to. Most “we need a vector database” problems are actually retrieval-quality problems.
3. Add tracing before you add features. Debugging an untraced multi-agent system is guesswork.

---