Training, Evals, and Ops

Fine-tuning, evaluation, guardrails, observability, and LLMOps tooling.

Fine-Tuning & Training

Unsloth

Python · Apache-2.0 · 🟢 stable

Fine-tune LLMs roughly 2x faster with far less VRAM, without accuracy loss.

• Edge: Hand-written Triton kernels and a manual backprop engine. Makes fine-tuning a mid-size model on a single free Colab GPU realistic instead of aspirational.

Axolotl

Python · Apache-2.0 · 🟢 stable

Post-training framework configured entirely through YAML — full fine-tune, LoRA, QLoRA, DPO, ORPO, and more.

• Edge: One config file describes the entire run, which makes experiments reproducible and diffable in git.

LLaMA-Factory

Python · Apache-2.0 · 🟢 stable

Unified fine-tuning for 100+ models, with a web UI.

• Edge: Zero-code training via LlamaBoard. The widest model coverage of any tuning toolkit.

PEFT

Python · Apache-2.0 · 🟢 stable

Hugging Face’s parameter-efficient fine-tuning library — LoRA, QLoRA, adapters, prompt tuning.

• Edge: The reference implementation everything else builds on. Integrates directly with Transformers, Accelerate, and TRL.

Distilabel

Python · Apache-2.0 · 🟢 stable

Synthetic data pipelines for SFT and preference tuning, from the Argilla team.

• Edge: treats dataset generation as a reproducible pipeline rather than a pile of one-off scripts, and loops through Argilla so a human can curate what the model generated. The bottleneck in fine-tuning is almost always data, not compute.

TRL

Python · Apache-2.0 · 🟢 stable

Train transformer models with reinforcement learning — SFT, DPO, GRPO, reward modeling.

• Edge: The standard path from a base model to an aligned, instruction-following one.

---

Evals, Testing & Guardrails

promptfoo

TypeScript · MIT · 🟢 stable

Test and evaluate prompts, agents, and RAG systems — plus LLM red teaming and vulnerability scanning.

• Edge: Declarative test cases in YAML that run in CI. Side-by-side model comparison plus adversarial red-teaming in one tool. Local-first — your prompts never leave your machine.

ClawBench

Python · Apache-2.0 · Docker/browser harness · 🟡 active

Evaluate web agents on 153 everyday tasks across 144 live websites, with the final submission request intercepted to keep runs side-effect-free.

• Edge: Captures session replay, screenshots, HTTP traffic, browser actions, and agent messages in one reproducible run, making failures diagnosable beyond a final pass/fail score.

DeepEval

Python · Apache-2.0 · 🟢 stable

“Pytest for LLMs” — unit-test LLM outputs with research-backed metrics.

• Edge: Feels like a normal test suite. G-Eval, faithfulness, answer relevancy, hallucination, and RAG-specific metrics run locally on the model of your choice.

Ragas

Python · Apache-2.0 · 🟢 stable

Evaluation toolkit for RAG pipelines.

• Edge: Splits retrieval quality from generation quality, so you know which half to fix. Can synthesize a test set from your own documents.

Guardrails

Python · Apache-2.0 · 🟢 stable

Add input/output validators to LLM applications — structure, safety, PII, and custom rules.

• Edge: Validators are composable and re-ask the model on failure rather than just erroring out.

NeMo Guardrails

Python · Apache-2.0 · 🟢 stable

Programmable rails for conversational systems, defined in the Colang modeling language.

• Edge: Dialogue-level control — keep a bot on topic, block jailbreaks, enforce a conversation flow.

Garak

Python · Apache-2.0 · 🟢 stable

LLM vulnerability scanner — probes for prompt injection, jailbreaks, data leakage, and toxicity.

• Edge: nmap for language models. Run it before you ship, not after the incident.

---

Observability & LLMOps

Langfuse

TypeScript · MIT (core) · 🟢 stable

Open-source LLM engineering platform — tracing, evals, prompt management, and cost tracking.

• Replaces: LangSmith
• Edge: MIT-licensed core that you can genuinely self-host. Framework-agnostic via OpenTelemetry. Nested traces make multi-agent debugging tractable, and prompt versioning decouples prompt changes from deploys.

Phoenix

Python + TypeScript · Elastic-2.0 · 🟢 stable

AI observability and evaluation, built on OpenTelemetry and OpenInference.

• Edge: Runs in a notebook for local debugging or as a server for production. Strong embedding-drift and retrieval-quality visualizations.

OpenLLMetry

Python + TypeScript · Apache-2.0 · 🟢 stable

OpenTelemetry instrumentation for LLM applications.

• Edge: Standards-based — ship traces to Datadog, Honeycomb, Grafana, or whatever you already run. No new observability vendor.

Helicone

TypeScript · Apache-2.0 · 🟢 stable

Observability platform for LLM apps — one-line proxy integration, caching, and rate limiting.

• Edge: Change your base URL and you have logging. Lowest-friction start of any tool in this section.

---