Best open-source LLM eval command-line tools

Open-source tools for testing, evaluating and tracing LLM applications.

Filters · 2 Evals & observability, CLI

For

Personal2Enterprise3

Purpose

AI & LLM tooling90Agents & frameworks31Local inference8RAG & vector DBs6Coding assistants43Evals & observability3Agent skills10MCP servers14Developer tools100Web frameworks5Databases & backends4Editors & IDEs6API tools2Self-hosted apps15Notes & docs8Automation13Data & analytics7Infra & DevOps34Security & identity28Communication1Media & photos8Design & creative2

Runs on

Web1Docker / self-host7Desktop0Mobile0CLI3Library / SDK7Extension / plugin0

License

Permissive3Copyleft0Source-available0Other / custom0

More

Has live demo0Commit in last 90 days3

3 projects

SortWorth scoreTrendingMost starsRecently updatedNewest in catalog

p

promptfoopromptfoo/promptfoo

Test and red-team LLM appsEnterpriseEvals & observabilityCLILibrary / SDK★ 26kMITcommit 19 hours ago

85worth

☆ Save

g

garakNVIDIA/garak

LLM vulnerability scannerPersonalEnterpriseEvals & observabilitySecurity & identityCLI★ 9.4kApache-2.0commit yesterday

77worth

☆ Save

i

iFixAiifixai-ai/iFixAi

Independent auditing of AI agentsPersonalEnterpriseEvals & observabilityCLI★ 19kApache-2.0commit 23 hours ago

76worth

☆ Save

Narrow down

For

Personal · 2Enterprise · 3