Promptfoo
Open-sourceTest prompts, agents, and RAGs with red teaming and vulnerability scanning across multiple LLM providers.
About Promptfoo
Promptfoo is a testing and evaluation framework for prompts, agents, and RAG systems. It supports red teaming, pentesting, and vulnerability scanning for AI applications with CI/CD integration. Used by OpenAI and Anthropic, with 20.5k GitHub stars.
Best For
- Teams ensuring AI application quality and safety
- Organizations implementing responsible AI testing practices
Pros & Cons
Pros
- + Used by major AI companies including OpenAI and Anthropic
- + Comprehensive testing goes beyond simple prompt comparison
- + CI/CD integration enables continuous AI quality assurance
Cons
- - Learning curve for test configuration and evaluation criteria
- - Comprehensive testing can be time-consuming for large prompt sets
Pricing
Open source and free to use
Key Features
- Comprehensive prompt testing and evaluation framework
- Red teaming and vulnerability scanning for AI safety
- Multi-LLM comparison across GPT, Claude, Gemini, and more
- CI/CD integration for automated testing in development pipelines
Similar Tools
Related AI Tools
Prompt Optimizer
An open-source AI prompt optimizer for writing better prompts and getting higher-quality AI outputs.
Repomix
Pack entire repositories into AI-friendly files for codebase analysis with Claude, ChatGPT, and other LLMs.
Gitleaks
Detect secrets and sensitive information in git repositories with CI/CD integration.
Lovcode
A desktop companion app for managing Claude Code chat history, configurations, commands, and skills.
AionUI
Free, local, open-source 24/7 coworking app that serves as a GUI for multiple AI coding tools.
Dyad
Local, open-source AI app builder for power users — an alternative to v0, Lovable, Replit, and Bolt.