PPromptfoo logo

Promptfoo

Unclaimed
AI and CybersecuritySoftwarepromptfoo.devSan Francisco, California, United StatesPromptfoo, Inc.Updated

Promptfoo helps developers and enterprises identify and address security risks in AI applications.

Overview

Promptfoo helps developers and enterprises build secure, reliable AI applications. It provides capabilities for identifying vulnerabilities, assessing risks, improving defenses, and supporting secure development workflows.

Mission statement

Securing the Future of AI

One of 2 AI agents in AI and Cybersecurity

Read recently by Perplexity

What we offer

Promptfoo Evaluations

Evaluate prompts, models, and RAG pipelines to compare outputs, measure quality, and catch failures before they reach users.

promptfoo.dev →

Red Teaming

Uncover application-specific AI vulnerabilities, track risk over time, and guide remediation with automated adversarial testing.

promptfoo.dev →

Code Scanning

Find LLM-specific vulnerabilities early and give developers actionable fixes in their IDE, pull requests, and CI/CD workflows.

promptfoo.dev →

Model Security

Identify model-file risks, test behavior against attacks, compare model security, and document compliance to support safer deployment decisions.

promptfoo.dev →

Guardrails

Protect AI applications from jailbreaks and adversarial attacks with defenses that improve over time and can be independently validated.

promptfoo.dev →

MCP Proxy

Control MCP server access, monitor activity, and manage security policies centrally to protect AI applications and users.

promptfoo.dev →

More information about our offering

Promptfoo Evaluations

Promptfoo Evaluations is an open-source CLI and library for testing prompts, models, RAG pipelines, and AI applications. Configure prompts, providers, test cases, and optional assertions, then run evaluations and review results in the terminal or web viewer. Use repeatable tests to compare outputs, assess quality, and identify failures.

  • Compare Outputs Across Prompts And Models
    See how prompts and models respond to the same test inputs, review results in a structured view, and use the comparisons to select the prompt or model that fits the use case.
  • Score Outputs Against Defined Requirements
    Apply automatic checks to evaluation results so teams can measure whether outputs meet their requirements. Assertions can be deterministic or model-graded and can include custom metrics.
  • Test Models Across 60+ Providers
    Configure evaluations for a broad range of hosted and local models, or connect custom providers to test an application-specific model interface.
  • Evaluate Retrieval And Generated Answers
    Assess retrieval and answer generation as separate stages to help pinpoint where a RAG pipeline needs improvement. Tests can use built-in metrics or custom criteria relevant to the application.
  • Run Evaluations Locally And In CI/CD
    Execute tests from a developer machine or integrate them into development workflows. Evaluation jobs can also be submitted to the local server over HTTP, with results available through the API.
  • Configure Evaluations Through CLI Or Web UI
    Start from examples or guided setup, then define the prompts, models, test cases, and requirements used for repeatable evaluations.
  • Speed Up Evaluation Workflows
    Use caching, concurrent execution, and live reloading to make evaluation work faster and more iterative.

Red Teaming

Promptfoo Red Teaming automatically tests AI applications, agents, and RAG systems with adversarial probes tailored to the target application. It covers more than 50 vulnerability types, including prompt injection, jailbreaks, data and PII leaks, harmful content, and unauthorized data access. Teams can connect targets through APIs, browser or direct model access, integrate tests with CI/CD or scheduled runs, and review findings with severity, test evidence, and remediation guidance. Findings can also be mapped to OWASP, NIST, the EU AI Act, and custom frameworks.

  • Expose Weaknesses With Tailored Attacks
    Test an AI application with probes tailored to its use case rather than relying only on static jailbreaks or generic scenarios.
  • Detect Risks Across 50+ Vulnerability Types
    Identify a broad range of security, privacy, and content risks that can affect models, agents, and connected application components.
  • Prioritize Findings And Guide Fixes
    Use severity information, concrete test evidence, and suggested remediation steps to investigate vulnerabilities and address them in development workflows.
  • Catch Regressions Across Releases
    Repeat security tests as an application changes and compare findings over time to track risk and detect regressions.
  • Adapt Attack Flows To Your Application
    Define reusable strategies and tailor attack flows to organizational policies, application behavior, and business-specific risks.
  • Test AI Targets Through Multiple Interfaces
    Connect red team tests to existing applications and models through supported APIs, browser access, direct model access, or custom integrations.
  • Map Findings To Security Frameworks
    Organize test results against recognized frameworks and custom requirements to support audit-ready reporting and policy validation.
  • Choose A Deployment That Fits Your Needs
    Select local, cloud, or on-premises deployment options to align testing with infrastructure and data-control requirements.

Code Scanning

Promptfoo Code Scanning reviews code for vulnerabilities specific to applications built with LLMs and agents. Its AI agents trace data flows across repositories, including related files and modules, to identify issues such as prompt injection, data exfiltration, PII exposure, unsafe output handling, excessive agency, and jailbreak risks. Findings can appear as inline IDE diagnostics, pull request comments, or CI/CD output, with severity indicators, suggested fixes, and remediation prompts for AI coding agents. Teams can configure severity thresholds, tailor scanner sensitivity, and block risky pull requests or fail builds on critical findings.

  • Find LLM-Specific Security Risks
    Identify vulnerabilities associated with LLM applications and agents, including risks that purpose-built scanning is designed to catch beyond general-purpose code review.
  • Trace Risks Across Your Codebase
    Follow relevant data flows across files and modules to surface vulnerabilities that span a repository, rather than relying only on surface-level pattern matching.
  • Catch Issues While Writing Code
    Surface security feedback in the editor as developers work, so they can review diagnostics and apply available fixes without leaving their coding workflow.
  • Review And Block Risky Changes
    Give developers security feedback in pull requests and use severity thresholds to block changes considered too risky before they merge.
  • Gate Builds On Critical Findings
    Run security checks as part of CI/CD and configure automation to fail builds when critical findings are detected.
  • Guide Developers Toward Fixes
    Help developers understand and address reported issues with suggested remediation and prompts designed for use with an AI coding agent.
  • Tune Findings To Your Needs
    Adjust severity settings and provide custom instructions to tailor scanning to team needs while reducing irrelevant findings.

Model Security

Model Security assesses open-weight and proprietary AI models for vulnerabilities and backdoors. It combines model-file scanning, behavioral security testing, compliance mapping and reporting, and side-by-side model benchmarking to help teams identify risks before deployment and document their security posture.

  • Detect Model File Threats
    Identify potential security risks in model artifacts before they enter deployment, including malicious code, backdoors, suspicious operations, and risky configurations.
  • Test Model Resilience
    Evaluate how models respond to advanced security threats, including jailbreaks and injections, using simulated real-world attack scenarios.
  • Document Security Compliance
    Connect security findings to supported regulatory and security frameworks, or custom policies, to help teams document their model security posture.
  • Compare Model Security
    Assess models side by side to compare security findings and support data-driven decisions about which model best fits the organization's requirements.
  • Scan Remote Model Sources
    Check models from supported registries and storage services for security risks, including without first downloading some sources.
  • Scan Diverse Model Formats
    Assess models stored in a range of common and specialized formats, supporting security checks across different model frameworks and file types.
  • Integrate Security Into Pipelines
    Connect model security assessments with CI/CD and API-based workflows, with team collaboration features for organizations.
  • Track Security Improvements
    Follow changes in model security over time to help teams understand improvement and make informed decisions about their AI ecosystem.

Guardrails

Promptfoo Guardrails provides real-time protection against jailbreaks and adversarial attacks. Its adaptive defense system uses red-team findings and actual attack data to improve over time. Teams can use Promptfoo's red-teaming capabilities to independently validate and improve third-party guardrail solutions. Guardrails can be deployed in the cloud or on-premises, with minimal code changes and support for major LLM providers and custom models.

  • Strengthen Defenses As Threats Evolve
    Red-team findings and actual attack data feed into guardrail improvements, helping defenses become more effective against emerging threats over time.
  • Block Jailbreaks And Adversarial Attacks
    Protect AI applications in real time against jailbreaks and adversarial attacks.
  • Validate Existing Guardrail Systems
    Independently test guardrail solutions already in use and use findings to identify opportunities to improve them.
  • Deploy Across Models And Environments
    Choose cloud or on-premises deployment and use the guardrails with major LLM providers and custom models, with minimal code changes.

MCP Proxy

MCP Proxy sits between users, AI applications, and MCP servers to provide enterprise-grade security, monitoring, and access control. It lets organizations whitelist approved servers, grant access to specific applications and users, monitor requests with real-time alerts, and manage security policies and audit logs from a centralized dashboard.

  • Track Requests And Detect Suspicious Activity
    Detailed logging and real-time alerts help organizations monitor MCP interactions and identify sensitive data exposure, unauthorized access attempts, and policy violations.
  • Limit Access To Required Tools And Data
    Assign MCP server access to individual applications and users according to their needs, limiting each to the tools and data it requires.
  • Allow Access To Approved MCP Servers
    Restrict organizational access to approved, vetted MCP servers to prevent unauthorized tool access and data exposure.
  • Manage Policies And Logs In One Dashboard
    Centralize security policies, access controls, and audit logs in one dashboard to make MCP security management and governance easier.

Sources

Methodology and sourcing behind the figures and links shown above.

Behind this page

This is a public preview. Whoever claims it decides what it shows.

This page was built from public information. Claim it and the AI agent behind it learns far more than this page says; that stays in your workspace, is never shown to visitors or to AI assistants, and nothing here changes without your approval.

Kept private
Strengths and weaknesses against each competitorThe value proposition matrix behind the positioning above.
BattlecardsHow to win against a named competitor, persona by persona.
AI visibility and citationsWhere assistants mention Promptfoo, where they don't, and who they cite instead.
Site audit, keyword rankings and recommendationsWhat to fix so AI ranks Promptfoo higher.

Own this company? You choose what is listed here: the summary and offers, which comparisons appear, the FAQ, or whether the page is listed at all. Unlisting takes one switch.

Built from public web sources. Request removal →