--- title: Contributing to promptfoo sidebar_label: Contributing description: Contribute to promptfoo by submitting code, documentation, providers, and features following our development guidelines --- We welcome contributions from the community to help make promptfoo better. This guide will help you get started. If you have any questions, please reach out to us on [Discord](https://discord.gg/promptfoo) or through a [GitHub issue](https://github.com/promptfoo/promptfoo/issues/new). ## Project Overview Promptfoo is an MIT-licensed tool for testing and evaluating LLM apps. ### How to Contribute There are several ways to contribute to promptfoo: 1. **Submit Pull Requests**: Anyone can contribute by forking the repository and submitting pull requests. You don't need to be a collaborator to contribute code or documentation changes. 2. **Report Issues**: Help us by reporting bugs or suggesting improvements through GitHub issues or [Discord](https://discord.gg/promptfoo). 3. **Improve Documentation**: Documentation improvements are always welcome, including fixing typos, adding examples, or writing guides. We particularly welcome contributions in the following areas: - Bug fixes - Documentation updates, including examples and guides - Updates to providers including new models, new capabilities (tool use, function calling, JSON mode, file uploads, etc.) - Features that improve the user experience of promptfoo, especially relating to RAGs, Agents, and synthetic data generation. ## Getting Started 1. Fork the repository on GitHub by clicking the "Fork" button at the top right of the [promptfoo repository](https://github.com/promptfoo/promptfoo). 2. Clone your fork locally: ```bash git clone https://github.com/[your-username]/promptfoo.git cd promptfoo ``` 3. Set up your development environment: 3.1. Setup locally ```bash # Use the Node.js version specified in .nvmrc nvm use # If you're using npm, match the major used in CI so release-age policy is applied npm install -g npm@11 # Install dependencies (CI uses npm; pnpm and yarn also work for local development) npm install ``` 3.2 Set up using `devcontainer` (requires Docker and VSCode) Open the repository in VSCode and click on the "Reopen in Container" button. This will build a Docker container with all the necessary dependencies. Now install node based dependencies: ```bash npm install -g npm@11 npm install ``` 4. Run the tests to make sure everything is working: ```bash npm test ``` 5. Build the project: ```bash npm run build ``` 6. Run the project: ```bash npm run dev ``` This will run the express server on port 15500 and the web UI on port 3000. Both the API and UI will be automatically reloaded when you make changes. :::info The development experience is a little bit different than how it runs in production. In development, the web UI is served using a Vite server. In all other environments, the front end is built and served as a static site via the Express server. ::: If you're not sure where to start, check out our [good first issues](https://github.com/promptfoo/promptfoo/issues?q=state%3Aopen%20label%3Agood-first-issue%20is%3Aissue) or join our [Discord community](https://discord.gg/promptfoo) for guidance. ## Development Workflow 1. Create a new branch for your feature or bug fix: ```bash git checkout -b feature/your-feature-name ``` 2. Make your changes and commit them. We follow the [Conventional Commits](https://www.conventionalcommits.org/) specification for PR titles when merging into `main`. Individual commits can use any format, since we squash merge all PRs with a conventional commit message. :::note All pull requests are squash-merged with a conventional commit message. ::: 3. Push your branch to your fork: ```bash git push origin your-branch-name ``` 4. [Open a pull request](https://github.com/promptfoo/promptfoo/compare) (PR) against the `main` branch of the promptfoo repository. When opening a pull request: - Keep changes small and focused. Avoid mixing refactors with new features. - Ensure test coverage for new code or bug fixes. - Provide clear instructions on how to reproduce the problem or test the new feature. - Be responsive to feedback and be prepared to make changes if requested. - Ensure your tests are passing and your code is properly linted and formatted. You can do this by running `npm run lint -- --fix` and `npm run format` respectively. :::tip If you're unsure about how to implement something, feel free to open a draft PR to get early feedback. ::: Don't hesitate to ask for help. We're here to support you. If you're worried about whether your PR will be accepted, please talk to us first (see [Getting Help](#getting-help)). ## Tests ### Running Tests We use both Vitest and Jest. To run the test suite: ```bash npm test ``` To run tests in watch mode: ```bash npm run test:watch ``` You can also run specific tests with: ```bash # Vitest npx vitest [pattern] # Jest npx jest [pattern] # Example: # Runs all provider tests npx vitest providers ``` ### Writing Tests When writing tests, please: - **Use Vitest for new test files.** When modifying existing files, use whichever framework that file uses. - Ensure proper test isolation by: - Using `beforeEach` and `afterEach` to set up and clean up mocks - Calling `vi.clearAllMocks()` or `vi.restoreAllMocks()` for Vitest (or `jest.clearAllMocks()` for Jest) - Avoiding shared state between tests - Check the coverage report to ensure your changes are covered. - Avoid adding additional logs to the console. ## Linting and Formatting We use Biome for JavaScript/TypeScript linting and formatting, and Prettier for CSS/HTML/Markdown. Before submitting a pull request, please run: ```bash npm run format npm run lint ``` It's a good idea to run the lint command as `npm run lint -- --fix` to automatically fix some linting errors. ## Building the Project To build the project: ```bash npm run build ``` For continuous building of the API during development: ```bash npm run build:watch ``` ## Contributing to the CLI ### Running the CLI During Development We recommend using `npm link` to link your local `promptfoo` package to the global `promptfoo` command: ```bash npm link promptfoo eval --config examples/provider-cloudflare/ai/chat_config.yaml ``` Alternatively, you can use `npm run local` to run your local build directly: ```bash npm run local -- eval --config examples/provider-cloudflare/ai/chat_config.yaml ``` **Important:** Always use `--` before flags so they're passed to promptfoo, not npm. When working on a new feature, we recommend setting up a local `promptfooconfig.yaml` that tests your feature. Think of this as an end-to-end test for your feature. Here's a simple example: ```yaml title="promptfooconfig.yaml" # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json providers: - id: openai:gpt-5 prompts: - Translate "{{input}}" to {{language}} tests: - vars: input: 'Hello, world!' language: 'English' assert: - type: new-assertion-type ``` ## Adding a New Provider Providers are defined in TypeScript. We also provide language bindings for Python and Go. To contribute a new provider: 1. Ensure your provider doesn't already exist in promptfoo and fits its scope. For OpenAI-compatible providers, you may be able to re-use the openai provider and override the base URL and other settings. If your provider is OpenAI compatible, feel free to skip to step 4. 2. Implement the provider in `src/providers/yourProviderName.ts` following our [Custom API Provider Docs](/docs/providers/custom-api/). Please use our cache `src/cache.ts` to store responses. If your provider requires a new dependency, please add it as an optional dependency. 3. Write unit tests in `test/providers/yourProviderName.test.ts` and create an example in the `examples/` directory. 4. Document your provider in `site/docs/providers/yourProviderName.md`, including a description, setup instructions, configuration options, and usage examples. You can also add examples to the `examples/` directory. Consider writing a guide comparing your provider to others or highlighting unique features or benefits. 5. Update `src/providers/index.ts` and `site/docs/providers/index.md` to include your new provider. Update `src/envars.ts` to include any new environment variables your provider may need. 6. Ensure all tests pass (`npm test`) and fix any linting issues (`npm run lint`). ## Adding a New Assertion Assertions define different ways to compare and validate the output of an LLM against expected results. To contribute a new assertion: 1. **Define the Assertion Type**: - Add your new assertion type to the `BaseAssertionTypesSchema` enum in `src/types/index.ts`. - Run `npm run jsonSchema:generate` to update the JSON schema located at `site/static/config-schema.json` 2. **Implement the Assertion Handler**: - Create a new file in `src/assertions/` for your assertion logic. - Implement a handler function that takes `AssertionParams` and returns a `GradingResult`. Basic handler structure: ```typescript import type { AssertionParams, GradingResult } from '../types'; import invariant from '../util/invariant'; export function handleYourAssertion({ assertion, inverse, outputString, renderedValue, provider, // Use if your assertion needs provider-specific logic test, // Access to test case data }: AssertionParams): GradingResult { // Validate inputs invariant( typeof renderedValue === 'string' || Array.isArray(renderedValue), '"your-assertion" assertion must have a string or array value' ); // Implementation logic const threshold = assertion.threshold ?? 0.5; // Set a sensible default // Calculate the score const score = /* your scoring logic */; // Determine if test passes const pass = (score >= threshold) !== inverse; return { pass, score: inverse ? 1 - score : score, reason: pass ? 'Assertion passed' : `Your assertion scored ${score.toFixed(2)} vs threshold ${threshold}`, assertion, }; } ``` 3. **Register the Assertion Handler**: - In `src/assertions/index.ts`, import your handler function and add it to the handlers mapping. ```typescript import { handleYourAssertion } from './yourAssertion'; // In the handlers mapping 'your-assertion': handleYourAssertion, ``` 4. **Document Your Assertion**: - Update the appropriate documentation files: - For standard assertions, add details to `site/docs/configuration/expected-outputs/deterministic.md` - Include your assertion in the reference table in `site/docs/configuration/expected-outputs/index.md` - For model-graded assertions: - Add an entry to the list in `site/docs/configuration/expected-outputs/model-graded/index.md` - Create a dedicated documentation page at `site/docs/configuration/expected-outputs/model-graded/your-assertion.md` 5. **Write Tests**: - Create a test file in `test/assertions/yourAssertion.test.ts`. - Test scenarios including: - Standard use cases - Edge cases and error handling - Provider-specific behavior (if applicable) - Schema validation (if applicable) - Backward compatibility (if refactoring existing assertions) ## Contributing to the Web UI The web UI is written as a React app. It is exported as a static site and hosted by a local express server when bundled. To run the web UI in dev mode: ```bash npm run dev ``` This will host the web UI at http://localhost:3000. This allows you to hack on the React app quickly (with fast refresh). If you want to run the web UI without the express server, you can run: ```bash npm run dev:app ``` To test the entire thing end-to-end, build the project and run it: ```bash npm run build promptfoo view # Or: npm run dev ``` :::note This will not update the web UI if you make further changes to the code. You have to run `npm run build` again. ::: ## Python Contributions While promptfoo is primarily written in TypeScript, we support custom Python prompts, providers, asserts, and many examples in Python. We strive to keep our Python codebase simple and minimal, without external dependencies. Please adhere to these guidelines: - Use Python 3.9 or later - For linting and formatting, use `ruff`. Run `ruff check --fix` and `ruff format` before submitting changes - Follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html) - Use type hints to improve code readability and catch potential errors - Write unit tests for new Python functions using the built-in `unittest` module - When adding new Python dependencies to an example, update the relevant `requirements.txt` file ## Documentation If you're adding new features or changing existing ones, please update the relevant documentation. We use [Docusaurus](https://docusaurus.io/) for our documentation. We strongly encourage examples and guides as well. ### Documentation Standards Our documentation follows several standards to ensure accessibility: - **Human-readable**: Clean, well-structured markdown with clear navigation - **LLM-friendly**: Automated generation of [LLMs.txt files](https://llmstxt.org) for AI tool integration - **Searchable**: Proper headings, tags, and cross-references - **Example-driven**: Real-world examples and use cases ### Development Workflow To run the documentation in development mode: ```bash cd site npm start ``` This will start the Docusaurus development server on port 3100 by default (or a custom port if you set the `PORT` environment variable). You can then view the documentation at http://localhost:3100. To build the documentation for production: ```bash cd site npm run build ``` This will generate static content in the `build` directory that can be served using any static content hosting service. Building the documentation may occasionally catch errors that do not surface when running `npm start`. ## Advanced Topics ### Database Promptfoo uses SQLite as its default database, managed through the Drizzle ORM. By default, the database is stored in `~/.promptfoo/`. You can override this location by setting `PROMPTFOO_CONFIG_DIR`. The database schema is defined in `src/database/schema.ts` and migrations are stored in `drizzle/`. Note that the migrations are all generated and you should not access these files directly. #### Main Tables - `evals`: Stores eval details including results and configuration. - `prompts`: Stores information about different prompts. - `datasets`: Stores dataset information and test configurations. - `evalsToPrompts`: Manages the relationship between evals and prompts. - `evalsToDatasets`: Manages the relationship between evals and datasets. You can view the contents of each of these tables by running `npx drizzle-kit studio`, which will start a web server. #### Adding a Migration 1. **Modify Schema**: Make changes to your schema in `src/database/schema.ts`. 2. **Generate Migration**: Run the command to create a new migration: ```bash npm run db:generate ``` This command will create a new SQL file in the `drizzle/` directory. 3. **Review Migration**: Inspect the generated migration file to ensure it captures your intended changes. 4. **Apply Migration**: Apply the migration with: ```bash npm run db:migrate ``` ### Release Process This project uses [release-please](https://github.com/googleapis/release-please) for automated releases: 1. Merge PRs to main with conventional commit messages 2. release-please creates/updates a release PR with changelog entries, version bumps, and updated `package.json` 3. When merged, a GitHub release is created and the npm package is published For urgent releases, contact maintainers on [Discord](https://discord.gg/promptfoo). ## AI-Assisted Contributions We welcome AI-assisted development. Use whatever tools help you ship better work. We don't judge contributions by how they were produced. We judge them by the quality of the result. The same standards apply regardless of whether you wrote every character by hand or used Copilot, Cursor, Claude Code, or any other tool. :::tip You own the change If you open a pull request, you are responsible for what you submit. AI can help you write code faster, but it cannot take accountability for correctness, security, or maintainability. That stays with you. ::: ### Prove the change works A pull request should include clear instructions so that a reviewer can verify correctness by running the code locally. Don't make reviewers reverse-engineer how to test your change. **What to include:** - What changed and why (a few sentences is fine) - Steps to reproduce or verify the change (commands, config files, expected output) - Any tradeoffs or edge cases you considered If the change is hard to demonstrate, include something concrete: a minimal repro, a log excerpt, a screenshot, or a short recording. ### Be able to explain the diff You don't need to have typed every character. You do need to be able to answer reviewer questions about your changes: - Why is this implemented this way? - What alternatives did you consider? - What did you test? - What could break? If you can't explain a non-trivial part of your PR, revise it until you can. ### No hypothetical platform support Don't submit changes for environments you cannot actually run or test. If you want to add support for a platform you don't have access to, open an issue or discussion describing what you need and what you can verify today. ### Maintain agent guidance files This repository uses `AGENTS.md` files to provide context to AI coding assistants. If your changes affect how agents should work in a particular area (new patterns, conventions, or gotchas), update the relevant `AGENTS.md` file following existing patterns. See the root `AGENTS.md` for the directory structure and conventions. ### Disclosure Disclosure of AI usage is optional. We won't try to guess whether you used AI. That said, if a maintainer asks about your process, please answer candidly. If an agent produced a large portion of the change, a short note describing how you validated the output can help reviewers and speed up the process. ## Changelog **Do not manually edit `CHANGELOG.md`.** release-please generates entries from conventional commit messages. Use [Conventional Commits](https://www.conventionalcommits.org/) format for PR titles (they become squash-merge commit messages). ## Getting Help If you need help or have questions, you can: - Open an issue on GitHub. - Join our [Discord community](https://discord.gg/promptfoo). ## Code of Conduct We follow the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/). Please read and adhere to it in all interactions within our community.