Promptfoo Evaluation Workbench

Compare prompt variants, assert expected behavior and inspect actual evaluation results with Promptfoo 0.124.1. Start with a t3.medium (4 GiB RAM), 24 GiB root volume and AWS Systems Manager access. No inbound ports are required.

First evaluation

Connect with Session Manager, then run:

sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

The demo sends real CLI evaluation requests to a bounded localhost fixture. It uses no model or provider account. Two prompt variants over two cases produce four successful assertions. A second suite deliberately fails all four with Promptfoo's original exit 100. The command checks both result files and the SQLite database, and writes results beneath ~/promptfoo-results/evaluation-*. On every later invocation, both earlier evaluation IDs and the unchanged JSON results must still be present before any new evaluation starts. A failed assertion is a meaningful evaluation result; unexpected installer or provider errors fail the command separately. No model weights are downloaded.

Evaluate your model

Use promptfoo eval --config /path/to/your/promptfooconfig.yaml. Set provider credentials after launch in a file readable only by ubuntu; never pass keys as command arguments. Your provider may charge for requests. For an instance role that permits Amazon Nova Lite in us-east-1, the included real-provider example is:

/opt/sushi/bin/promptfoo-run-suite --bedrock

That command makes eight short real model requests and checks their actual responses. The service role must allow bedrock:InvokeModel for amazon.nova-lite-v1:0; model availability is a customer prerequisite.

The CLI is installed in /usr/local/bin/promptfoo; Node and application files are read-only under /opt/promptfoo. Evaluation state persists in ~/.local/share/promptfoo/promptfoo.db. Back up that directory together with ~/promptfoo-results and your evaluation configuration. Stop CLI processes before copying the database. First boot preserves existing state on restart.

No viewer starts automatically. The optional upstream viewer has no built-in authentication and listens on network interfaces; it has no --host flag. Keep inbound application ports closed. To review the ubuntu user's saved evaluations, start this bounded viewer from your administrative Session Manager shell:

sudo systemd-run --unit=promptfoo-viewer --uid=ubuntu \
  --setenv=HOME=/home/ubuntu \
  --property=RuntimeMaxSec=3600 \
  --property=IPAddressDeny=any \
  --property=IPAddressAllow=localhost \
  /usr/local/bin/promptfoo view --no --port 15500

The systemd network restriction permits only communication within the instance. The unit stops after one hour. An instance administrator can stop it sooner with sudo systemctl stop promptfoo-viewer. Other local users may access its results; use an instance dedicated to trusted users.

From your workstation with AWS CLI credentials and the Session Manager plugin, forward the same port:

aws ssm start-session --region us-east-1 --target YOUR_INSTANCE_ID \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["15500"],"localPortNumber":["15500"]}'

Replace YOUR_INSTANCE_ID with your launched instance ID, then open http://127.0.0.1:15500 on your workstation. Your instance needs an SSM-enabled role and outbound access to Systems Manager, and your administrator needs permission to start port-forwarding sessions. No inbound rule or key pair is required. The software includes no viewer password; the authenticated tunnel controls remote access. Never expose the viewer directly to the Internet.

Licenses and notices

Package notices remain in the installed packages; root and Node notices plus the pinned application source are under /usr/share/doc/promptfoo. Selected Node/libsql/sharp-libvips source archives are under sources/, source/patch candidates under sources/native/, and individual dependency notices under dependency-notices/. Full GPL/LGPL terms and native package notices are under native-notices/; native libraries are not covered by the application MIT notice alone. These files also include conditional source candidates. No enterprise features or provider subscription are included. Upstream documentation: https://www.promptfoo.dev/docs/intro/.