TypeScript SDK for OpenEval

LLM-as-a-Judge abstraction layer using ai-sdk and plugins

Diff your AI agent's behavior between two runs. Swap a model, change a prompt, run whatbroke, see exactly what changed.

AI Security Scanner - Test your AI systems for prompt injection and extraction vulnerabilities

Collector and post-collect runtime package for Agent Skill Harbor.

A terminal-based tool for local runs and debugging of eva-run

Deterministic PHI-leak assertions for LLM output, runnable in your normal test suite. HIPAA Safe Harbor + NHS/UK. No LLM calls, no data leaves your machine.

A set of CLI tools to help you iterate on your LLM prompts.

LLM eval & testing toolkit

Catch prompt regressions from model drift — on a schedule, not just on PRs. Runs a small eval set against your LLM on a cron, compares pass-rate to a stored baseline, and alerts (GitHub issue + non-zero exit) when it regresses.

Terrain test intelligence CLI.

MCP server that simulates malicious behaviors for security testing

Completion verification gate for AI agents & automation: before anything reports "done", re-fetch the real world state with a probe and block the completion claim on empty/mismatch/error. Framework-agnostic, zero-dependency, no LLM.

Snapshot-style regression testing for LLM prompts & agents — catch silent quality drift when you tweak a prompt. China-model & Chinese-first (DeepSeek / Kimi / Qwen / Doubao / Ollama).

Pipeline-kit eval foundation — defineEval, runEval, case/scorer/score types

Local development platform for chat agents — workspaces, multi-provider LLMs, feedback corpus.

Security evals for the AI era. Probes · Targets · Graders · Proof. Confirmed XSS / SQLi / BOLA / prompt-injection / MCP-RCE with reproducible proof attached to every finding.

Reserved name — future eval harness (golden paths, injection, traffic replay) for the Maestro agent runtime. See https://github.com/costasoftware/maestro.

Trident CLI — sign in, wire your repo into Trident with your coding agent, pentest what you deploy, and read findings from the terminal.

Grep AI coding-session transcripts (Claude Code, Codex) with bounded message context — built for agents. Includes rarity-ranked multi-word search, session overviews, sampled spines, and a built-in self-test.