Minimal TypeScript library for running LLM-as-a-judge evaluations

Agent Looper — repo-agnostic fix-until-green harness (Cursor / Cline / OpenCode / Pi / Codex; configurable reviewRuntime judge)

judge is iPhoneX

Vendo's policy core at one choke point: guard.bind, approvals, grants, audit, Vendo Auto judge, breakers.

Test/optimize loop for agent skills — run spec'd scenarios on pi, LLM-judge, score, review, re-run

A query statement similar to mongodb, judge and retrieve data.

Hey LLM, you okay? — a unified, pyramid-ordered LLM testing CLI for CI/CD. YAML-defined layers (static → exec → http → llm → judge), LLM-as-a-judge gates, and an automated failure-triage protocol (A/B probe) that tells you whether a red test is your promp

Parallel prompt/model testing surface: versioned prompt candidates, experiment matrices, judge + objective grading. Isolated from the core product so it can be extracted.

Atbash safety judge plugin for AutoGen-style multi-agent orchestration

Automated agent benchmarking, trajectory scoring, and LLM-as-a-Judge evaluation framework for nestjs-agentic.

Evaluation toolkit for HazelJS AI apps — golden datasets, RAG metrics, agent trajectories, LLM-as-judge, CI reports

OpenCode plugin that detects prompt injection in tool call outputs using an LLM judge

Evidence-driven /until-done goal loops for Pi with TDD planning, mise verification, and mandatory LLM judge gating.

LLM-judged CLI test runner — define test cases in YAML, execute commands, let an LLM judge PASS/FAIL. Outputs JSONL trace.

looprun eval harness: run a generated subject's cases against a model target (governed or ungoverned variant), dump per-case traces for the LLM judge, fold verdicts, and certify against the bar.

Evaluation framework for Reactive Agents — LLM-as-judge scoring, regression detection, dataset loading

Evaluate LLM and rule-based judges with mutation cases whose generation does not depend on the judge under test.

Stronger answers for hard Pi questions via a parallel model panel + judge, built on pi-subagents

CLI for publishing and validating contract tests using judge-d API

LLM-as-a-Judge abstraction layer using ai-sdk and plugins