Add agent-convergence-scorer to LLM & MCP Testing - #118
Conversation
CLI and Python library that scores how lexically similar N agent or LLM outputs are: exact-match rate, Jaccard token overlap, divergence point, and a composite convergence score. Zero deps.
|
Review changes with |
Reviewer's guide (collapsed on small PRs)Reviewer's GuideThe README now lists agent-convergence-scorer under LLM & MCP Testing, highlighting its zero-dependency tooling for measuring lexical convergence, reproducibility, and fan-out collapse across agent or LLM runs. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper |
PR Summary by QodoAdd agent-convergence-scorer to LLM and MCP testing resources
AI Description
High-Level Assessment
Files changed (1)
|
|
Tick the box to add this pull request to the merge queue (same as
|
There was a problem hiding this comment.
LlamaPReview — No blocking issues found
A single well-formed README entry is added to the correct LLM & MCP Testing section, alphabetically placed before mcp-server-fuzzer, with no other content touched — safe to merge as documentation-only.
2 non-blocking findings retained — highest: Undisclosed maintainer-affiliated self-submission to awesome list.
Review details and evidence
| Priority | File | Finding | Evidence |
|---|---|---|---|
| P2 | README.md |
Undisclosed maintainer-affiliated self-submission to awesome list | needs verification |
| P2 | README.md |
External link and "zero deps" claim unverifiable from evidence | needs verification |
Finding details
P2 · Undisclosed maintainer-affiliated self-submission to awesome list
README.md
The added entry points at the hermes-labs-ai organization, and the same exact-head README.md already lists another hermes-labs-ai project (quick-gate-python under Tools), while the PR author account is roli-lpci. That combination is consistent with a maintainer-affiliated submission to an awesome list. No CONTRIBUTING file, awesome-list policy, or repository acceptance requirement is present in the supplied evidence, so no observed mechanism establishes that disclosure is required here; this stays nonblocking.
Owner action: Confirm affiliation disclosure if the project's contribution policy asks for it.
Verification boundary: needs verification; scope: changed region.
P2 · External link and "zero deps" claim unverifiable from evidence
README.md
Whether https://github.com/hermes-labs-ai/agent-convergence-scorer resolves, and whether the "zero deps" characterization is accurate, are facts about a repository outside the evidence set (coverage_type: non_repository for the PR body; no catalog item covers the external target). Awesome-list entries carry these claims as-is, so this does not gate merge; it is recorded only so a maintainer who spot-checks links knows the check was not performed.
Owner action: Spot-check the external link and the "zero deps" claim if desired; not required for merge.
Verification boundary: needs verification; scope: changed region.
LlamaPReview checks
- Read the complete PR-head file
README.md.
LlamaPReview is an open-source pull request reviewer. See exactly what it will and will not publish.
Adds agent-convergence-scorer to LLM & MCP Testing, alphabetically before mcp-server-fuzzer.
agent-convergence-scorer is a CLI and Python library that scores how lexically similar N agent or LLM outputs are — exact-match rate, Jaccard token overlap, divergence point, and a composite convergence score over any list of runs. An eval primitive for measuring reproducibility and fan-out collapse across parallel agent runs. Zero deps.
Summary by Sourcery
Add agent-convergence-scorer to the LLM and MCP testing resources.
New Features:
Documentation:
Affiliation:
Agent Convergence Scoreris maintained by Hermes Labs; Rolando Bosch is the founder of Hermes Labs.This contribution was autonomously selected and produced by agents through Hermes Labs' engineering infrastructure. Rolando Bosch is the responsible human contributor and has authorized autonomous publication from his personal GitHub account under Hermes Labs' established execution and escalation controls.