OpenJudge
Open-source Python evaluation framework with ready-made and custom graders for scoring AI agent quality and producing reward signals.
Harness by agentscope-ai
Chalk Labs
Need help deploying OpenJudge?
Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.
Install
From the package registry. Read the repository before you give any tool access to your data.
pip install py-openjudge
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: llm-evaluation
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Fact sheet
- Languages
- PythonREADME
- Packages
- py-openjudge (pypi)README
- Maturity
- Betagithub.com
Facts extracted by Claude Haiku 5.5 from the linked sources on Oct 24, 2026. Wrong fact? Tell us.
Alternatives
Frameworks for the same language with a similar way of orchestrating agents.
- 0.853score
LlamaIndex
LlamaIndex
Open-source Python toolkit for RAG and agents over your own data, with event-driven workflows and 300+ integration packages.
HarnessAFreshness: A★ 52.6K↓ 1.4M/wk - 0.753score
LangChain
LangChain
Python framework of interoperable components and integrations for building agents and LLM apps, with LangGraph and Deep Agents layered above it.
HarnessAFreshness: A★ 148.4K↓ 37.3M/wk - 0.748score
LangGraph
LangChain
Low-level Python orchestration framework for long-running stateful agents, with durable execution, human review points and short- and long-term memory.
HarnessAFreshness: A★ 43.8K↓ 13.1M/wk - 0.732score
CrewAI
CrewAI
Open-source Python framework for multi-agent workflows, with role-based Crews and event-driven Flows.
HarnessAFreshness: A★ 60.1K↓ 644.6K/wk - 0.627score
AutoGen
Microsoft
Archived Microsoft Python framework for multi-agent apps; Microsoft directs new users to Microsoft Agent Framework.
HarnessAFreshness: A★ 61.5K↓ 89K/wk
Signals
- Stars
- 888
- Forks
- 73
- Open issues
- 13
- Weekly downloads
- 231
- Last commit
- Oct 20, 2026 (10 days ago)
- Last release
- Feb 12, 2026 (8 months ago)
- Last package publish
- May 15, 2026 (5 months ago)
From GitHub (fetched Oct 24, 2026) and the package registry.
Details
- Packages
- pypi
py-openjudge - License
- Apache-2.0
- Repository
- github.com/agentscope-ai/OpenJudge
- Website
- openjudge.me/
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- READMEgithub-graphql
More from agentscope-ai
Done for you
Rather not set it up yourself?
Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.
See how we build AI agents →Read the case studies
Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.
- launches shipped
- 50+
- launches shipped
- media placements
- 500+
- media placements
- founders on every account
- 2
- founders on every account
Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block
Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.
Get OpenJudge running in your business
Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.
Cite this page
OpenJudge. Agent Mart. https://agentmart.dev/harnesses/openjudge (last updated 2026-10-24)