Openchainbench

Tool annotations

No tools have read-only/destructive annotations

Add readOnlyHint or destructiveHint annotations to every tool so hosts can warn users before invoking.

All four hints declared on every tool

3/3 tools missing one or more hints — list_benchmarks (missing: readOnlyHint, destructiveHint, idempotentHint, openWorldHint); get_benchmark (missing: readOnlyHint, destructiveHint, idempotentHint, openWorldHint); query_prom (missing: readOnlyHint, destructiveHint, idempotentHint, openWorldHint). OpenAI's directory rejects tools where any of the four hints are missing or non-boolean.

For every tool, set all four hints (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) to explicit true/false values that match the handler’s actual behaviour.

Tool inputs are validated

2/3 tool handlers declare input schemas (67%)

Declare an inputSchema with zod/joi/yup on every tool definition.

Tool handlers catch errors

Only 1/3 tool handlers wrap calls in try/catch (33%)

Wrap each tool handler body in try/catch and return a structured error response.

Tool test coverage

Only 0/3 tools referenced in tests (0%)

Write tests that reference each tool by name so every tool has at least one test.

Shell command execution

7 child_process calls — runs shell commands

Prefer library functions over shell-outs. If you must shell out, ensure all inputs are properly escaped.

Secrets stay with their owner

1 secret/sensitive value flow into network calls (RENDER_API_TOKEN → dynamic)

Audit where credentials are sent. A NOTION_TOKEN should only reach api.notion.com — never a third-party host.