docker/context-profiles/complex-corpus-v4.json
Rama main · réplica de lectura del repositorio original.
{
"schemaVersion": "ecc.context-eval-complex-corpus.v1",
"id": "complex-tasks@4",
"sampling": "Realistic multi-file engineering tasks, fixed before any provider call, with deterministic hidden graders scoring partial credit (ECC_EVAL_SCORE). Descriptive pilot: no population-representativeness claim. See complex-eval/DESIGN.md for the preregistered methodology.",
"minimumDistinctTasks": 4,
"nonInferiorityMargin": 0.05,
"selection": [
{
"id": "complex-chained-tickets",
"category": "complex-long-horizon",
"query": "Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies. Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it. We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it. Track redirect hits per link and expose them at GET /links/:code/stats, consistent with the existing API.",
"expectedIds": [
"skill:backend-patterns"
]
},
{
"id": "complex-idempotent-webhooks",
"category": "complex-concurrency-trap",
"query": "The payments provider retries aggressively and sometimes delivers the same webhook many times concurrently. Our receiver double-charges customers when that happens — see the incident note in SPEC.md. Rework the receiver so the full contract in SPEC.md holds: duplicates must never double-apply under any interleaving, and the documented API and the store contract stay intact. `npm test` must pass, and add regression coverage for the concurrent-duplicate case so INC-104 can't come back.",
"expectedIds": [
"skill:error-handling"
]
},
{
"id": "complex-production-ready",
"category": "complex-vague-hardening",
"query": "This goes to production Monday morning. Get it ready. The bar we hold production services to is in docs/production-bar.md — meet all of it, keep the documented API working, and leave the repo in a state you'd be comfortable being on-call for.",
"expectedIds": [
"skill:backend-patterns"
]
},
{
"id": "complex-recurring-incident",
"category": "complex-learning-loop",
"query": "Customers are reporting duplicate refunds again — see docs/incidents.md. When a refund request comes in without an idempotency key and the caller retries, we refund the customer twice. Fix it, and make sure this stops being a recurring incident — it's the third time this quarter. Finance just flagged that their payout batch job is about to start retrying on timeouts, and payout retries can double-pay vendors. Same family of problem as the refunds — handle it. One hard requirement: a retried payout must never pay a vendor twice, even if the service restarts between the attempts. You're rolling off this area. Write the handoff note for whoever picks this up next.",
"expectedIds": [
"skill:continuous-learning"
]
}
],
"tasks": [
{
"id": "chained-tickets",
"category": "long-horizon-chain",
"manualIds": [],
"files": {
"API.md": "# Shortlink API\n\n- `POST /links` — body `{ \"url\": string, \"ttlSeconds\"?: number }`.\n - `201` → `{ \"code\", \"shortUrl\", \"expiresAt\" }`. `code` is 6–10\n alphanumeric characters; `shortUrl` is `/<code>`; `expiresAt` is an ISO\n timestamp. Default TTL is 7 days; `ttlSeconds` must be an integer between\n 1 and 2592000 (30 days).\n - Missing/invalid `url` or out-of-range `ttlSeconds` → `400`.\n- `GET /<code>` — `302` with `Location` set to the original URL.\n Unknown code → `404`. Expired link → `410`.\n- `DELETE /links/<code>` — `204`. Unknown code → `404`.\n\nAll error responses follow the envelope in `CONTRIBUTING.md`.\n",
"CONTRIBUTING.md": "# Engineering conventions\n\nThese conventions apply to every ticket, every route, every change:\n\n- **Errors**: every error response is JSON with the envelope\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"<human readable>\" } }`\n and the matching HTTP status. No HTML error pages, no stack traces.\n- **Layering**: HTTP handling in `src/routes.js`, business logic in\n `src/service.js`, storage in `src/store.js`. `src/app.js` wires them.\n- **Runtime config** comes from environment variables, read at startup.\n- **Every ticket**: add tests under `test/`, add a `CHANGELOG.md` entry\n describing what shipped, and keep `README.md` accurate.\n- No external dependencies.\n",
"package.json": "{\n \"name\": \"shortlink\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# shortlink\n\nInternal link shortener service. Node.js standard library only, CommonJS.\n\n- `API.md` — the HTTP contract.\n- `CONTRIBUTING.md` — engineering conventions. Every ticket follows them.\n- `src/app.js` exports `createApp()` returning an `http.Server` that is not yet\n listening; `node src/index.js <port>` starts the service.\n- Run the tests with `npm test`.\n"
},
"steps": [
{
"query": "Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies.",
"check": "'use strict';\n// Step 1 grader: core API contract + conventions (envelope, layering, changelog, tests).\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 10; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 10, passed: ok, total: 10 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst sleep = ms => new Promise(resolve => setTimeout(resolve, ms));\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }\n if (typeof createApp === 'function') {\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const post = (body) => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n const get = (p) => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });\n\n const created = await post({ url: 'https://example.com/landing' });\n const createdBody = await created.json().catch(() => null);\n record('create-happy-201', created.status === 201 && createdBody\n && /^[A-Za-z0-9]{6,10}$/.test(createdBody.code || '') && typeof createdBody.shortUrl === 'string'\n && typeof createdBody.expiresAt === 'string' && !Number.isNaN(Date.parse(createdBody.expiresAt)));\n\n let code = createdBody && createdBody.code;\n if (code) {\n const redirect = await get(`/${code}`);\n record('redirect-302-location', redirect.status === 302\n && redirect.headers.get('location') === 'https://example.com/landing');\n } else record('redirect-302-location', false);\n\n const unknown = await get('/nope00');\n record('unknown-code-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));\n\n const badUrl = await post({ url: 'notaurl' });\n record('invalid-url-400-envelope', badUrl.status === 400 && hasEnvelope(await badUrl.json().catch(() => null)));\n const noBody = await post({});\n record('missing-url-400-envelope', noBody.status === 400 && hasEnvelope(await noBody.json().catch(() => null)));\n const badTtl = await post({ url: 'https://example.com', ttlSeconds: 99999999 });\n record('ttl-bounds-400-envelope', badTtl.status === 400 && hasEnvelope(await badTtl.json().catch(() => null)));\n\n const expiring = await post({ url: 'https://example.com/gone', ttlSeconds: 1 });\n const expiringBody = await expiring.json().catch(() => null);\n if (expiringBody && expiringBody.code) {\n await sleep(1300);\n const gone = await get(`/${expiringBody.code}`);\n record('expired-link-410-envelope', gone.status === 410 && hasEnvelope(await gone.json().catch(() => null)));\n } else record('expired-link-410-envelope', false);\n\n if (code) {\n const del = await fetch(`http://127.0.0.1:${port}/links/${code}`, { method: 'DELETE' });\n const after = await get(`/${code}`);\n record('delete-flow-204-then-404', del.status === 204 && after.status === 404);\n } else record('delete-flow-204-then-404', false);\n app.close();\n } catch { /* remaining checks unscored */ }\n } else {\n for (const name of ['create-happy-201', 'redirect-302-location', 'unknown-code-404-envelope',\n 'invalid-url-400-envelope', 'missing-url-400-envelope', 'ttl-bounds-400-envelope',\n 'expired-link-410-envelope', 'delete-flow-204-then-404']) record(name, false);\n }\n\n // Conventions.\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const testCount = (tests.match(/\\btest\\(/g) || []).length;\n record('changelog-and-tests', changelog.length > 20 && testCount >= 3);\n record('layering-files', ['routes.js', 'service.js', 'store.js']\n .every(f => fs.existsSync(path.join(root, 'src', f))));\n\n finish();\n})();\n",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000
},
{
"query": "Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it.",
"check": "'use strict';\n// Step 2 grader: persistence across a simulated restart (fresh module state,\n// same DATA_FILE), expiry state survives, fresh/corrupt-start tolerance, conventions.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 7; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 7, passed: ok, total: 7 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst sleep = ms => new Promise(resolve => setTimeout(resolve, ms));\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\nasync function start() {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise((resolve, reject) => { app.once('error', reject); app.listen(0, '127.0.0.1', resolve); });\n return app;\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n // First boot: create a durable link and a 1s-expiring link.\n let app = await start();\n let port = app.address().port;\n const post = body => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n const durable = await (await post({ url: 'https://example.com/durable' })).json().catch(() => null);\n const short = await (await post({ url: 'https://example.com/short', ttlSeconds: 1 })).json().catch(() => null);\n await new Promise(resolve => app.close(resolve));\n\n // Restart: fresh modules, same DATA_FILE.\n app = await start();\n port = app.address().port;\n const get = p => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });\n\n const after = durable && durable.code ? await get(`/${durable.code}`) : null;\n record('link-survives-restart', after && after.status === 302\n && after.headers.get('location') === 'https://example.com/durable');\n\n await sleep(1300);\n const expiredAfter = short && short.code ? await get(`/${short.code}`) : null;\n record('expiry-survives-restart', expiredAfter && expiredAfter.status === 410);\n await new Promise(resolve => app.close(resolve));\n\n // Data file is real JSON on disk.\n let dataOk = false;\n try { JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')); dataOk = true; } catch { /* missing/invalid */ }\n record('data-file-is-json', dataOk);\n\n // Fresh start with no data file present.\n fs.rmSync(DATA_FILE, { force: true });\n app = await start();\n port = app.address().port;\n const fresh = await fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ url: 'https://example.com/fresh' }) });\n record('fresh-start-without-data-file', fresh.status === 201);\n await new Promise(resolve => app.close(resolve));\n\n // Corrupt data file must not kill the service.\n fs.mkdirSync(path.dirname(DATA_FILE), { recursive: true });\n fs.writeFileSync(DATA_FILE, 'garbage{{{');\n app = await start();\n port = app.address().port;\n const afterCorrupt = await get('/anything1');\n record('corrupt-data-file-tolerated', afterCorrupt.status === 404\n && hasEnvelope(await afterCorrupt.json().catch(() => null)));\n await new Promise(resolve => app.close(resolve));\n fs.rmSync(DATA_FILE, { force: true });\n } catch { /* remaining checks unscored */ }\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 2 && /persist|restart|data/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 6);\n\n finish();\n})();\n",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000
},
{
"query": "We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it.",
"check": "'use strict';\n// Step 3 grader: abuse handling — URL validation, size limits, rate limiting —\n// plus conventions. Hammer probe runs last so earlier probes stay unthrottled.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links-step3.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const post = body => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n\n const okCreate = await post({ url: 'https://example.com/normal' });\n record('normal-create-still-201', okCreate.status === 201);\n\n const js = await post({ url: 'javascript:alert(1)' });\n record('javascript-scheme-400-envelope', js.status === 400 && hasEnvelope(await js.json().catch(() => null)));\n const ftp = await post({ url: 'ftp://files.example.com/x' });\n record('non-http-scheme-400-envelope', ftp.status === 400 && hasEnvelope(await ftp.json().catch(() => null)));\n const huge = await post({ url: `https://example.com/${'a'.repeat(10000)}` });\n const hugeBody = await huge.json().catch(() => null);\n record('oversize-url-4xx-envelope', huge.status >= 400 && huge.status < 500 && hasEnvelope(hugeBody));\n\n // Hammer: 60 rapid creates must trip a 429 with the envelope.\n const responses = await Promise.all(Array.from({ length: 60 }, (_, i) =>\n post({ url: `https://example.com/flood-${i}` })));\n const limited = [];\n for (const r of responses) if (r.status === 429) limited.push(await r.json().catch(() => null));\n record('rate-limit-429-envelope', limited.length > 0 && limited.every(hasEnvelope));\n app.close();\n } catch { /* remaining checks unscored */ }\n\n let sources = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'src'))) {\n if (f.endsWith('.js')) sources += fs.readFileSync(path.join(root, 'src', f), 'utf8');\n }\n } catch { /* missing */ }\n record('rate-limiting-implemented', /429|rate.?limit/i.test(sources));\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 3 && /abuse|rate|valid|secur/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 9);\n\n finish();\n})();\n",
"manualIds": [
"skill:security-review"
],
"checkTimeoutMs": 60000
},
{
"query": "Track redirect hits per link and expose them at GET /links/:code/stats, consistent with the existing API.",
"check": "'use strict';\n// Step 4 grader: hit analytics consistent with the existing API, conventions,\n// docs and tests. (Runs in a later process than step 3, so rate windows cleared.)\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links-step4.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n\n const created = await fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ url: 'https://example.com/tracked' }) });\n const body = await created.json().catch(() => null);\n const code = body && body.code;\n record('create-still-works', created.status === 201 && Boolean(code));\n\n if (code) {\n const before = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);\n const beforeBody = await before.json().catch(() => null);\n record('stats-zero-before-redirects', before.status === 200 && beforeBody && beforeBody.hits === 0);\n\n for (let i = 0; i < 3; i++) {\n await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });\n }\n const stats = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);\n const statsBody = await stats.json().catch(() => null);\n record('stats-count-three-hits', stats.status === 200 && statsBody && statsBody.hits === 3);\n\n const redirect = await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });\n record('redirect-still-302', redirect.status === 302);\n\n const missing = await fetch(`http://127.0.0.1:${port}/links/zzzzzz/stats`);\n record('stats-unknown-404-envelope', missing.status === 404\n && hasEnvelope(await missing.json().catch(() => null)));\n } else {\n for (const name of ['stats-zero-before-redirects', 'stats-count-three-hits',\n 'redirect-still-302', 'stats-unknown-404-envelope']) record(name, false);\n }\n app.close();\n } catch { /* remaining checks unscored */ }\n\n let readme = '';\n try { readme = fs.readFileSync(path.join(root, 'README.md'), 'utf8'); } catch { /* missing */ }\n record('readme-documents-stats', /\\/stats|hits|analytics/i.test(readme));\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 4 && /stat|analytic|hit/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 12);\n\n finish();\n})();\n",
"manualIds": [
"skill:api-design"
],
"checkTimeoutMs": 60000
}
]
},
{
"id": "idempotent-webhooks",
"category": "concurrency-trap",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000,
"query": "The payments provider retries aggressively and sometimes delivers the same webhook many times concurrently. Our receiver double-charges customers when that happens — see the incident note in SPEC.md. Rework the receiver so the full contract in SPEC.md holds: duplicates must never double-apply under any interleaving, and the documented API and the store contract stay intact. `npm test` must pass, and add regression coverage for the concurrent-duplicate case so INC-104 can't come back.",
"files": {
"package.json": "{\n \"name\": \"webhook-receiver\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# webhook-receiver\n\nReceives payment webhooks. There is an open incident: customers were\ndouble-charged when the provider retried deliveries. See `SPEC.md` for the\ncontract, including the exactly-once rules.\n\n- `src/app.js` exports `createApp()` returning an `http.Server` that is not\n yet listening; `node src/index.js <port>` starts the service.\n- `src/store.js` is shared infrastructure: it keeps its current exports\n (`store`) and records every applied payment in `store.paymentLog`.\n- No external dependencies. `npm test` runs the tests. `CHANGELOG.md` records\n every shipped change.\n",
"SPEC.md": "# Payment webhook contract\n\n`POST /webhooks/payments` with JSON body\n`{ \"eventId\": string, \"orderId\": string, \"amountCents\": number, \"type\": \"payment.succeeded\" }`.\n\nExactly-once is the point. The provider retries aggressively and may deliver\nthe same event many times, concurrently, or out of order.\n\n- A new, valid `eventId`: apply the payment exactly once → `200`\n `{ \"status\": \"processed\", \"orderId\" }`.\n- The same `eventId` seen again (any number of times, any interleaving):\n `200` `{ \"status\": \"duplicate\", \"orderId\" }` — never applied twice.\n- A payment event (new `eventId`) for an order that is already paid:\n `200` `{ \"status\": \"already_paid\", \"orderId\" }` — an order is paid at most\n once, ever.\n- `amountCents` not matching the order's amount: `422`, not applied.\n- Unknown `orderId`: `404`. Malformed body (bad JSON, missing/invalid\n fields): `400`.\n- Error responses use the envelope\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"...\" } }`.\n\n`GET /orders/:id` → `200` `{ \"id\", \"status\", \"paidAt\", \"paymentsApplied\" }`\nor a `404` envelope.\n\n## Incident note\n\nINC-104: concurrent duplicate deliveries double-applied payments. The naive\nreceiver checked \"have we seen this event?\" and applied the payment in two\nseparate steps with an async gap in between, so parallel duplicates both\npassed the check.\n",
"src/app.js": "'use strict';\nconst http = require('node:http');\nconst { store } = require('./store');\n\n// INC-104 receiver: checks \"seen this event?\" and applies the payment in two\n// steps with an async gap in between. Concurrent duplicates both pass the\n// check. Do not keep this shape.\nfunction createApp() {\n return http.createServer((req, res) => {\n const url = new URL(req.url, 'http://localhost');\n\n if (req.method === 'POST' && url.pathname === '/webhooks/payments') {\n let body = '';\n req.on('data', chunk => { body += chunk; });\n req.on('end', async () => {\n const parsed = JSON.parse(body);\n const { eventId, orderId } = parsed;\n if (store.processedEvents.has(eventId)) {\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ status: 'duplicate', orderId }));\n return;\n }\n await new Promise(resolve => setImmediate(resolve)); // async gap\n const order = store.orders.get(orderId);\n order.status = 'paid';\n order.paidAt = new Date().toISOString();\n order.paymentsApplied++;\n store.paymentLog.push({ eventId, orderId, amountCents: parsed.amountCents });\n store.processedEvents.add(eventId);\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ status: 'processed', orderId }));\n });\n return;\n }\n\n const match = /^\\/orders\\/([\\w-]+)$/.exec(url.pathname);\n if (req.method === 'GET' && match) {\n const order = store.orders.get(match[1]);\n if (!order) {\n res.writeHead(404, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'no such order' } }));\n return;\n }\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify(order));\n return;\n }\n\n res.writeHead(404, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'not found' } }));\n });\n}\n\nmodule.exports = { createApp };\n",
"src/index.js": "'use strict';\nconst { createApp } = require('./app');\n\nconst port = Number(process.argv[2] || 8080);\ncreateApp().listen(port, () => {\n console.log(`webhook-receiver listening on ${port}`);\n});\n",
"src/store.js": "'use strict';\n\n// Shared infrastructure. Every applied payment is appended to paymentLog;\n// orders and processedEvents track receiver state. Keep the `store` export.\nconst store = {\n orders: new Map([\n ['o1', { id: 'o1', amountCents: 5000, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o2', { id: 'o2', amountCents: 12500, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o3', { id: 'o3', amountCents: 800, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o4', { id: 'o4', amountCents: 9999, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o5', { id: 'o5', amountCents: 250, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o6', { id: 'o6', amountCents: 7300, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ]),\n paymentLog: [],\n processedEvents: new Set(),\n};\n\nmodule.exports = { store };\n",
"test/webhooks.test.js": "'use strict';\nconst test = require('node:test');\nconst assert = require('node:assert/strict');\nconst { createApp } = require('../src/app');\nconst { store } = require('../src/store');\n\ntest('a single payment event processes', async () => {\n const server = createApp();\n await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));\n try {\n const port = server.address().port;\n const res = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ eventId: 'ev-test-1', orderId: 'o1', amountCents: 5000, type: 'payment.succeeded' }) });\n assert.equal(res.status, 200);\n assert.equal((await res.json()).status, 'processed');\n assert.equal(store.orders.get('o1').status, 'paid');\n } finally {\n server.close();\n }\n});\n"
},
"check": "'use strict';\n// Hidden grader for idempotent-webhooks: exactly-once under sequential,\n// concurrent, and mixed-concurrent duplicates, plus the documented API,\n// regression coverage, and hygiene. Prints ECC_EVAL_SCORE and always exits 0.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 12; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 12, passed: ok, total: 12 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n let store;\n try {\n ({ createApp } = require(path.join(root, 'src', 'app.js')));\n ({ store } = require(path.join(root, 'src', 'store.js')));\n } catch { /* scored below */ }\n if (typeof createApp === 'function' && store && Array.isArray(store.paymentLog)) {\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const send = (eventId, orderId, amountCents) => fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ eventId, orderId, amountCents, type: 'payment.succeeded' }) });\n const logsFor = orderId => store.paymentLog.filter(p => p.orderId === orderId).length;\n\n // 1: single delivery applies once.\n const single = await send('ev-1', 'o1', 5000);\n const singleBody = await single.json().catch(() => null);\n record('single-delivery-processed', single.status === 200 && singleBody\n && singleBody.status === 'processed' && singleBody.orderId === 'o1' && logsFor('o1') === 1);\n\n // 2: sequential retry replays without re-applying.\n const retry = await send('ev-1', 'o1', 5000);\n const retryBody = await retry.json().catch(() => null);\n record('sequential-duplicate-inert', retry.status === 200 && retryBody\n && retryBody.status === 'duplicate' && logsFor('o1') === 1);\n\n // 3: fifty concurrent identical deliveries apply exactly once.\n const storm = await Promise.all(Array.from({ length: 50 }, () => send('ev-2', 'o2', 12500)));\n const stormBodies = [];\n for (const r of storm) stormBodies.push(await r.json().catch(() => null));\n const processedCount = stormBodies.filter(b => b && b.status === 'processed').length;\n const duplicateCount = stormBodies.filter(b => b && b.status === 'duplicate').length;\n record('concurrent-storm-exactly-once', storm.every(r => r.status === 200)\n && processedCount === 1 && duplicateCount === 49 && logsFor('o2') === 1\n && store.orders.get('o2').paymentsApplied === 1);\n\n // 4: a different event for an already-paid order is already_paid and inert.\n const second = await send('ev-3', 'o2', 12500);\n const secondBody = await second.json().catch(() => null);\n record('already-paid-order-inert', second.status === 200 && secondBody\n && secondBody.status === 'already_paid' && logsFor('o2') === 1);\n\n // 5-7: contract errors with envelopes.\n const unknown = await send('ev-4', 'nope', 100);\n record('unknown-order-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));\n const malformed = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: '{bad json' });\n record('malformed-body-400-envelope', malformed.status === 400 && hasEnvelope(await malformed.json().catch(() => null)));\n const mismatch = await send('ev-5', 'o3', 999999);\n record('amount-mismatch-422-envelope', mismatch.status === 422\n && hasEnvelope(await mismatch.json().catch(() => null)) && logsFor('o3') === 0);\n\n // 8: mixed storm — three orders, three eventIds, ten duplicates each, all concurrent.\n const mixed = await Promise.all(['o4', 'o5', 'o6'].flatMap(orderId =>\n Array.from({ length: 10 }, () => send(`ev-${orderId}`, orderId, store.orders.get(orderId).amountCents))));\n for (const r of mixed) await r.json().catch(() => null);\n record('mixed-storm-each-order-once', ['o4', 'o5', 'o6'].every(orderId =>\n logsFor(orderId) === 1 && store.orders.get(orderId).paymentsApplied === 1));\n\n // 9: order inspection endpoint reflects reality.\n const orderView = await fetch(`http://127.0.0.1:${port}/orders/o2`);\n const orderBody = await orderView.json().catch(() => null);\n record('order-endpoint-accurate', orderView.status === 200 && orderBody\n && orderBody.status === 'paid' && orderBody.paymentsApplied === 1 && Boolean(orderBody.paidAt));\n\n app.close();\n } catch { /* remaining checks unscored */ }\n } else {\n for (const name of ['single-delivery-processed', 'sequential-duplicate-inert', 'concurrent-storm-exactly-once',\n 'already-paid-order-inert', 'unknown-order-404-envelope', 'malformed-body-400-envelope',\n 'amount-mismatch-422-envelope', 'mixed-storm-each-order-once', 'order-endpoint-accurate']) record(name, false);\n }\n\n // Conventions.\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n record('concurrency-regression-tests', (tests.match(/\\btest\\(/g) || []).length >= 4\n && /Promise\\.all|concurrent|duplicate|retry/i.test(tests));\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n record('changelog-entry', /idem|duplicat|retry|inc-104|race/i.test(changelog));\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);\n } catch { record('no-external-dependencies', false); }\n\n finish();\n})();\n"
},
{
"id": "production-ready",
"category": "vague-hardening",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000,
"query": "This goes to production Monday morning. Get it ready. The bar we hold production services to is in docs/production-bar.md — meet all of it, keep the documented API working, and leave the repo in a state you'd be comfortable being on-call for.",
"files": {
"docs/production-bar.md": "# The production bar\n\nEvery production service here meets all of the following, all the time:\n\n- **Validation**: malformed JSON, missing fields, and wrong types are rejected\n with `400` and a structured JSON error body\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"...\" } }`. Unknown\n resources are `404` in the same envelope. No stack traces, no HTML errors,\n no hanging connections.\n- **Body limits**: request bodies over 64 KB are rejected with `413`, same\n envelope.\n- **Health**: `GET /health` returns `200` with `{ \"status\": \"ok\" }`.\n- **Logging**: one structured JSON log line per request with at least\n `method`, `path`, and `status` fields.\n- **Configuration**: runtime configuration (port, limits) comes from\n environment variables, read at startup. Nothing secret is hardcoded.\n- **Shutdown**: the service closes cleanly on `SIGTERM` (stops accepting,\n drains, exits).\n- **Headers**: responses carry `X-Content-Type-Options: nosniff`.\n- **Tests**: the suite covers error paths, not just the happy path.\n- **Changelog**: every shipped change has a `CHANGELOG.md` entry.\n",
"package.json": "{\n \"name\": \"notes-service\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# notes-service\n\nTiny notes API. Hobby prototype state: it works on the happy path and that's\nabout all that can be said for it.\n\n## API\n\n- `POST /notes` — body `{ \"title\": string, \"body\": string }` → `201` with\n `{ \"id\", \"title\", \"body\" }`.\n- `GET /notes/:id` — `200` with the note, or `404`.\n- `GET /notes` — `200` with `{ \"notes\": [...] }`.\n\n`src/app.js` exports `createApp()` returning an `http.Server` that is not yet\nlistening; `node src/index.js` starts the service. `npm test` runs the tests.\n\n## Operations\n\n`docs/production-bar.md` lists what every production service here must meet.\n`CHANGELOG.md` records every shipped change.\n",
"src/app.js": "'use strict';\nconst http = require('node:http');\n\n// Prototype state: happy path only.\nconst notes = new Map();\nlet nextId = 1;\n\nfunction createApp() {\n return http.createServer((req, res) => {\n console.log('got a request');\n const url = new URL(req.url, 'http://localhost');\n\n if (req.method === 'POST' && url.pathname === '/notes') {\n let body = '';\n req.on('data', chunk => { body += chunk; });\n req.on('end', () => {\n const parsed = JSON.parse(body);\n const id = `n_${nextId++}`;\n notes.set(id, { id, title: parsed.title, body: parsed.body });\n res.writeHead(201, { 'content-type': 'application/json' });\n res.end(JSON.stringify(notes.get(id)));\n });\n return;\n }\n\n const match = /^\\/notes\\/([\\w-]+)$/.exec(url.pathname);\n if (req.method === 'GET' && match) {\n const note = notes.get(match[1]);\n if (!note) {\n res.writeHead(404);\n res.end('<html><body>not found</body></html>');\n return;\n }\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify(note));\n return;\n }\n\n if (req.method === 'GET' && url.pathname === '/notes') {\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ notes: [...notes.values()] }));\n return;\n }\n\n res.writeHead(404);\n res.end('<html><body>not found</body></html>');\n });\n}\n\nmodule.exports = { createApp };\n",
"src/index.js": "'use strict';\nconst { createApp } = require('./app');\n\ncreateApp().listen(8080, () => {\n console.log('notes listening on 8080');\n});\n",
"test/notes.test.js": "'use strict';\nconst test = require('node:test');\nconst assert = require('node:assert/strict');\nconst { createApp } = require('../src/app');\n\ntest('create and read a note', async () => {\n const server = createApp();\n await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));\n try {\n const port = server.address().port;\n const created = await fetch(`http://127.0.0.1:${port}/notes`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ title: 'first', body: 'hello' }) });\n assert.equal(created.status, 201);\n const { id } = await created.json();\n const read = await fetch(`http://127.0.0.1:${port}/notes/${id}`);\n assert.equal((await read.json()).title, 'first');\n } finally {\n server.close();\n }\n});\n"
},
"check": "'use strict';\n// Hidden grader for production-ready: probes every dimension of the documented\n// production bar. Prints ECC_EVAL_SCORE and always exits 0.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 16; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 16, passed: ok, total: 16 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }\n if (typeof createApp === 'function') {\n // Capture console output during the probe run to inspect request logging.\n const logged = [];\n const originalLog = console.log;\n const originalError = console.error;\n const originalStdoutWrite = process.stdout.write.bind(process.stdout);\n const originalStderrWrite = process.stderr.write.bind(process.stderr);\n console.log = (...args) => { logged.push(args.join(' ')); };\n console.error = (...args) => { logged.push(args.join(' ')); };\n // Agents may log through an injectable writer straight to the streams\n // instead of console.*. Capture-then-pass-through: the bytes always reach\n // the stream untouched, so the grader's own ECC_EVAL_SCORE line (emitted\n // via process.stdout.write) can never be swallowed or corrupted.\n const tap = write => (chunk, encoding, callback) => {\n try { logged.push(Buffer.isBuffer(chunk) ? chunk.toString('utf8') : String(chunk)); } catch { /* capture must never break a write */ }\n return write(chunk, encoding, callback);\n };\n process.stdout.write = tap(originalStdoutWrite);\n process.stderr.write = tap(originalStderrWrite);\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const api = (p, options) => fetch(`http://127.0.0.1:${port}${p}`, options);\n const post = body => api('/notes', { method: 'POST', headers: { 'content-type': 'application/json' }, body });\n\n // Documented API still works.\n const created = await post(JSON.stringify({ title: 'deploy', body: 'checklist' }));\n const createdBody = await created.json().catch(() => null);\n record('api-roundtrip-preserved', created.status === 201 && createdBody && createdBody.id\n && (await (await api(`/notes/${createdBody.id}`)).json().catch(() => ({}))).title === 'deploy'\n && Array.isArray((await (await api('/notes')).json().catch(() => ({}))).notes));\n\n // Validation and envelope discipline.\n const badJson = await post('{not json');\n record('malformed-json-400-envelope', badJson.status === 400 && hasEnvelope(await badJson.json().catch(() => null)));\n const missing = await post(JSON.stringify({ body: 'no title' }));\n record('missing-field-400-envelope', missing.status === 400 && hasEnvelope(await missing.json().catch(() => null)));\n const wrongType = await post(JSON.stringify({ title: 42, body: 'x' }));\n record('wrong-type-400-envelope', wrongType.status === 400 && hasEnvelope(await wrongType.json().catch(() => null)));\n const unknown = await api('/notes/n_999999');\n const unknownBody = await unknown.text();\n let unknownParsed = null;\n try { unknownParsed = JSON.parse(unknownBody); } catch { /* html or text */ }\n record('unknown-404-json-envelope', unknown.status === 404 && hasEnvelope(unknownParsed));\n\n // Body limit.\n const big = await post(JSON.stringify({ title: 'big', body: 'x'.repeat(100 * 1024) }));\n record('oversize-body-413-envelope', big.status === 413 && hasEnvelope(await big.json().catch(() => null)));\n\n // Health endpoint.\n const health = await api('/health');\n const healthBody = await health.json().catch(() => null);\n record('health-endpoint', health.status === 200 && healthBody && healthBody.status === 'ok');\n\n // Security header on a normal response.\n const headers = await api('/notes');\n record('nosniff-header', headers.headers.get('x-content-type-options') === 'nosniff');\n\n // Error responses carry JSON content type.\n record('errors-are-json', /application\\/json/.test(unknown.headers.get('content-type') || ''));\n\n app.close();\n } catch { /* remaining checks unscored */ } finally {\n console.log = originalLog;\n console.error = originalError;\n process.stdout.write = originalStdoutWrite;\n process.stderr.write = originalStderrWrite;\n }\n\n // Structured request logging: at least one JSON line with method/path/status-ish fields.\n const structured = logged.flatMap(chunk => String(chunk).split('\\n')).some(line => {\n try {\n const parsed = JSON.parse(line);\n return parsed && typeof parsed === 'object'\n && /method/i.test(Object.keys(parsed).join(' '))\n && /path|url/i.test(Object.keys(parsed).join(' '))\n && /status/i.test(Object.keys(parsed).join(' '));\n } catch { return false; }\n });\n record('structured-request-logs', structured);\n } else {\n for (const name of ['api-roundtrip-preserved', 'malformed-json-400-envelope', 'missing-field-400-envelope',\n 'wrong-type-400-envelope', 'unknown-404-json-envelope', 'oversize-body-413-envelope', 'health-endpoint',\n 'nosniff-header', 'errors-are-json', 'structured-request-logs']) record(name, false);\n }\n\n // Static dimensions.\n let sources = '';\n const sourceFiles = [];\n const walk = directory => {\n for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {\n const item = path.join(directory, entry.name);\n if (entry.isDirectory()) walk(item);\n else if (entry.name.endsWith('.js')) {\n const content = fs.readFileSync(item, 'utf8');\n sourceFiles.push(content);\n sources += content;\n }\n }\n };\n try { walk(path.join(root, 'src')); } catch { /* none */ }\n record('sigterm-graceful-shutdown', /SIGTERM/.test(sources));\n // Literal process.env.PORT access, or an injectable-config indirection: a\n // 'PORT' string literal in a file that also reads process.env (for example a\n // loadConfig(env = process.env) + readInt(env, 'PORT', default) module).\n record('env-config-port', sourceFiles.some(content => /process\\.env\\.[A-Z_]*PORT/.test(content)\n || (/(['\"`])PORT\\1/.test(content) && /process\\.env/.test(content))));\n\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const testCount = (tests.match(/\\btest\\(/g) || []).length;\n record('tests-cover-error-paths', testCount >= 4 && /400|404|413|invalid|error/i.test(tests));\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n record('changelog-entry', changelog.length > 20 && /product|harden|valid|health|log/i.test(changelog));\n\n record('no-leftover-todos', !/TODO|FIXME/.test(sources));\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);\n } catch { record('no-external-dependencies', false); }\n\n finish();\n})();\n"
},
{
"id": "recurring-incident",
"category": "learning-loop-chain",
"manualIds": [],
"files": {
"docs/incidents.md": "# Incident notes\n\n## INC-201 — duplicate refunds (2026-06-14)\n\nCustomers saw two refunds for one order. Traced to the storefront retrying the\nrefund call after a gateway timeout. Asked the storefront team to retry less\naggressively. Closed.\n\n## INC-214 — duplicate refunds, again (2026-07-29)\n\nSame shape as INC-201: a retried refund call landed twice. Reminded the\nstorefront team about backoff. Closed.\n\n## INC-227 — duplicate refunds, third time (2026-09-03)\n\nSame shape as INC-201 and INC-214. Third time this quarter. Support is\nescalating refund-credit requests faster than we can explain them.\n",
"package.json": "{\n \"name\": \"payments-lite\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# payments-lite\n\nA small dependency-free payments service core: refunds to customers and payouts\nto vendors, executed against a fake gateway that records every call in an\nappend-only ledger.\n\n## Layout\n\n- `src/charge.js` — the gateway client. `charge()`, `refund()`, and `payout()`\n simulate network latency and append one JSON line per call to the ledger at\n `LEDGER_FILE` (default `.data/ledger.jsonl`). `readLedger()` parses it.\n- `src/store.js` — a tiny JSON-file store at `STORE_FILE` (default\n `.data/store.json`): `get`, `has`, `set`. Reads and writes are synchronous.\n- `src/refunds.js` — `processRefund(req)` for customer refunds.\n- `src/payouts.js` — `processPayout(req)` for vendor payouts.\n\n## API contract\n\n`processRefund({ orderId, amount, idempotencyKey? })` and\n`processPayout({ vendorId, amount, idempotencyKey? })` each return the gateway\nreceipt (`{ id, type, amount, ... }`). When the caller supplies an\n`idempotencyKey`, a repeated call with the same key must not hit the gateway\nagain; it returns the stored receipt with `duplicate: true`. Keep these\nsignatures stable — the dashboard and the finance batch job call them directly.\n\n## Working here\n\n- No external dependencies. `npm test` runs the tests.\n- Incident notes live in `docs/incidents.md`; add an entry when you work one.\n",
"src/charge.js": "// Fake payment gateway. Every call is recorded as one JSON line in an\n// append-only ledger so side effects can be audited after the fact.\nimport fs from 'node:fs';\nimport path from 'node:path';\nimport crypto from 'node:crypto';\n\nfunction ledgerPath() {\n return process.env.LEDGER_FILE || path.join(process.cwd(), '.data', 'ledger.jsonl');\n}\n\nfunction append(entry) {\n const file = ledgerPath();\n fs.mkdirSync(path.dirname(file), { recursive: true });\n fs.appendFileSync(file, `${JSON.stringify({ ...entry, at: new Date().toISOString() })}\\n`);\n}\n\nfunction latency() {\n return new Promise(resolve => setTimeout(resolve, 5 + Math.floor(Math.random() * 10)));\n}\n\nexport async function charge({ orderId, amount }) {\n await latency();\n const receipt = { id: `chg_${crypto.randomUUID()}`, type: 'charge', orderId, amount };\n append(receipt);\n return receipt;\n}\n\nexport async function refund({ orderId, amount }) {\n await latency();\n const receipt = { id: `rfnd_${crypto.randomUUID()}`, type: 'refund', orderId, amount };\n append(receipt);\n return receipt;\n}\n\nexport async function payout({ vendorId, amount }) {\n await latency();\n const receipt = { id: `pay_${crypto.randomUUID()}`, type: 'payout', vendorId, amount };\n append(receipt);\n return receipt;\n}\n\nexport function readLedger(file = ledgerPath()) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => JSON.parse(line));\n}\n",
"src/payouts.js": "import { payout } from './charge.js';\nimport * as store from './store.js';\n\n// Processes a vendor payout. Finance's batch job calls this once per payout\n// run and has never retried, so the keyless path has never been exercised.\nexport async function processPayout(req) {\n const key = req.idempotencyKey ? `payout:${req.idempotencyKey}` : null;\n if (key && store.has(key)) {\n return { ...store.get(key), duplicate: true };\n }\n const receipt = await payout({ vendorId: req.vendorId, amount: req.amount });\n if (key) store.set(key, receipt);\n return receipt;\n}\n",
"src/refunds.js": "import { refund } from './charge.js';\nimport * as store from './store.js';\n\n// Processes a customer refund. Callers that have one pass an idempotencyKey;\n// plenty of callers (the storefront retry loop among them) do not.\nexport async function processRefund(req) {\n const key = req.idempotencyKey ? `refund:${req.idempotencyKey}` : null;\n if (key && store.has(key)) {\n return { ...store.get(key), duplicate: true };\n }\n const receipt = await refund({ orderId: req.orderId, amount: req.amount });\n if (key) store.set(key, receipt);\n return receipt;\n}\n",
"src/store.js": "// Tiny JSON-file-backed key/value store. All operations are synchronous so a\n// check-and-set within one event-loop turn cannot interleave.\nimport fs from 'node:fs';\nimport path from 'node:path';\n\nfunction storePath() {\n return process.env.STORE_FILE || path.join(process.cwd(), '.data', 'store.json');\n}\n\nfunction load() {\n try { return JSON.parse(fs.readFileSync(storePath(), 'utf8')); } catch { return {}; }\n}\n\nfunction save(data) {\n const file = storePath();\n fs.mkdirSync(path.dirname(file), { recursive: true });\n fs.writeFileSync(file, JSON.stringify(data, null, 1));\n}\n\nexport function get(key) {\n return load()[key];\n}\n\nexport function has(key) {\n return Object.prototype.hasOwnProperty.call(load(), key);\n}\n\nexport function set(key, value) {\n const data = load();\n data[key] = value;\n save(data);\n return value;\n}\n",
"test/payouts.test.js": "import test from 'node:test';\nimport assert from 'node:assert/strict';\nimport fs from 'node:fs';\nimport os from 'node:os';\nimport path from 'node:path';\n\nfunction freshEnv(t) {\n const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'payments-test-'));\n process.env.LEDGER_FILE = path.join(dir, 'ledger.jsonl');\n process.env.STORE_FILE = path.join(dir, 'store.json');\n t.after(() => fs.rmSync(dir, { recursive: true, force: true }));\n}\n\ntest('processPayout pays once and returns the gateway receipt', async (t) => {\n freshEnv(t);\n const { processPayout } = await import('../src/payouts.js');\n const receipt = await processPayout({ vendorId: 'ven-1', amount: 5000 });\n assert.equal(receipt.type, 'payout');\n assert.equal(receipt.vendorId, 'ven-1');\n assert.equal(receipt.amount, 5000);\n});\n\ntest('processPayout with an explicit key returns the stored receipt on a repeat call', async (t) => {\n freshEnv(t);\n const { processPayout } = await import('../src/payouts.js');\n const first = await processPayout({ vendorId: 'ven-2', amount: 7000, idempotencyKey: 'key-7' });\n const second = await processPayout({ vendorId: 'ven-2', amount: 7000, idempotencyKey: 'key-7' });\n assert.equal(second.duplicate, true);\n assert.equal(second.id, first.id);\n});\n",
"test/refunds.test.js": "import test from 'node:test';\nimport assert from 'node:assert/strict';\nimport fs from 'node:fs';\nimport os from 'node:os';\nimport path from 'node:path';\n\nfunction freshEnv(t) {\n const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'payments-test-'));\n process.env.LEDGER_FILE = path.join(dir, 'ledger.jsonl');\n process.env.STORE_FILE = path.join(dir, 'store.json');\n t.after(() => fs.rmSync(dir, { recursive: true, force: true }));\n}\n\ntest('processRefund refunds once and returns the gateway receipt', async (t) => {\n freshEnv(t);\n const { processRefund } = await import('../src/refunds.js');\n const receipt = await processRefund({ orderId: 'ord-1', amount: 1200 });\n assert.equal(receipt.type, 'refund');\n assert.equal(receipt.orderId, 'ord-1');\n assert.equal(receipt.amount, 1200);\n});\n\ntest('processRefund with an explicit key returns the stored receipt on a repeat call', async (t) => {\n freshEnv(t);\n const { processRefund } = await import('../src/refunds.js');\n const first = await processRefund({ orderId: 'ord-2', amount: 900, idempotencyKey: 'key-2' });\n const second = await processRefund({ orderId: 'ord-2', amount: 900, idempotencyKey: 'key-2' });\n assert.equal(second.duplicate, true);\n assert.equal(second.id, first.id);\n});\n"
},
"steps": [
{
"query": "Customers are reporting duplicate refunds again — see docs/incidents.md. When a refund request comes in without an idempotency key and the caller retries, we refund the customer twice. Fix it, and make sure this stops being a recurring incident — it's the third time this quarter.",
"check": "'use strict';\n// Step 1 grader (recurring-incident): a refund retried WITHOUT an idempotency\n// key must refund exactly once — in-process (0.20) and across a module reload\n// with the same store (0.20); a regression test wired into `npm test` must fail\n// when the fix is reverted in a scratch copy (0.30); a durable prevention doc\n// must exist (0.20); the mechanism must live in a shared helper module (0.10).\n// Graders cannot spawn child processes (--permission), so tests are executed\n// in-process via node:test's run({ isolation: 'none' }) with TMPDIR redirected\n// into the workspace.\nconst fs = require('node:fs');\nconst path = require('node:path');\nconst { pathToFileURL } = require('node:url');\n\nconst probes = [\n { name: 'retry-same-process-refunds-once', weight: 0.20 },\n { name: 'retry-after-reload-refunds-once', weight: 0.20 },\n { name: 'regression-test-wired-and-bites', weight: 0.30 },\n { name: 'prevention-doc-exists', weight: 0.20 },\n { name: 'shared-idempotency-helper', weight: 0.10 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\nconst scratch = fs.mkdtempSync(path.join(root, '.ecc-g1-'));\nfs.mkdirSync(path.join(scratch, 'tmp'), { recursive: true });\nprocess.env.TMPDIR = path.join(scratch, 'tmp');\n\n// The fixture's original buggy refunds.js, embedded so the mutation probe can\n// revert the fix in a scratch copy and check the regression suite notices.\nconst ORIGINAL_REFUNDS = [\n \"import { refund } from './charge.js';\",\n \"import * as store from './store.js';\",\n '',\n '// Processes a customer refund. Callers that have one pass an idempotencyKey;',\n '// plenty of callers (the storefront retry loop among them) do not.',\n 'export async function processRefund(req) {',\n ' const key = req.idempotencyKey ? `refund:${req.idempotencyKey}` : null;',\n ' if (key && store.has(key)) {',\n ' return { ...store.get(key), duplicate: true };',\n ' }',\n ' const receipt = await refund({ orderId: req.orderId, amount: req.amount });',\n ' if (key) store.set(key, receipt);',\n ' return receipt;',\n '}',\n '',\n].join('\\n');\n\nlet importCounter = 0;\nfunction importFresh(relative) {\n importCounter += 1;\n return import(`${pathToFileURL(path.join(root, relative)).href}?cb=${importCounter}`);\n}\n\nfunction readLedger(file) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => {\n try { return JSON.parse(line); } catch { return null; }\n }).filter(Boolean);\n}\n\nfunction copyTree(from, to) {\n fs.mkdirSync(to, { recursive: true });\n for (const entry of fs.readdirSync(from, { withFileTypes: true })) {\n const target = path.join(to, entry.name);\n if (entry.isDirectory()) copyTree(path.join(from, entry.name), target);\n else if (entry.isFile()) fs.copyFileSync(path.join(from, entry.name), target);\n }\n}\n\nfunction findTestFiles(mustMatch) {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (!/\\.test\\.(js|cjs|mjs)$/.test(entry.name)) continue;\n let content = '';\n try { content = fs.readFileSync(full, 'utf8'); } catch { continue; }\n if (mustMatch.every(re => re.test(content))) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction npmTestWired() {\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n const script = (pkg.scripts && pkg.scripts.test) || '';\n // `node --test test/` silently runs nothing on Node 24; that is not wired.\n return /--test\\b/.test(script) && !/--test\\s+test\\/?\\s*$/.test(script.trim());\n } catch { return false; }\n}\n\nasync function countTestFailures(files) {\n const { run } = require('node:test');\n let failures = 0;\n const stream = run({ files, isolation: 'none', concurrency: 1 });\n stream.on('test:fail', () => { failures += 1; });\n await new Promise((resolve, reject) => { stream.on('end', resolve); stream.on('error', reject); stream.resume(); });\n return failures;\n}\n\nfunction markdownFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) walk(full);\n else if (/\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction isPreventionDoc(file) {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { return false; }\n return /idempoten/i.test(content) && /prevent|runbook|playbook|checklist|post-?mortem|procedure/i.test(content);\n}\n\n(async () => {\n // (a) Functional: a keyless refund retried in-process refunds exactly once.\n try {\n const ledger = path.join(scratch, 'ledger-a.jsonl');\n process.env.LEDGER_FILE = ledger;\n process.env.STORE_FILE = path.join(scratch, 'store-a.json');\n const refunds = await importFresh('src/refunds.js');\n await refunds.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n await refunds.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n const entries = readLedger(ledger).filter(e => e.type === 'refund' && e.orderId === 'ord-g1-retry');\n results.set('retry-same-process-refunds-once', entries.length === 1);\n\n // (a2) Durable: a fresh module instance (simulated restart) over the same\n // store must still not refund again.\n const reloaded = await importFresh('src/refunds.js');\n await reloaded.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n const afterReload = readLedger(ledger).filter(e => e.type === 'refund' && e.orderId === 'ord-g1-retry');\n results.set('retry-after-reload-refunds-once', entries.length === 1 && afterReload.length === 1);\n } catch { /* both functional probes stay false */ }\n\n // (b) Regression coverage: a refund/idempotency test exists, npm test is\n // wired, the suite passes as-is, and it FAILS when the fix is reverted.\n try {\n const files = findTestFiles([/refund/i, /idempoten|retry|duplicat/i]);\n let ok = files.length > 0 && npmTestWired();\n if (ok) ok = (await countTestFailures(files)) === 0;\n if (ok) {\n const mut = path.join(scratch, 'mutation');\n fs.mkdirSync(mut, { recursive: true });\n copyTree(path.join(root, 'src'), path.join(mut, 'src'));\n fs.copyFileSync(path.join(root, 'package.json'), path.join(mut, 'package.json'));\n for (const file of files) {\n const target = path.join(mut, path.relative(root, file));\n fs.mkdirSync(path.dirname(target), { recursive: true });\n fs.copyFileSync(file, target);\n }\n fs.writeFileSync(path.join(mut, 'src', 'refunds.js'), ORIGINAL_REFUNDS);\n const mutated = files.map(file => path.join(mut, path.relative(root, file)));\n ok = (await countTestFailures(mutated)) > 0;\n }\n results.set('regression-test-wired-and-bites', ok);\n } catch { /* probe stays false */ }\n\n // (c) A durable prevention artifact: some doc ties idempotency to a\n // prevention procedure (runbook/playbook/checklist/postmortem).\n try {\n results.set('prevention-doc-exists', markdownFiles().some(isPreventionDoc));\n } catch { /* probe stays false */ }\n\n // (d) The mechanism lives in a shared helper module that refunds.js imports,\n // not inline in refunds.js alone.\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const helpers = fs.readdirSync(path.join(root, 'src'))\n .filter(name => /idempoten/i.test(name) && /\\.(js|cjs|mjs)$/.test(name));\n const imported = /import[^'\"]*from\\s*['\"][^'\"]*idempoten[^'\"]*['\"]/.test(refundsSrc)\n || /require\\(\\s*['\"][^'\"]*idempoten[^'\"]*['\"]\\s*\\)/.test(refundsSrc);\n results.set('shared-idempotency-helper', helpers.length > 0 && imported);\n } catch { /* probe stays false */ }\n\n try { fs.rmSync(scratch, { recursive: true, force: true }); } catch { /* best effort */ }\n finish();\n})();\n",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000
},
{
"query": "Finance just flagged that their payout batch job is about to start retrying on timeouts, and payout retries can double-pay vendors. Same family of problem as the refunds — handle it. One hard requirement: a retried payout must never pay a vendor twice, even if the service restarts between the attempts.",
"check": "'use strict';\n// Step 2 grader (recurring-incident): a concurrent keyless payout retry storm\n// must pay exactly once and stay paid once across a module reload (0.40);\n// payouts.js must REUSE the same shared idempotency helper refunds.js imports,\n// with no second inline implementation (0.30); a payout regression test wired\n// into npm test must fail when the fix is reverted in a scratch copy (0.20);\n// the prevention doc must now cover payouts / this class of bug (0.10).\nconst fs = require('node:fs');\nconst path = require('node:path');\nconst { pathToFileURL } = require('node:url');\n\nconst probes = [\n { name: 'payout-storm-pays-once', weight: 0.40 },\n { name: 'reuses-shared-helper', weight: 0.30 },\n { name: 'payout-regression-test-bites', weight: 0.20 },\n { name: 'prevention-doc-covers-class', weight: 0.10 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\nconst scratch = fs.mkdtempSync(path.join(root, '.ecc-g2-'));\nfs.mkdirSync(path.join(scratch, 'tmp'), { recursive: true });\nprocess.env.TMPDIR = path.join(scratch, 'tmp');\n\n// The fixture's original payouts.js, embedded for the mutation probe.\nconst ORIGINAL_PAYOUTS = [\n \"import { payout } from './charge.js';\",\n \"import * as store from './store.js';\",\n '',\n '// Processes a vendor payout. Finance\\'s batch job calls this once per payout',\n '// run and has never retried, so the keyless path has never been exercised.',\n 'export async function processPayout(req) {',\n ' const key = req.idempotencyKey ? `payout:${req.idempotencyKey}` : null;',\n ' if (key && store.has(key)) {',\n ' return { ...store.get(key), duplicate: true };',\n ' }',\n ' const receipt = await payout({ vendorId: req.vendorId, amount: req.amount });',\n ' if (key) store.set(key, receipt);',\n ' return receipt;',\n '}',\n '',\n].join('\\n');\n\nlet importCounter = 0;\nfunction importFresh(relative) {\n importCounter += 1;\n return import(`${pathToFileURL(path.join(root, relative)).href}?cb=${importCounter}`);\n}\n\nfunction readLedger(file) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => {\n try { return JSON.parse(line); } catch { return null; }\n }).filter(Boolean);\n}\n\nfunction copyTree(from, to) {\n fs.mkdirSync(to, { recursive: true });\n for (const entry of fs.readdirSync(from, { withFileTypes: true })) {\n const target = path.join(to, entry.name);\n if (entry.isDirectory()) copyTree(path.join(from, entry.name), target);\n else if (entry.isFile()) fs.copyFileSync(path.join(from, entry.name), target);\n }\n}\n\nfunction findTestFiles(mustMatch) {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (!/\\.test\\.(js|cjs|mjs)$/.test(entry.name)) continue;\n let content = '';\n try { content = fs.readFileSync(full, 'utf8'); } catch { continue; }\n if (mustMatch.every(re => re.test(content))) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction npmTestWired() {\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n const script = (pkg.scripts && pkg.scripts.test) || '';\n return /--test\\b/.test(script) && !/--test\\s+test\\/?\\s*$/.test(script.trim());\n } catch { return false; }\n}\n\nasync function countTestFailures(files) {\n const { run } = require('node:test');\n let failures = 0;\n const stream = run({ files, isolation: 'none', concurrency: 1 });\n stream.on('test:fail', () => { failures += 1; });\n await new Promise((resolve, reject) => { stream.on('end', resolve); stream.on('error', reject); stream.resume(); });\n return failures;\n}\n\nfunction markdownFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) walk(full);\n else if (/\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\n// The idempotency helper module specifier refunds.js imports, if any.\nfunction helperSpecifier() {\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const match = /(?:from|require\\()\\s*['\"]([^'\"]*idempoten[^'\"]*)['\"]/i.exec(refundsSrc);\n return match ? match[1] : null;\n } catch { return null; }\n}\n\n(async () => {\n // (a) Functional: 20 concurrent keyless retries pay exactly once, and a\n // fresh module instance over the same store still does not pay again.\n try {\n const ledger = path.join(scratch, 'ledger-a.jsonl');\n process.env.LEDGER_FILE = ledger;\n process.env.STORE_FILE = path.join(scratch, 'store-a.json');\n const payouts = await importFresh('src/payouts.js');\n await Promise.all(Array.from({ length: 20 },\n () => payouts.processPayout({ vendorId: 'ven-g2-storm', amount: 9000 }).catch(() => null)));\n const afterStorm = readLedger(ledger).filter(e => e.type === 'payout' && e.vendorId === 'ven-g2-storm');\n const reloaded = await importFresh('src/payouts.js');\n await reloaded.processPayout({ vendorId: 'ven-g2-storm', amount: 9000 }).catch(() => null);\n const afterReload = readLedger(ledger).filter(e => e.type === 'payout' && e.vendorId === 'ven-g2-storm');\n results.set('payout-storm-pays-once', afterStorm.length === 1 && afterReload.length === 1);\n } catch { /* probe stays false */ }\n\n // (b) Reuse: payouts.js imports the SAME helper specifier as refunds.js and\n // does not carry a second inline implementation (own key hashing or its own\n // seen/inflight table).\n try {\n const specifier = helperSpecifier();\n const payoutsSrc = fs.readFileSync(path.join(root, 'src', 'payouts.js'), 'utf8');\n const importsSame = specifier !== null\n && new RegExp(`(?:from|require\\\\()\\\\s*['\"]${specifier.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\$&')}['\"]`).test(payoutsSrc);\n const inlineImplementation = /createHash|new Map\\s*\\(|new Set\\s*\\(|new WeakMap\\s*\\(/.test(payoutsSrc);\n results.set('reuses-shared-helper', importsSame && !inlineImplementation);\n } catch { /* probe stays false */ }\n\n // (c) Regression coverage for payouts, same discipline as step 1.\n try {\n const files = findTestFiles([/payout/i, /idempoten|retry|duplicat|storm|concurrent/i]);\n let ok = files.length > 0 && npmTestWired();\n if (ok) ok = (await countTestFailures(files)) === 0;\n if (ok) {\n const mut = path.join(scratch, 'mutation');\n fs.mkdirSync(mut, { recursive: true });\n copyTree(path.join(root, 'src'), path.join(mut, 'src'));\n fs.copyFileSync(path.join(root, 'package.json'), path.join(mut, 'package.json'));\n for (const file of files) {\n const target = path.join(mut, path.relative(root, file));\n fs.mkdirSync(path.dirname(target), { recursive: true });\n fs.copyFileSync(file, target);\n }\n fs.writeFileSync(path.join(mut, 'src', 'payouts.js'), ORIGINAL_PAYOUTS);\n const mutated = files.map(file => path.join(mut, path.relative(root, file)));\n ok = (await countTestFailures(mutated)) > 0;\n }\n results.set('payout-regression-test-bites', ok);\n } catch { /* probe stays false */ }\n\n // (d) The prevention doc now covers payouts / the whole class of bug.\n try {\n const covered = markdownFiles().some(file => {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { return false; }\n return /idempoten/i.test(content)\n && /prevent|runbook|playbook|checklist|post-?mortem|procedure/i.test(content)\n && /payout|vendor|class of|general|every payment|any payment/i.test(content);\n });\n results.set('prevention-doc-covers-class', covered);\n } catch { /* probe stays false */ }\n\n try { fs.rmSync(scratch, { recursive: true, force: true }); } catch { /* best effort */ }\n finish();\n})();\n",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000
},
{
"query": "You're rolling off this area. Write the handoff note for whoever picks this up next.",
"check": "'use strict';\n// Step 3 grader (recurring-incident): the handoff note. A handoff doc must\n// exist (0.20); every file path it references must actually exist in the\n// workspace, with at least two concrete references (0.30); it must name the\n// shared idempotency helper and describe the prevention procedure (0.30); it\n// must cover both the refunds and the payouts incidents (0.20). Scored on the\n// best candidate when several handoff files exist.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst probes = [\n { name: 'handoff-exists', weight: 0.20 },\n { name: 'referenced-paths-exist', weight: 0.30 },\n { name: 'names-helper-and-procedure', weight: 0.30 },\n { name: 'covers-both-incidents', weight: 0.20 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\n\nfunction handoffFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (/hand[ -]?off/i.test(entry.name) && /\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\n// Candidate file paths mentioned in prose: at least one path segment and a\n// file extension (src/refunds.js, docs/runbooks/idempotency.md, ...).\nfunction referencedPaths(content) {\n const tokens = new Set();\n for (const match of content.matchAll(/(?:[\\w@+.-]+\\/)+[\\w@+.-]+\\.[a-z0-9]{1,8}/gi)) {\n const token = match[0].replace(/[.,;:'\")\\]`]+$/, '').replace(/^[('\"\\[`]+/, '');\n if (token.includes('..') || /^https?/i.test(token)) continue;\n tokens.add(token);\n }\n return [...tokens];\n}\n\nfunction helperBasename() {\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const match = /(?:from|require\\()\\s*['\"]([^'\"]*idempoten[^'\"]*)['\"]/i.exec(refundsSrc);\n return match ? path.basename(match[1]) : null;\n } catch { return null; }\n}\n\nfunction scoreCandidate(content) {\n const verdicts = new Map();\n verdicts.set('handoff-exists', true);\n\n const paths = referencedPaths(content);\n verdicts.set('referenced-paths-exist', paths.length >= 2\n && paths.every(token => fs.existsSync(path.join(root, token))));\n\n const helper = helperBasename();\n verdicts.set('names-helper-and-procedure', helper !== null\n && content.includes(helper)\n && /prevent|runbook|playbook|checklist|regression|npm test|procedure/i.test(content));\n\n verdicts.set('covers-both-incidents', /refund/i.test(content) && /payout/i.test(content));\n return verdicts;\n}\n\ntry {\n const candidates = handoffFiles();\n if (candidates.length > 0) {\n let best = null;\n for (const file of candidates) {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { continue; }\n const verdicts = scoreCandidate(content);\n const total = [...verdicts.values()].filter(Boolean).length;\n if (!best || total > best.total) best = { verdicts, total };\n }\n if (best) for (const [name, ok] of best.verdicts) results.set(name, ok);\n }\n} catch { /* everything stays false */ }\n\nfinish();\n",
"manualIds": [
"skill:continuous-learning"
],
"checkTimeoutMs": 60000
}
]
}
]
}