HHostingByteSoftware
Ctrl K
ComunidadGuías…

docs/adr/0020-scoring-confidence-formula.md

Rama master · réplica de lectura del repositorio original.

Archivo original ↗Preguntar a KiBot
# ADR-0020: Scoring Confidence Formula and Intrinsic Quality Coupling

## Metadata

| Field | Value |
|-------|-------|
| **Status** | Accepted |
| **Date** | 2026-06-11 |
| **Authors** | AlessioBrillo |
| **Deciders** | AlessioBrillo |
| **Supersedes** | N/A |
| **Relates to** | ADR-0002, ADR-0008 |
| **Project** | DOMINUS |

## Context

The scoring engine produces a `confidence` value (0.0–0.8 by default) alongside
`expectedValue` and `weightedScore`. Confidence answers the question: *how much
of the available signal weight was covered by actual data?* A domain with
keyword volume, comparable sales, and expiry history gets high confidence; a
domain with only an intrinsic evaluation gets low confidence.

The original formula (v0.1) used a simple per-signal additive model:
`confidenceBase + sum(confidencePerSignal × hasData)` with a hard cap at
`confidenceCap`. This was replaced in v0.2.1 with a
weight-covered-proportion formula that accounts for each signal's configured
weight.

However, the v0.2.1 formula introduced an undocumented coupling: 12% of the
confidence range is reserved for the intrinsic quality score, reducing the
signal-coverage contribution proportionally. This creates a scenario where a
domain with excellent commercial/market/expiry data but poor intrinsic quality
(hyphens, numbers, long string) has its confidence capped below the theoretical
maximum — even though all non-intrinsic signals have full data coverage.

This ADR documents the formula, the rationale for the coupling, and the
decision to expose the coupling factor as a configurable parameter.

## Decision Drivers

1. **Signal-coverage accuracy** — Confidence should primarily reflect how many
   signals have real data, not the quality of that data.
2. **Intrinsic quality floor** — A name that is objectively bad (long,
   hyphen-heavy, numeric-only) should have a realistically damped confidence
   even if keyword volume and sales data are excellent.
3. **Configurable conservatism** — The coupling strength should be tunable
   without code changes, respecting Principle 5 (scoring conservatism).
4. **Backward compatibility** — Existing scoring runs must retain their
   relative ordering after configuration changes.

## Considered Options

### Option A: Promote `intrinsicQualityInfluence` to a config constant (chosen)

Move the hardcoded `0.12` into `ScoringConstants.intrinsicQualityInfluence`
with a default of `0.12`, exposed as `SCORING_INTRINSIC_QUALITY_INFLUENCE`
in the environment configuration.

**Advantages:**
- Zero behavioural change for existing installations (default matches current)
- Operator can tune the coupling without code changes
- Documented in the Zod schema with full JSDoc rationale
- Follows the existing pattern of all scoring calibration constants
- The `confidencePerSignal` deprecated field can finally be removed from the
  Zod schema (made optional) since the weight-covered formula supersedes it

**Disadvantages:**
- Adds one more env var to the ~125 existing ones
- Requires a new ADR to document the formula retroactively

**Risk Assessment:** Minimal. The default is identical to the current value.
Changing it only affects relative confidence values, not buy/sell decisions
(the `recommended` threshold is independent).

---

### Option B: Remove intrinsic coupling entirely

Set `intrinsicQualityInfluence = 0` and derive confidence purely from the
proportion of signal weight covered by data.

**Advantages:**
- Mathematically pure: confidence reflects data coverage, not quality
- Simpler to explain and document

**Disadvantages:**
- A domain like `xkcd-7832-xyzzy.net` with keyword volume 100k + $20 CPC
  would receive the same confidence as `example.com` with the same data,
  despite the hyphen/number penalties making it a clearly worse investment
- Violates Principle 5 (conservatism): confidence should be conservative,
  and penalising bad names is conservative
- Existing tuning data based on the coupling would shift

**Cost Implications:** No implementation cost, but requires re-tuning all
existing weight configurations.

**Risk Assessment:** Medium. Changes the relative ordering of all candidates
and invalidates existing backtest-to-tuning feedback loops.

---

### Option C: Dynamic coupling based on signal variance

Compute `intrinsicQualityInfluence` dynamically from the variance of intrinsic
scores across the candidate pool — names with unusually high/low intrinsic
quality get more/less influence.

**Advantages:**
- Adapts to the candidate set automatically
- No manual tuning needed

**Disadvantages:**
- Non-deterministic across different candidate lists
- Impossible to backtest: the same domain would get different confidence
  depending on which other domains were in the same pipeline run
- Complexity: adds runtime state to a pure function

**Risk Assessment:** High. Non-determinism violates the principle of
deterministic scoring required for backtest correctness (ADR-0008).

## Decision

**Chosen option: Option A** — Expose `intrinsicQualityInfluence` as a
configurable constant in `ScoringConstants`, defaulting to `0.12`.

The choice preserves full backward compatibility while making the coupling
visible, documented, and tunable. The value `0.12` was empirically chosen
during v0.2.1 development: it provides enough damping to penalise clearly
bad names (long, hyphen-heavy) without overwhelming the signal-coverage
contribution when three of four signals have real data.

The confidence formula is:

```
coveredWeight = intrinsic.weight
              + (hasCommercialData ? commercial.weight : 0)
              + (hasMarketData ? market.weight : 0)
              + (expiryHasData ? expiry.weight : 0)

extraCovered = max(0, coveredWeight - intrinsic.weight)
variableRange = 1 - intrinsic.weight

signalConfidence = (extraCovered / variableRange)
                 × (confidenceCap - confidenceBase)
                 × (1 - intrinsicQualityInfluence)

qualityBoost = intrinsic.score
             × intrinsicQualityInfluence
             × (confidenceCap - confidenceBase)

confidence = min(confidenceCap,
                 confidenceBase
                 + signalConfidence
                 + qualityBoost)
```

The `intrinsicQualityInfluence` factor splits the confidence range into two
portions: `(1 - influence)` goes to signal coverage, `influence` goes to
intrinsic quality. At the default `0.12`, 88% of the range above the base is
driven by data coverage and 12% by intrinsic quality. This means:
- A perfect name (all data present, high intrinsic) reaches `confidenceCap` (0.8)
- A terrible name (all data present, intrinsic=0) reaches `0.8 - 0.12 × 0.6 = 0.728`
- A name with no data but high intrinsic reaches `0.2 + 0.12 × 1.0 × 0.6 = 0.272`

## Consequences

### Positive
- The coupling is now documented, versioned, and tunable via environment config
- Default zero-change for existing installations
- The `SCORING_CONFIDENCE_PER_SIGNAL` env var can be deprecated properly
  (made optional) since the weight-covered formula supersedes it
- Follows the established pattern for all scoring calibration constants
  (`SCORING_BUY_MAX_RATIO`, `SCORING_CONFIDENCE_CAP`, etc.)

### Negative
- Adds one configuration parameter to the ~125 existing env vars
- Requires updating the composition root to pass the new parameter

### Compliance and Security Implications
None. This is a pure computation parameter with no security or compliance
surface.

### Migration and Monitoring Plan
- The change is deployed as part of a hardening sprint — no separate rollout
- Existing pipeline runs retain their original confidence values in `scoring_runs`
  (the formula is applied at scoring time, not retroactively)
- New runs use the configurable value; default `0.12` matches the old hardcoded value

### Validation
- All existing scoring tests continue to pass with the default value
- The backtest engine's calibration reports show the same confidence-bucket
  distribution before and after the change (default-to-default comparison)
- Operators can validate by setting `SCORING_INTRINSIC_QUALITY_INFLUENCE=0`
  and observing that confidence now depends solely on data coverage

---

*This ADR was created following the MADR 4.0.0 standard. See
[ADR-0002](0002-scoring-engine-design.md) for the conservatism principle and
[ADR-0008](0008-backtest-engine.md) for backtest correctness requirements.*