Reality-first specification (v2.0.0, 2026-06-03). This document describes what the ingestion/selection layer actually does in code today, reconciled against the version-controlled implementation in ora-project/. The original v1.0.0 design — a continuously-running 10-layer GDELT-anchored selector with an editorial-supervisor MindSpec, a hold queue, a selection budget, and a feedback loop — was largely never implemented and the one module that came closest (scripts/cluster_selector.py) was deleted on 2026-06-02 (commit f74d36c65, “prune the GDELT subsystem”). That design is preserved for reference in the Design intent appendix and in vault git history (pre-rename, through commit 24ad52a44b, under the filename Framework — News Cluster Selector.md).

Display Name

MSI News Cluster Selector

Display Description

Layer 1 of the publication pipeline: decide what becomes news. In practice today this is not a single selector agent but a fan-out of independent per-source ingestion adapters, each of which discovers stories from one public feed, applies a small set of editorial filters, synthesizes a single-source “cluster” object, and durably hands off article preparation for the News Article Generator. Canonical source URLs prevent repeat intake; the final news editor compares finished reporting with recent coverage before deciding whether it publishes or adds a source. Similarity alone cannot suppress reporting or establish corroboration.

Implementation Status (read this first)

Spec concept (v1.0.0)Status todayWhere reality lives
cluster_selector.py runtime❌ Deleted (f74d36c65, 2026-06-02)—
GDELT polling (15-min cadence)❌ Removed (subsystem pruned)gdelt_event_ids is emitted as a hardcoded []
Editorial-supervisor MindSpec floor scoring⚠️ Orphaned code — tools/floor_engagement.py exists but its only caller (stage1_indexer) is dead; not in the live pathtools/floor_engagement.py::query_floor_engagement
10-layer cycle / milestones / self-evaluation❌ Not implemented—
Selection budget (daily cap, topic-diversity cap, confidence)❌ Not enforced on article selectionconfig/selection-budget.json read only by the dead stage1_indexer
Hold queue (tiers / expiration)❌ Does not existnearest analog: queue reset_stale / requeue_failed (crash recovery, not editorial)
Feedback loop adjusting reliability tiers❌ Does not existreliability tiers are hardcoded per adapter
Source-disqualification / reliability-tier configs❌ Dead config (read by nothing)tiers hardcoded in each adapter
Bad-faith coordinated-pattern detection at selection❌ Does not exist—
entity-resolution-config.json✅ Livetools/entity_resolver.py
Per-source ingestion adapters✅ Live (this is the real Layer 1)see Real pipeline
Same-event source-update handoff✅ Live; the final news editor judges finished reporting before copy; unavailable optional comparison fails opentools/event_dedup.py, tools/voice_editor_approval.py
US-anchored relevance filter✅ Live, default ontools/us_relevance.py (MSI_US_ANCHOR)
Event-driven review queue (op-eds/analyses)✅ Live (not in original spec)scripts/review_drain.py (MSI_REVIEW_QUEUE)

The Real Pipeline

Architecture

There is no central “selector.” tools/unified_production.py schedules hourly source intake into the existing SQLite production queue. Workers drain it asynchronously; the older backfill queue remains compatible.

unified_production hourly source jobs
  ├─ NPR        tools/npr_stage1.py        (discovery + extraction)
  ├─ RSS        tools/rss_ingest.py        (BBC → Guardian → ProPublica → UPI)
  ├─ WSJ news   tools/wsj_news.py          (shared subscriber browser)
  └─ AP         tools/ap_stage1.py         (realtime discovery)
       │  acquired article → single-member cluster JSON
       ▼
  canonical source-URL handoff (also used by backfill compatibility)
       └─ article.prepare → author → finish → final news editor
            ├─ new reporting → publish, revising a repeated lead when needed
            └─ wholly repeated reporting → article.source_update
                         → wait for confirmed live publication → patch Sources
                         → fresh news-index → deploy followup

UPI uses https://rss.upi.com/news/news.rss with the existing identifying User-Agent, HTTP client, and paragraph extractor for feed/article requests; no trafilatura requirement or RSS-description body fallback. ProPublica uses https://www.propublica.org/feeds/propublica/main with its existing exclusions intact.

For an original NPR URL, tools/npr_article.py reads a recognized storytext container with the standard HTML parser before trying optional generic extraction. Balanced nesting preserves reporting paragraphs, headings, and list items across embedded widgets, without duplicate list/paragraph output. Recognized image/caption, ad, related-story, audio, and subscription subtrees are excluded. A missing container produces a diagnostic warning and uses the existing fallback; an empty recognized story remains insufficient. Non-NPR callers retain their generic extraction, and Wayback retrieval retains the original NPR URL for selection of the reader.

Before expensive fetching, adapters deduplicate URLs, remove seen entries and cheap rejections, then order RSS/WSJ by publication instant newest first with URL ties; AP uses URL order because its date is unknown before fetching. For pool size N, cap L, and scheduled UTC hour h (hours since epoch), selection starts at h % N and wraps for up to L distinct candidates. Slot retries repeat the window; the next slot advances despite an early stop. In a fixed discoverable pool, each item becomes first within N slots if a run can attempt its first item. Changing feeds/discovery failures prevent a deadline guarantee. Standalone calls use the current UTC hour.

Seen means an editorial rejection or verified durable handoff; failures remain retryable while discoverable, without clearing old seen records. A bounded source lock covers seen/URL-key recheck, cluster write, enqueue acknowledgement, and marking; fetching stays outside. A concurrent terminal-seen result without a durable winner counts as skipped, not accepted. Canonical URL plus source key makes ingress independent of run date. Existing winners retain their stage, payload, cluster, and identity, including after a crash before seen marking. Callers without source URLs retain cluster-path keys; standalone AP/NPR discovery retains an explicit early-marking option.

RSS invokes publishers in the order above, each with a hard 360-second limit inside the 1,800-second job and the same schedule slot. Timeout/cancellation terminates and reaps children; failures do not stop later publishers. Existing task output/error detail preserves per-source HTTP/parse outcomes, newest supplied date, candidate/seen/rejection counts, body failures, and accepted/failed handoffs. Missing dates/interrupted counts stay unknown. Empty eligible feeds succeed; acquisition failures produce aggregate partial/nonzero after all sources run. Existing bounded retry preserves accepted work.

What “selection” actually is

Each adapter emits one acquired article as a one-member cluster. Event matching does not mutate an admitted cluster or make multiple sources a publication requirement. The selection logic remains the per-adapter editorial filter set:

  1. Section / category filter — drops sports / entertainment / lifestyle / weather (npr_article.section_verdict, _DROP_PATTERNS).
  2. Analysis / opinion drop — keeps the consensus newsfeed to reported news (npr_article.analysis_verdict).
  3. US-anchored relevance filter — tools/us_relevance.py::keep_story / is_us_anchored (regex anchors on US agencies, states, officials). Gated MSI_US_ANCHOR (default on).
  4. Body-length floor — requires at least 300 characters of extracted body; insufficient extraction stays retryable. AP/NPR check this before marking an editorial rejection seen.
  5. Recency / backdate window — drops evergreen hub articles outside the window (MSI_AP_MAX_BACKDATE_DAYS, default 7); realtime-vs-Wayback routing by article age (ap_stage1).

Reliability tier and outlet_class are hardcoded per adapter (AP = 1 / wire, NPR = 1 / public_broadcaster, BBC = 1, UPI = 2 / wire, Guardian = 2 / national_daily, ProPublica = 1 / nonprofit). The fields disqualifying_sources_present, protected_category_concerns_present, and gdelt_event_ids are emitted as hardcoded False/False/[] — no screening runs behind them.

Same-event source handoff (tools/event_dedup.py)

Source ingress preserves exact canonical-URL identity and admits fresh cross-source material for authoring. The final news editor compares the normalized finished headline, lede, and nut paragraph with actual recent reporting. Existing embeddings retrieve up to five eligible articles in the two-day window; deterministic comparison with recent article files covers indexing delay. Shared names and similarity select evidence, not a suppression verdict. The former raw-source thresholds remain diagnostic settings, not production admission rules.

The editor keeps new events and material developments publishable, revises a repeated lead around the unique reporting, and routes wholly repeated reporting to an exact prior article. It sees original source material and drafts; a prior article that required a rewrite stays in the comparison through revisions, with its current eligible reporting reread. Comparison articles are not new source evidence. Incomplete or unavailable comparison cannot suppress. A confirmed duplicate or required rewrite stays unresolved through retry, including an editor outage; a failed handoff never becomes publication of the known duplicate.

The URL first resolves any immutable winning row. A confirmed duplicate inserts or acknowledges one exact article.source_update row before completion, retaining the incoming cluster and target. Unified attempts preserve their real pipeline parent; direct/manual callers use the existing source-URL or exact cluster-path identity without inventing a parent. Winning clusters and publication identities never change. No second article, image, analysis, advocacy, or public storyline membership is created.

Targets require proven admitted authorship or an exact existing public article. The task resolves admitted publication through parent publish/live evidence, including exact copy-intent recovery. A public article without historical queue ownership is verified at its real URL without fabricating old tasks. Only confirmed live publication permits a source update, followed by an identity recheck under the article lock. Missing, changed, ambiguous, or unconfirmed targets retain material through bounded retry/dead handling. A real terminal author without an article or recovery instead sends the incoming cluster through fresh ordinary authoring and the final duplicate check; that handoff must succeed before completion.

Citations require real URL/title/outlet/available-date data and a substantive acquired body. URLs deduplicate; original IDs survive and added IDs are stable and unique. Every queued addition sets added_after_publication: true after live confirmation, carried through Markdown/site extraction to render “Additional coverage added after publication.” This includes reporting acquired during authoring. It does not certify corroboration; outlet-only metadata cannot supply a citation.

Every overlapping article writer shares a bounded lock around rereading, narrow edits, and atomic sibling replacement; expensive work stays outside. Deployment’s synchronized sanitize-articles.py and frontmatter-gate.mjs use the shared Python ownership helper; deferred quarantine decisions are rechecked under ownership. Where a registry is involved, its ownership comes first; multiple article locks use deterministic path order and retain existing rollback protection. Sources and totals change together; prose, citations, headline, slug, dates, and unrelated metadata survive. Temporary sibling files are removed on failure. Results are added/already-present/failed. A changed source gets its own followup.news_index → followup.deploy chain, without advocacy, analysis, or syndication.

Entity resolution (tools/entity_resolver.py)

The one config file from the original design that survives. entity-resolution-config.json drives entity_resolver, which resolves named entities to canonical identifiers. This runs as enrichment in the author path, not as a selector layer.

Hand-off to the article generator

The News Article Generator consumes the immutable admitted cluster through the production queue. Normal source-binding and publication-identity validation remain; editorial followups are separate.

Code Seams (canonical references)

  • Conductor / cycle: ora-project/tools/unified_production.py — hourly source handlers, production queue, prerequisite recovery, publish lane, and followups.
  • Source adapters:
    • ora-project/tools/ap_stage1.py — run_index_task, _synthesize_cluster (also available through backfill compatibility).
    • ora-project/tools/rss_ingest.py — SOURCES dict, run_source, _synthesize_cluster (BBC / Guardian / ProPublica / UPI).
    • ora-project/tools/npr_stage1.py — run_index_task; filters in tools/npr_article.py (section_verdict, analysis_verdict).
    • ora-project/tools/wsj_news.py — PER_FEED_MAX per-desk caps.
  • Relevance / dedup / entities: tools/us_relevance.py::keep_story; tools/event_dedup.py supplies finished-news comparison candidates to the final news editor; tools/entity_resolver.py.
  • Queue + drain: tools/unified_production.py; compatible ingress in tools/backfill_queue.py; author preparation through tools/msi_engine.py.
  • Downstream review (not selection): scripts/review_drain.py.
  • The real “budget” (governs images, not story selection): tools/article_salience.py — see MSI News Image Generator.

WSJ news, setup, cookie import, shared fetch (including opinion/backfill), diagnosis, and cleanup share tools/wsj_editorial_fetcher.py profile ownership from initialization through browser closure. Waiting is at most 30 seconds; busy is retryable and health cleanup reports deferred. Private lock-held cleanup avoids nested locking; only provably dead owners allow stale-file removal, never age-based killing. Success, exceptions, cancellation, and launch failure release ownership. Ora’s guard reauthenticates only on positive authentication/paywall evidence; busy/network/feed/browser failures, HTTP errors, and body shortfalls without it remain retryable operational errors. The WSJ free state means no detected authentication wall, not a complete body; partial and full require positive wall evidence. Shared fetch/diagnosis still require five substantive article paragraphs for health; news retains its three-paragraph floor and existing partial-wall treatment. Its parent allows 450 seconds around 420 seconds of children.

Configuration status

Of the original nine config files, only config/entity-resolution-config.json is live. These are present but read by nothing in the live path (decorative): feed-registry.json (every adapter hardcodes its own feeds), source-reliability-tiers.json, source-disqualification-list.json, protected-category-rules.json, and selection-budget.json (read only by the dead stage1_indexer).

Environment flags

MSI_US_ANCHOR (default on) · MSI_AP_REALTIME · MSI_AP_WAYBACK (off) · MSI_AP_MAX_BACKDATE_DAYS (7) · MSI_REVIEW_QUEUE · MSI_DISTRIBUTIONAL. MSI_DEDUP_THRESHOLD (0.70) and MSI_DEDUP_AUTO (0.85) apply to the retained diagnostic matcher. Live on/off state for the gated flags is environment-dependent (ora.env on the server) and not assertable from the repo — verify on the box before relying on it.

Known gaps / dead code (flagged, not fixed here)

  • tools/stage1_indexer.py (cmd_prepare/cmd_finalize) and tests/test_cluster_selector_bundles.py still reference the deleted scripts/cluster_selector.py. That code path is broken; the live AP path bypasses it by calling ap_stage1.run_index_task directly, which is why production still works. Candidate for removal/rewiring.
  • tools/floor_engagement.py (the MindSpec floor-scoring implementation) is reachable only through the dead stage1_indexer path — orphaned.
  • The dead config files above are candidates for deletion or wiring-in.

Editorial intent still in force

The original framework’s editorial commitments — the consensus values floor as the selection criterion, symmetric standards across political alignment, source-corroboration discipline, and bad-faith-pattern awareness — remain the publication’s stated standard. They are largely not enforced at this layer in code today. Where they bite at all, it is downstream: the floor and bad-faith discipline are carried in the author-model system prompt (see MSI News Article Generator) and in the editorial reference docs (MSI Consensus Values Floor, MSI Treatise), not as gates in the ingestion adapters. A future revival of MindSpec-based floor scoring at selection would re-activate tools/floor_engagement.py.

Appendix — Design intent (original v1.0.0, NOT implemented)

The original specification (2026-05-05) described an idealized continuously-running Layer-1 agent. It is retained here as design intent; none of it runs as written. Full text is in vault git history (pre-rename, through commit 24ad52a44b, filename Framework — News Cluster Selector.md).

The unbuilt design comprised:

  • Continuous GDELT polling (GKG + Events, 15-min cadence) plus a feed-registry.json of supplementary RSS/API feeds, with operational profiles (live-publication | dry-run | replay).
  • A 10-layer cycle with four milestones: (1) cycle init + config hot-reload + feedback drain; (2) feed polling + working-set assembly from new feeds and a hold queue; (3) entity/geography/temporal enrichment; (4) source-quality assessment + disqualification screening; (5) bad-faith coordinated-pattern detection (manufactured_controversy, coordinated_message_discipline, flooding_the_zone, goalpost_shifting, overton_window_manipulation); (6) editorial-supervisor MindSpec floor-value scoring (the canonical selection criterion); (7) selection-criteria + selection-budget application (daily cap, topic-diversity cap, selection confidence); (8) emission; (9) self-evaluation against ten criteria; (10) state persistence.
  • A hold queue with floor-engagement-tiered expirations (Tier 1 = 96h, Tier 2 = 48h, Tier 3 = 24h) for clusters awaiting corroboration.
  • A feedback loop adjusting per-outlet reliability tiers from downstream rejection / correction signals, bounded against oscillation.
  • A configuration bundle of nine hot-reloadable files (floor, editorial router, bad-faith catalog, disqualification list, reliability tiers, selection budget, protected-category rules, entity-resolution config, feed registry).

The editorial rationale behind that design — AI surfaces candidates against the consensus floor; asymmetric coverage from symmetric standards is the system working correctly; the supervisor MindSpec is the canonical floor-engagement locus — remains the intended north star for any future re-implementation.

Change Log

  • 2026-06-03 (v2.0.0) — Reality-first rewrite + rename to Framework — MSI News Cluster Selector.md. Documents the real per-source-adapter ingestion pipeline (continuous_cycle → adapters → event_dedup → SQLite queue → produce_article) as primary content; demotes the unimplemented 10-layer GDELT/MindSpec/hold-queue/selection-budget design to a design-intent appendix. Records the 2026-06-02 deletion of cluster_selector.py + GDELT subsystem, the dead config files, and the broken stage1_indexer reference. Added Obsidian aliases for the prior filename.
  • 2026-05-05 (v1.0.0) — Initial design specification (10-layer GDELT-anchored selector). See appendix.