创建人员:dev.developer.project.secondary / infodev-2
文件职责:仅闭环 V001 设计审核 HOLD/5 的 F1—F5。V001 中未被本文件明确替代的合同继续有效;冲突时以本文件为准。
基线:V001=24165 bytes / DA597448EFD47361779464B44BF92C8CDE9B4AC1769764EE0AF2301075C86045;审核回执=msg_20260813122712535_be5c8514;审计 ID=DEV-AUDIT-PROJECT-INFO-BILI-DYNAMIC-REFRESH-COLLECTOR-V001-DESIGN-REVIEW-20260813-001-TERMINAL。
记录方式:重型方案 focused amendment。V001+V002 取得原 reviewer PASS/0 前,产品 code/test/fixture、正式数据和小时任务继续冻结。
本修订只处理 F1_EVIDENCE_NO_NEW_PROOF_NOT_DETERMINISTIC、F2_REFRESH_PENDING_CRASH_RECOVERY_UNDEFINED、F3_MIXED_HISTORY_CATALOG_CONTRACT_UNDERSPECIFIED、F4_DUAL_MANIFEST_COMMIT_HAS_LOST_UPDATE_RACES、F5_BOUNDED_RUN_EVIDENCE_NOT_ATOMIC_WITH_TERMINAL_RESULT。
继续冻结:唯一账号 青枫浦上Q / UID 1420210197、exact dynamic URL、单轮最多一次 navigate/reload、15 秒 page ready、8 秒 DOM read、72 小时默认窗口、五终态优先级、公开 discovery 永不证明 no-new、正文/原图/video metadata+cover、no-overwrite、secret/raw-error fail-closed,以及两个 2026-08-13 真实样本的解释。本修订不新增真实刷新、API 重试、网络 provider、视频下载、扩展、policy/HKCU、正式写入或第二账号。
page_observation 与 no-newV001 evidence 的 coverage 输入 object 废止。浏览器步骤不得提交 coverage_complete/pagination_complete/page_truncated/all_cards_parsed/reached_window_start/end_of_feed 作为受信结论;携带这些字段即 E_EVIDENCE_SCHEMA。
READABLE evidence 根新增:
extractor={contract_id,contract_sha256,parser_version,parser_sha256}
page_observation={schema_version,limits,observations,terminal_marker}
实现新增 source-controlled bili_dynamic_page_observation_contract.json 和 bili_dynamic_page_extract.js,exact bytes/SHA-256 进入实现审核。运行证据必须匹配固定 contract_id=bili-visible-dynamic-page-observation-v1、parser_version=1 及已审核 hash;不得从 evidence 路径加载代码、selector 或正则。extractor 只读可见 DOM,禁止 Cookie/localStorage/header/profile/网络响应。
page_observation.schema_version=1。limits exact 值:
max_observations=64
max_unique_cards=200
max_cards_per_observation=50
max_marker_text_bytes=128
max_proof_bytes=524288
observations 为 1..64 项且 ordinal=0..N-1 连续;每项 exact keys:
ordinal,observed_at,cursor_before,cursor_after,
visible_node_count,parsed_card_count,cards,parser_error_count,limit_hit
observed_at offset-aware、非递减且位于 pending 的 refresh/read 时间内;cursor_before=0,后一项 before=前一项 after,cursor 非负;position,identifiers,stable_keys,published_at,content_type,source_url,parse_complete;identifiers exact keys=dynamic_id,opus_id,bvid,缺项用 JSON null;position 从 0 连续,parsed_card_count=len(cards);identifiers 和 source_url 重算、ASCII 排序去重,至少一个;不接受 opaque ID;parse_complete 由必需身份、时间、类型、URL 重算;parser_error_count 等于未能形成完整 card 的节点数;visible_node_count=parsed_card_count+parser_error_count;任何计数不一致拒绝,确保观察过但未解析的节点不能从 proof 中消失;limit_hit=NONE|OBSERVATION_LIMIT|UNIQUE_CARD_LIMIT|CARD_PER_OBSERVATION_LIMIT|PROOF_BYTE_LIMIT|TIME_LIMIT,CLI 按实物数/bytes/time 重算;达到任一上限即 hit,不允许截断后 no-new。canonical proof 最大 524288 bytes,distinct card component 最大 200。除最后 observation 外,每项必须新增 canonical component并推进 cursor,或已经出现有效 terminal condition。cursor 变化但 card set 不增长、重复页、cursor 回退、ordinal 断裂均为 NO_PROGRESS,coverage incomplete。
terminal_marker 为 null 或 exact object:
observation_ordinal,kind,selector_id,normalized_text,marker_sha256
只允许:
EXACT_END_OF_FEED:selector/text 必须匹配 source-controlled allowlist,语义仅为可见 footer“没有更多动态/已经到底”;具体 selector/text 在实现合同中冻结。隐藏、空、加载中、重试或未知 marker 拒绝。WINDOW_START_CARD:selector_id=CARD_PUBLISHED_AT,绑定一个已完整解析 card 的 stable keys 与 published_at<=pending.window_start;marker hash 由 card canonical JSON 重算。空页面只有 exact EXACT_END_OF_FEED 才可能 no-new;空列表容器、公开空结果或无 marker 不行。marker 必须绑定最后一个 observation;marker 后出现新 card 使 proof 无效。
identity_match = creator exact + final URL exact
sequence_complete = ordinals/cursors/times/cards 连续且无 NO_PROGRESS
parse_complete = all cards complete and total parser_error_count == 0
not_truncated = all limit_hit == NONE and no actual bound reached
window_terminated = valid WINDOW_START_CARD or EXACT_END_OF_FEED
coverage_complete = identity_match and sequence_complete and parse_complete
and not_truncated and window_terminated
只有 READABLE、catalog/manifest 健康、observed components 相对 catalog 无可保存新增,且 CLI-derived coverage_complete 才是 REFRESH_CONFIRMED_NO_NEW。200 项、64 observations、重复页、空页无 marker、未解析卡、parser/contract 漂移只能 partial(access/identity/unreadable 更高优先级不变)。
terminal slot 嵌入 bounded coverage_proof:extractor identities、observation/unique counts、ordered observation hashes、ordered stable-component hashes、terminal marker、derived booleans、derivation_version=1。不存正文、图像、DOM/HTML、raw error、header/query。完整 evidence 终态后删除,proof 随 168 小时 slot 有界保留。
public CLI 反例:伪造 complete 字段、200 items、64 observations、50 cards/524288 bytes 边界、重复页、cursor 回退/断裂、空页无 marker、隐藏/未知 marker、marker 后新卡、parser error、extractor/parser hash 漂移;全部 no-new token=0,正式 manifests 不变。
refresh-begin 在 existing StateLock 下拥有:
state_dir/refresh/pending.json
state_dir/refresh/incoming/<run_id>.json.partial
state_dir/refresh/incoming/<run_id>.json
state_dir/refresh/intake/<run_id>/
pending schema 2 exact core keys:
schema_version,run_id,owner_nonce,phase,task_id,creator_uid,
creator_dynamic_url,started_at,deadline_at,window_start,window_end,
config_sha256,state_manifest_preimage,formal_manifest_preimage,
evidence_path,intake_root,evidence_identity,planned_terminal,
transaction_identity,last_transition_at
owner_nonce=128-bit OS random 32 lowercase hex,仅作唯一性。evidence/intake 路径由 run_id 机械派生,不由输入覆盖。phase exact allowlist:AWAITING_EVIDENCE|EVIDENCE_BOUND|TRANSACTION_INTENT|BUSINESS_COMMITTED|TERMINAL_RECORDED。每次 phase 原子 replace+flush/fsync。
deadline=started+timeout_seconds,timeout 1..60s、默认30;未来超界、时间倒退、unknown phase/hash/path/reparse 都 E_RECOVERY_AMBIGUOUS,保留实物且零业务修改。
AWAITING_EVIDENCE pending并 fsync;RUN_STARTED;2后3前崩溃时 reopen 从 pending 重建同 run STARTED;slot失败不得返回可用 run,pending保留,不能 unlink 后创建新 run。
浏览器 controller 只写 exact .partial,flush/fsync 后 rename 到 exact .json,不覆盖 final。commit input 必须等于 begin 返回的 lexical final。CLI strict JSON/bytes/schema/secret/path校验后,把 bytes/hash/extractor identities 写入 pending并推进 EVIDENCE_BOUND;完整 evidence 已存在但 pending 尚 awaiting 时,reopen完成同一推进。
| 实物 | now | 唯一动作 |
|---|---|---|
| 无 pending;本hour无slot | 合法 | 允许新begin |
| awaiting;无final或仅partial | now<=deadline |
E_BUSY,不清理 |
| 同上 | now>deadline |
同run终态 unreadable/E_EVIDENCE_MISSING_AFTER_DEADLINE/exit4;只删除 exact owned partial |
| awaiting;final完整有效 | 任意 | 绑定为 EVIDENCE_BOUND 并继续;失败 evidence也写终态 |
| final截断、run/config/hash/path不符、秘密/reparse | 任意 | E_RECOVERY_AMBIGUOUS/exit3,保留全部 |
| EVIDENCE_BOUND | 任意 | 重派生 planned terminal;失败/no-new写slot,new进入事务 |
| TRANSACTION_INTENT/BUSINESS_COMMITTED | 任意 | 第5/6节恢复,不重复append |
| TERMINAL_RECORDED且slot exact | 任意 | 补/核 latest,清理 exact leftovers,返回同终态 |
| pending与slot run不一致、slot第三内容、未来hour | 任意 | E_RECOVERY_AMBIGUOUS,零修改 |
第一次决定 terminal_at 后写进 pending,replay不得刷新时间。测试覆盖 pending/slot/partial/final/绑定/intent前普通异常、KeyboardInterrupt/SystemExit、真实子进程kill及下一小时reopen;不永久busy、不生成第二run、不丢歧义实物。
51598 bytes / 0E69AF2AF1C4B0014D2033A21F3986E283ECFD758EA652E7F2416006736C8D73
48 schema_version=1 events
22 canonical entity components
16 SAVED content components
5 video tracking components
1 retryable content component (opus:1233480647120519170)
16个 SAVED primary artifacts 已只读核验存在且 hash匹配。fixture 必须用脱敏同构结构冻结 22/16/5/1,不能把48 events当48动态。
schema1 每行严格校验 JSON object、schema、creator/status/stable_id/source_url/published_at 类型和控制字符。creator UID缺失只兼容:creator exact 青枫浦上Q、当前 exact formal manifest、source host是登记Bilibili host;机械补catalog UID 1420210197,不改历史行。creator缺失/不同或显式UID不符 fail closed。schema2不允许缺UID。
token规则:stable_id为BV产生 bvid:<lower>;1..32位数字产生 opus:<digits>;video/opus URL产生对应ID token及 url:<canonical>;schema2 explicit IDs产生 dynamic:/opus:/bvid:/url:。不从title/path/query猜ID。URL路径身份必须与 stable/explicit字段一致,否则 E_CATALOG_IDENTITY_CONFLICT。tokens ASCII排序去重。
共享 exact token做 union-find,行号升序;representative=component全部token ASCII最小,entity ID=sha256("\n".join(sorted tokens))[:24]。每component每namespace最多一个值;只有同一schema2行显式给出 dynamic+opus时允许二者不同。两个BVID/opus/dynamic、URL冲突、content/video混合或creator冲突 fail closed。跨token union只来自同一已校验行或共享 exact token,不做标题/时间近似合并。
基线 exact 分类:
SAVED;CONTENT_ACCESS_PENDING_CHROME、ARTICLE_TEXT_BLOCKED_LOGGED_IN_CHROME_BRIDGE_TIMEOUT;29 个非 SAVED status exact allowlist(禁止 prefix/regex 泛化):
ARTICLE_TEXT_BLOCKED_LOGGED_IN_CHROME_BRIDGE_TIMEOUT
CONTENT_ACCESS_PENDING_CHROME
TRANSCRIPTION_PASS_COMPLETE
VIDEO_DOWNLOADED_COMPLETE_HANDOFF_SENT
VIDEO_DOWNLOAD_BLOCKED_AUTH_REQUIRED
VIDEO_DOWNLOAD_BLOCKED_AUTH_SESSION_SOURCE_REVIEW_HOLD2
VIDEO_DOWNLOAD_BLOCKED_EXTENSION_IDENTITY_VISIBILITY
VIDEO_DOWNLOAD_BLOCKED_EXTENSION_NOT_LOADED
VIDEO_DOWNLOAD_BLOCKED_RUNTIME_STABILITY_GATE
VIDEO_DOWNLOAD_BLOCKED_STANDARD_EXTENSION_LOADING_DEVELOPMENT_DISPATCHED
VIDEO_DOWNLOAD_BLOCKED_YTDLP_ARCHIVE_METADATA_CONTRACT_REPAIR_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_ARCHIVE_METADATA_SOURCE_REVIEW_CAPACITY
VIDEO_DOWNLOAD_BLOCKED_YTDLP_ARCHIVE_METADATA_SOURCE_REVIEW_RESUME_SCHEDULED
VIDEO_DOWNLOAD_BLOCKED_YTDLP_BUILD003_ARCHIVE_METADATA_TYPE_FALSE_REJECTION
VIDEO_DOWNLOAD_BLOCKED_YTDLP_BUILD004_NOT_INSTALLABLE_TREE_HASH_MISMATCH
VIDEO_DOWNLOAD_BLOCKED_YTDLP_BUILD004_STATIC_PASS_HASH_ONLY_REVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_BUILD005_EXACT_PASS_INSTALL_SCHEDULING_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_BUILD005_STATIC_PASS_HASH_ONLY_REVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_CANONICAL_TREE_HASH_SOURCE_PASS_BUILD005_AUTHORIZED_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_CONTROLLED_BUILD003_IN_PROGRESS
VIDEO_DOWNLOAD_BLOCKED_YTDLP_CONTROLLED_BUILD004_IN_PROGRESS
VIDEO_DOWNLOAD_BLOCKED_YTDLP_METADATA_REPAIR_SOURCE_REREVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_REPLACEMENT_BUILD_002_ARCHIVE_METADATA_MISSING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_REPLACEMENT_BUILD_REPAIR_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_TREE_HASH_CONTRACT_REPAIR_SOURCE_REVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_TYPE_CONTRACT_REPAIR_COMPLETE_SOURCE_REREVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_TYPE_CONTRACT_REPAIR_SOURCE_REVIEW_PENDING
VIDEO_DOWNLOAD_BLOCKED_YTDLP_TYPE_CONTRACT_SOURCE_PASS_BUILD004_AUTHORIZED_PENDING
VIDEO_DOWNLOAD_PENDING_EXTENSION
unknown schema1 status 一律 E_CATALOG_STATUS。fixture逐项冻结,审核核对 allowlist与48行 status集合 exact 相等。
component outcome:有SAVED=>CONTENT_SAVED;有video tracking且无content token=>VIDEO_TRACKED;只有retryable content=>CONTENT_RETRYABLE;其余fail closed。SAVED胜过早期retryable;CONTENT_SAVED/VIDEO_TRACKED抑制重复formal SAVED/TODO;CONTENT_RETRYABLE遇到完整新正文可保存,仍不完整只能partial、不能no-new。
path/bytes/sha256 三者必有,archive内普通非reparse相对文件,实物精确;这是唯一 content-saved authority。image_path/image_bytes/image_sha256、cover_path/cover_bytes/cover_sha256 各自全有或全无;有则archive内实物精确;同component同path必须同hash。*_sha256 必须64hex,只有上述formal artifact triple访问实物。历史16 SAVED中15只有primary text,1另有image triple;fixture覆盖。artifact缺失/篡改使整个catalog fail closed,不能把单实体当新。
formal schema2 dedupe_keys必须等于CLI重算ASCII sorted exact list,entity_id必须等于component算法,UID exact字符串,artifact paths为archive-relative POSIX;unknown status/event fail closed。
state manifest保持原 manifest_schema_version=1;非视频完整保存写 exact CONTENT_SAVED 并加入 ACTIVE_OR_SUCCESS_STATUSES,含原core keys、重算dedupe_keys和 artifact_refs[path,bytes,sha256]。video仍写 TODO_QUEUED。旧check/move/handoff不读取formal schema2,退出码不变。
测试冻结48行22/16/5/1,两条pending后SAVED胜出、20条同BV合并、UID兼容、URL/stable冲突、两个BVID错合并、unknown status、artifact篡改、schema2 identity漂移和retryable后保存。
existing StateLock(state_dir/.collector.lock) 保护所有旧命令及 refresh 对 state manifest 的读写。新增 archive_dir/.bili-dynamic-formal-manifest.lock,CreateNew、普通非reparse,写pid/run/time并fsync;stale lock不自动删,只能确认owner不存在且无pending事务后恢复。项目内所有formal writer必须共用该锁。
锁顺序固定:StateLock后formal lock,释放反序。commit/recovery从双前像compare到formal replace、terminal slot、latest完成全程持两锁,不得反向取锁。
candidate exact:
state_candidate = state_preimage_exact_bytes + canonical_state_lines
formal_candidate = formal_preimage_exact_bytes + canonical_formal_lines
preimage/candidate均存bytes+sha256;非空旧文件须单LF结尾;new lines按entity ID ASCII排序,每行canonical JSON+LF;旧行不重序列化,48行formal prefix字节不变。
双锁内在任何artifact发布前同时比较两前像;漂移即 E_MANIFEST_RACE_REBEGIN,不创建artifact、不覆盖manifest;同run写partial terminal,下一UTC小时才可新begin。
提交:compare both→intent绑定candidates→stage/CreateNew artifacts→再次compare both→replace/read state→复核formal preimage且state candidate→replace/read formal(业务commit point)→在双锁内完成第6节terminal。
恢复:两份preimage则删本run精确artifacts;state candidate/formal preimage则恢复state后删artifacts;两份candidate则业务已提交,只补terminal;任何第三hash/额外行/未知artifact都 E_RECOVERY_AMBIGUOUS;formal candidate而state非candidate绝不猜测重建。
测试用独立子进程让refresh与旧check/move竞争state、synthetic compliant writer竞争formal;覆盖compare/intent/artifact/state/formal/read全点kill、反向锁、stale锁、绕锁第三hash和48行prefix。
V001“同slot覆盖”废止。固定 runs/slot-000.json..slot-167.json 和 latest.json。epoch_hour=floor(started_at_utc_unix/3600),index=epoch_hour mod 168。每UTC hour只允许一个accepted run;第二不同begin在任何Chrome/evidence/formal动作前 E_RUN_HOUR_OCCUPIED/exit3。相同run_id是replay。
slot只有旧 hour_epoch<=new-168、旧record TERMINAL、无旧run pending/transaction且时钟单调时可替换。旧STARTED/未知/future或时钟回退 safety stop。
每slot canonical JSON,max524288 bytes,exact core keys:
schema_version,slot_index,hour_epoch,run_id,run_state,task_id,
creator_uid,started_at,deadline_at,terminal_at,status,error_code,
exit_code,refresh_action,refresh_count,page_authoritative,
coverage_complete,coverage_proof,evidence_sha256,input_item_count,
new_item_count,saved_artifact_count,state_manifest,formal_manifest,
artifact_tree_sha256,transaction_receipt,warnings
run_state=RUN_STARTED|TERMINAL。STARTED terminal字段为固定null/0;TERMINAL完整。warnings仅enum。transaction_receipt内嵌 phase,transaction_id,intent_sha256,business_commit_kind,state/formal preimage+candidate,created artifacts,recovery_count。no-new/unreadable/blocked/partial=NO_FORMAL_CHANGE且preimage=after。
planned terminal先写pending,含首次冻结的 status/error/exit/terminal_at/proof/counts/after identities;replay不换wall clock。slot bytes由planned terminal机械生成,同run exact相同。
new-items:intent→双manifest事务→formal回读后pending BUSINESS_COMMITTED→冻结planned terminal→slot TERMINAL原子replace/read(可报告NEW的审计commit point)→latest→pending TERMINAL_RECORDED→清理。formal已commit但slot失败时当前调用只返内部 E_RUN_EVIDENCE_COMMIT,pending保留;reopen补同slot后才输出原业务终态,不重复append。
no-formal-change:EVIDENCE_BOUND+planned terminal→manifests after=preimage→slot TERMINAL/read(终态commit)→latest→TERMINAL_RECORDED/清理。slot durable前不得输出五业务终态。
latest只含 schema_version,slot_index,hour_epoch,run_id,slot_bytes,slot_sha256,status,terminal_at,是可重建索引。latest失败时当前调用不返业务终态,pending保留;reopen由slot补latest。latest第三内容且指向更新hour不覆盖。
slot/latest成功而cleanup失败,终态权威,stdout可带 W_PENDING_CLEANUP;下一入口按slot清理exact leftovers,不回滚formal、不生成第二run。
逐点注入STARTED、planned terminal、BUSINESS_COMMITTED、slot temp/write/fsync/replace/read、latest全点、TERMINAL_RECORDED、evidence/intake/pending cleanup的普通异常、KeyboardInterrupt/SystemExit和子进程kill。覆盖同hour两run、相同run exact replay、168小时合法替换、167小时/STARTED/future/时钟回退拒绝、formal committed后补slot、no-new slot失败不能报no-new、latest重建/第三内容、cleanup失败和总文件/bytes上界。
| status | proof | manifest | terminal authority |
|---|---|---|---|
| NEW_ITEMS_SAVED | READABLE;item自身完整 | 双锁事务有新SAVED | formal candidate+slot TERMINAL |
| REFRESH_CONFIRMED_NO_NEW | CLI-derived complete | 两份不变 | slot内coverage proof |
| REFRESH_FAILED_PAGE_UNREADABLE | timeout/error/deadline无evidence | 两份不变 | slot TERMINAL,无no-new proof |
| REFRESH_BLOCKED_AUTH_OR_ACCESS | exact access/identity | 两份不变 | slot TERMINAL |
| PARTIAL_DISCOVERY_UNCONFIRMED | discovery-only/incomplete/race | 两份不变 | slot+enum reason |
公开 -412仍只作附证;15s/8s timeout仍是unreadable。实现新增 observation contract/extractor、bili_dynamic_refresh.py、脱敏48-line fixture和refresh tests;原collector只做helper/CLI/CONTENT_SAVED兼容,不改move/handoff状态机。
focused验收必须单列:contract/extractor hash与漂移反例、48-line exact summary、两进程race/crash、168 slot、terminal-before-stdout、proof独立重算、旧13项完整通过。性能:200 cards/64 observations、51KB manifests/22 components本地各<1s。slot最大524288,达到上限必须partial,不能截断后no-new。
状态=V002_FOCUSED_DESIGN_COMPLETE_PENDING_ORIGINAL_REVIEWER_IMPLEMENTATION_FROZEN。
产品 code/test/fixture modification=0;新增 Chrome refresh=0;network retry=0;download/video=0;formal write=0;extension/policy/HKCU/session secret=0。只有 V001+V002 原 reviewer PASS/0 后方可实施。