Add isolated malware scanning and audit hostile archive handling before broader public use #12
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Priority recommendation: High security hardening for broader deployment with untrusted uploaders. This is not evidence of a confirmed remote-code-execution vulnerability, and no CVSS score is assigned without a demonstrated flaw.
What is already implemented and inspected:
Gap: no antivirus/signature scanning integration was found. Valid ZIP structure does not establish that the embedded media or metadata is safe for every parser.
Acceptance criteria:
Reference: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
Starting #12 after reading #14 and all current comments. Scope: audit upload admission through ZIP verification, metadata analysis and importer staging; add an opt-in, fail-closed isolated antivirus stage with explicit coverage outcomes, content binding, safe owner feedback and operator documentation. I will add harmless and malformed-input regression fixtures and verify scanner failure/limits, tenant authorization and container isolation. No production rebuild/restart or live-malware testing is planned. Implementation and runtime validation will be reported separately.
Implementation findings: ZIP CRC checking alone permitted ambiguous central/local headers and overlapping records. Added explicit single-disk ZIP/ZIP64 layout validation, encryption/duplicate-path rejection, and immutable completed-upload admission. The broker now enforces a separate offline scanner before verification, analysis, preview and import in required mode, validates exact-content receipts, and requires downstream hash rechecks. Flagged collections remain blocked until a fresh clean rescan or owner removal. Scanner failures distinguish flagged, incomplete and unavailable outcomes. Initial offline real-engine tests (harmless antivirus fixture, size and recursion limits) passed. FreshClam successfully fetched and tested managed signatures in a disposable volume. This exposed an outdated distro engine, so the final image is being pinned to the current official 1.4.6 LTS multi-architecture image. Final regression, isolation, UI and packaging checks are in progress. No production services have been rebuilt or restarted.
Implementation handoff: pushed commit
2905edfe48and opened draft PR #19: #19 . It is stacked on #15 so this review excludes the earlier diagnostics work.Implemented: broker-enforced offline ClamAV 1.4.6 LTS scanning in optional required mode; separate managed signature updater; explicit flagged/incomplete/unavailable results; stale-definition/version checks and resource limits; exact-content receipt validation and downstream hash binding; immutable completed uploads; persistent collection blocking and clean-rescan/owner-removal recovery; stricter ZIP/ZIP64 layout checks; owner feedback distinguishing antivirus from archive validity and metadata matching. Added the complete input-path audit, policy, activation/recovery instructions and security review in docs/malware-scanning.md and docs/malware-security-review.md.
Checks passed locally in Linux AMD64 containers: full Go tests/vet; backend and runner race checks; real offline ClamAV harmless antivirus fixture plus size/recursion limits; FreshClam official signature download/test/update; real managed-definition clean/flagged scans under scanner resource limits; actual Docker scanner kernel/mount/credential/egress isolation, missing-definition blocking and scan-to-analysis hash checks; browser/container smoke including both antivirus UI states; UI lint/build and zero-advisory npm audit; required-mode Compose configuration; Python CI/document contracts; redacted staged secret scan. Final source scan loaded all 36 files: 54 reviewed gosec findings, zero suppressions/loading errors; govulncheck passed the existing symbol-level policy. Scanner/updater image policies passed with explicit exact-version dispositions. Open native-library advisories remain documented, especially libxml2; these results do not mean zero vulnerabilities.
Limits and next step: required mode blocks any member above 512 MiB and nested archives. Review PR #19 together with #15, preserve the antivirus field when integrating the metadata/activity UI PRs, and run the native release gates on AMD64/ARM64. Then review final image identities, matching-source/notices publication, representative large-export capacity and current native-library patch status before coordinated activation. No production rebuild/restart, upload interruption, live malware or real photo-library access occurred. Test containers and the disposable signature volume were removed. Code is published on the review branch; images were built only locally. No deployment or issue closure is claimed. #12 remains open for the remaining acceptance/release verification.
Scope update approved by the user in this task: revise the fixed 512 MiB collection-blocking policy for normal Takeout compatibility. Add a configurable scan-size ceiling (within the engine limit), retain strict incomplete-coverage blocking, and add an administrator-controlled allow-large-media policy with explicit owner acknowledgement before preview/import. Oversized eligible media will be reported as not scanned; a partial result will never be called clean. Detections, outages, stale definitions, malformed/nested inputs and other incomplete outcomes remain blocking. I will update PR #19, tests and operating instructions; no deployment is requested.
Large-media revision completed and published in commit
dc60037430, updating draft PR #19: #19 . This supersedes the fixed 512 MiB limitation in the earlier handoff.The per-file scan ceiling is now configurable from 1 MiB through 2,000,000,000 bytes, defaulting to 1 GiB; engine and container limits change together. Default strict mode continues blocking incomplete coverage. The approved optional administrator policy
allow-large-mediaretains oversized recognizable media as explicitly unscanned, scans the remaining members, and permits analysis with a partial report. Each preview/import requires owner acknowledgement bound to the exact archive set, policy and reported gap. Stale consent, changed policy and mismatched previews are rejected. Partial coverage stays visible in saved reports/downloads and never becomes a clean result. Media headers only establish exception eligibility, not safety. Detections, outages, stale definitions, malformed/nested inputs and all other incomplete outcomes still block. Partial scans cannot clear prior detections; deleting only a flagged analysis record is also blocked. The separate updater continues automatic verified signature checks 12 times daily (about every two hours).Checks actually passed: full Go tests/vet; focused backend and runner race checks; real offline ClamAV tests including 640 MiB synthetic media members and harmless-marker detection beyond the old cutoff, with engine configuration at the 2,000,000,000-byte maximum; managed official signature update and actual Docker isolation/partial-coverage processing test (analysis allowed, verification blocked without acknowledgement, processing allowed with acknowledgement); API consent/policy/preview and recovery regressions; browser/container smoke with partial warnings, checkbox gating and reload reset; UI lint/build; local/production Compose configuration including custom policy/ceiling; Python CI/document checks; redacted staged secret scan. Final source review loaded 36 production files, with 57 reviewed findings, zero gosec suppressions/loading errors, and a passing govulncheck symbol policy. Native dependencies are unchanged from the documented image review; open advisories, particularly libxml2, remain open. A transient browser account-verification timeout passed on rerun; the final browser run passed.
No production service was rebuilt/restarted and no real media library was accessed. Disposable stack containers and signature volumes were removed; images were built locally, and code is published on the review branch. PR #19 remains draft and #12 remains open. Next: review the stacked PR with #15, run native AMD64/ARM64 release gates, review final image/source/notices publication and current native-library patches, test representative large Takeout capacity, then coordinate operator activation. Synthetic large-member tests do not replace that capacity check.
Resuming #12 at the owner's request after rereading #12, #14 and the detailed redesign in PR #19 comment 332. This supersedes the hold for this work only. First preserve the clean branch, fetch/rebase onto current main and establish a passing baseline, retaining all merged features. Then document and implement the incremental layered design: mandatory entry classification/validation, size-aware ClamAV, selective bounded structural checks, explicit file-level decisions, one security pass per run and aggregate resource controls. No extra antivirus engines, cloud submissions, per-file containers, deployment or interruption of active tasks is planned. Performance will be measured; the 15% target and representative workload will not be claimed without evidence. #12 and PR #19 remain open/draft.
Redesign findings and checkpoint plan after PR #19 comment 332:
The branch has been rebased onto current main
0a47f4f, preserving the merged activity/cancellation, metadata coverage, advanced-import options, notifications, SSO and mobile work. The old headdc60037is preserved in a backup ref. Baseline reconciliation is commitf011186(rebased predecessors b38c1ea/e516951). Full Go/vet and focused backend/runner race suites, UI lint/build, all six focused browser suites, the complete disposable upload/resume/restart/queue/browser/settings walkthrough, and nine CI/source/workflow/document contracts passed. Reconciliation retained stricter confirmed-cleanup behavior instead of weakening it to satisfy old scanner fixtures.Identified cost: integrity verification decompresses members, then the scanner decompresses them again; subsequent processor/preview/import stages repeat additional work. Identified security gap: a plausible extension and a short header are insufficient structural evidence for the proposed oversized exception. I have written an incremental specification in docs/layered-ingestion-design.md, separating mandatory classification, applicable AV, selective structural checks and final authorization. It preserves whole-collection blocking because the current importer sees the original ZIPs; it will not claim every file infected or forward rejected entries under an unimplemented filtering promise.
First implementation step is sharing the entry stream between integrity verification and AV. Regression tests explicitly cover a consumer leaving a corrupt suffix unread and a consumer swallowing a checksum error: neither may yield usable receipts. This removes the scanner's duplicate decompression without relaxing its existing gate. It does NOT yet implement the new classifier, structural eligibility, durable per-file diagnostics or cross-stage scan reuse. Those remain explicit next steps, including authenticated, expiring evidence bound to immutable inputs and policy/engine/definition identity. A client 'already scanned' flag is not acceptable.
The owner confirmed there is no available full-size synthetic workload and authorized creating one. The existing 23,000-image fixture measures diagnostics capacity only. The plan specifies a scalable media workload, reduced-byte trial first, fixed resource caps, realistic compressibility, current-main comparison and recorded wall/CPU/RSS/I/O. Neither the 15% target nor a real 150 GB ingestion benchmark is claimed. No production deployment, interruption or real photo-library access has occurred. #12 remains open and #19 draft; further evidence and exact next steps will be logged before handoff.
Resumable layered-redesign handoff (2026-10-27): source is published for review, not complete, merged or deployed.
Branch
issue-12-isolated-malware-scanningis clean at23cfc67c570bdc4f935f6d4a1b853626f5fd2a7d. PR #19 remains draft and now targets main: #19 . Current main0a47f4f6fc2455dc304df49fa8c45084ff4a2d65is an ancestor. The original published headdc6003743056822cca9c4e81fc1f436bc5ac8635is preserved both locally and remotely onbackup/issue-12-pre-layered-20261027. The owner-required rebase used an exact force-with-lease against that original head after publishing the backup; no merge or force merge occurred.Commits after rebase:
b38c1eaande516951preserve the two original scanner changes;f011186preserves merged lifecycle/metadata/advanced-option/UI behavior and establishes the passing baseline;23cfc67adds the first performance foundation and the detailed redesign specification. Read PR #19 comment 332, comments 402/405 here, and docs/layered-ingestion-design.md before the next code change.Implemented in this increment:
InspectArchivesreuses the existing archive safety checks and supplies the same bounded entry stream to the scanner. It drains unread bytes and retains any consumed read/checksum error. Receipts are withheld on inspection failure, CRC failure, actual-length mismatch, close failure or resource limits. The scanner no longer decompresses each member separately after integrity verification. This is one pass INSIDE the scanner; later processor verification and cross-stage repeated antivirus are still present and must not be described as fixed.Evidence actually passed:
Exact remaining implementation order:
No production service, active task/upload or real photo library was touched. Disposable baseline stack containers/volumes and benchmark containers were removed; local test images/build cache remain. #12 remains open and #19 draft because the layered redesign and representative validation are unfinished. This is the checkpoint requested by the owner for continuation if context/usage runs short.
Continuing from checkpoint 410 at the owner's request. Reread #12/#14/PR #19 steering; remote main is unchanged at
0a47f4fand the issue branch is clean at23cfc67. Next increment: bounded content classification for every member in the configured security pass, explicit media/ancillary format policy, and selective stream-based structural checks before oversized exemptions. Reuse the existing CRC/size-limited entry stream; no whole-file copies, full decoding, extra service or per-file container. Evaluate/pin the maintained pure-Go MIME detector, retain its license, and version the stronger validation policy so prior header-only evidence cannot authorize it. Add valid-media, disguised-executable, mismatched-type, malformed-container and resource-bound regressions. Per-file persistence and authenticated cross-stage reuse remain separate subsequent increments; no deployment or issue closure is claimed.Classification/structural-validation increment published:
2d589e2522(2d589e2), plus4b443ddfor LF scanner configuration on Windows. Draft PR #19 still targets main: #19 . The branch is clean; #12 remains open and the redesign is not complete or deployed.Completed:
.confnow checks out as LF. FreshClam subsequently fetched/tested verified daily 28166/main 63/bytecode 339; scheduled checking remains 12 times daily.Checks actually passed: final full Linux AMD64 Go/vet and focused backend/runner race suites (
immich-bridge:issue12-types-verified-tests); real offline engine harmless-fixture/size/recursion/640 MiB tests; managed official-definition scan of 11 generated media/JSON entries plus harmless detection; real Docker kernel/mount/network/credential isolation and structural partial analysis -> blocked unacknowledged verification -> successful acknowledged processing (39 seconds); UI lint/build; nine CI policy tests and staged source/workflow/document contracts; redacted staged secret scan. Final source review (temp/issue12-types-security-verified/): 47 files, 53 reviewed findings, no loading errors or suppressions; govulncheck symbol policy passed. Numeric handling was tightened rather than adding broad suppressions; only the changed existing ceiling-conversion fingerprint needed disposition. A limited 15-second one-worker host fuzz run passed 2,984 inputs; it is not comprehensive parser assurance. Native image advisory scans and full browser walkthrough were not repeated for this increment; the earlier rebase browser baseline, unchanged native dependencies/open advisories, current UI build and new backend/isolation checks are distinguished in the docs.Classification-only microbenchmark: one CPU, 256 MiB, three runs on the tiny MP4 fixture, median 2342 ns/op, 10,272 transient allocated bytes/op and 11 allocations/op. This is one format/sample, not job peak RAM, AV/structural throughput or the 15% end-to-end target. No representative 23,000-asset/150 GB workload has been generated or run yet. Legal movie padding is used only for functional small-threshold tests, not claimed as realistic video workload evidence.
Known limits to preserve: ordinary supported formats can be classified/scanned, but oversized HEIC/AVIF/TIFF/RAW/WebM/Matroska/AVI/GIF/WebP and fragmented/unknown movie layouts lack structural exemptions and are held explicitly. Some ambiguous RAW identities remain unsupported. Ancillary files cap at 16 MiB. Probe limits are 180 seconds/member, 16 MiB retained movie metadata, 100,000 boxes/chunks, depth 8, 128 tracks/references/descriptions, 1,000,000 samples/track and one billion declared image pixels. These checks establish selected invariants, not complete codec/media validity. Strict mode intentionally still blocks all incomplete coverage. Scanning remains opt-in, and not-configured never implies classification ran. Whole-collection blocking stays because the importer sees original ZIPs; no selected-file filtering is implemented.
Next steps, in order:
Ignored local evidence: temp/issue12-types-; fixture generation under temp/issue12-fixture- and the retained testdata recipe. Disposable signature volume and all test containers were removed; local test images/build cache remain. No production service, active task/upload, destination library or private photo was accessed or interrupted.
Continuing at the owner's request from checkpoint 419. Next increment is bounded, private per-file security evidence: retain identities, fixed validation/AV/probe outcomes and explicit unvisited status without adding file arrays to routine polling or broker memory. Inspecting existing diagnostics, runner streaming and retention boundaries before wiring persistence; final entry integrity must be checked before any file is reported usable. Cross-stage authenticated reuse and representative benchmarking remain subsequent work. No deployment, merge or issue closure is authorized or claimed.
Private-reporting implementation finding: Docker stdout is deliberately bounded (4 MiB rotating logs / 32 MiB response), so per-file arrays cannot safely use that channel. The scanner now has one narrowly scoped writable results submount, separate from read-only inputs/definitions. Bounded disk-backed inventory/results/index files keep identities and unvisited entries private and out of broker buffers. The existing verifier has a post-integrity observer so a preliminary clean AV reply cannot become an allowed file before draining/CRC/length/close checks. The scanner must sync/close its ledger before returning usable coverage; policy version 3 invalidates earlier aggregate-only validation. Owner-only API paging/download and existing seven-day retention/collection deletion are wired; no administrator privacy override. Work is still uncommitted while tests/review run. Full Linux tests/vet and focused race suites have passed an initial build; current final-source/security and actual Docker/managed-engine validation are in progress. Reporting-only three-run microbenchmark (23,000 synthetic identities, full result writes, periodic fsync, one CPU / 256 MiB) median about 1.054 seconds and 12,995,780 stored bytes. This excludes media/AV and is not the 15% end-to-end target or a real 150 GB benchmark. The writable mount is a deliberate isolation change requiring explicit kernel/scope checks; the 256 MiB ledger limit is application-enforced, not a new filesystem quota. No deployment or merge.
Private per-file reporting increment published for review:
0b318ee9f5143da62610d8b61398ba818fe63a63(0b318ee) onissue-12-isolated-malware-scanning. Branch is clean and synced; main remains0a47f4f, an ancestor. Draft PR #19: #19 . #12 remains open. Read PR comment 332, checkpoints 419/422, and docs/private-security-findings.md before continuing. Original headdc60037remains preserved locally/remotely on backup/issue-12-pre-layered-20261027.Implemented:
Checks actually passed:
immich-bridge:issue12-private-final; logtemp/issue12-private-final.log.Next implementation order:
Disposable signature volume and all test containers/volumes were removed; local images/build cache and ignored temp evidence remain. The definitions metadata file points to the now-deleted volume and must not be reused blindly. No production service, active upload/task, private photo library or actual destination was accessed, restarted or deployed. This is the resumable checkpoint requested by the owner; the layered redesign is unfinished.
Continuing at the owner's request from checkpoint 426. Main is unchanged at
0a47f4fand the branch is clean at0b318ee. Implementing broker-authenticated, expiring analysis evidence and one-use attempt execution permits so verification/preview/import consume a valid security run instead of silently rescanning. Bind immutable input hashes, explicit collection/operation, policy 3, worker images, actual managed-definition identity and coverage. Preserve owner consent/options/account matching and collection flags; reject stale/missing/forged/replayed evidence with explicit reanalysis, never a permissive fallback. Broker restart will invalidate outstanding capabilities. Root-only authentication material stays in the existing control volume, definitions are viewed read-only, and no new service is added. First establish the existing cached baseline; then wire protocol/processor/API changes with boundary and workflow tests. No deployment, merge or benchmark target claim.Authenticated reuse implementation is passing full Linux backend/vet and focused race checks, including the API private-evidence/permit transport and original-ledger retention tests. The fresh source scan still has 51 reviewed gosec findings with no suppressions, but the current vulnerability database adds GO-2026-6860 and GO-2026-6866 against indirect x/net v0.61.0 (module-only here; affected x/net/html is not imported). Updating to the released v0.62.0 fix and rerunning affected checks rather than expanding the advisory baseline. This remains incremental draft work; realistic workload timing and final release gates are still outstanding.
Authenticated cross-stage reuse is committed and pushed in
dcc2e5e2c3(dcc2e5e), draft PR #19 targeting unchanged main0a47f4f. The checkout is clean/synced. #12 remains open and the redesign is unfinished.Implemented this increment:
Checks actually passed:
All created test containers/networks/volumes, including the uniquely scoped managed-definition volume, were cleaned up. No production service/upload/task was interrupted; no merge/deployment/private-media access/gate waiver occurred. Draft PR and docs now describe final behavior rather than the superseded repeated-scan design.
Next implementation: aggregate evaluated/scanned/exempt/blocked/not-evaluated/errors/retry outcomes and honest overlapping stage timings using existing reports; then a deterministic scalable realistic workload, reduced-byte trials before 150 GB, measuring current-main versus redesigned analysis on the same host/caps/cache with CPU/peak RSS/I/O and at least three runs. The 23,000-image diagnostic fixture is not that benchmark; the 15% target is unmeasured. Broader oversized-format eligibility, native ARM64/current image security/release gates and coordinated installation remain required. Preserve the acknowledged unscanned distinction, whole-ZIP blocking and documented software-only report disk-cap limitation.
Continuing from
dcc2e5eat owner request. Latest #12/#14/PR steering read; checking current-main ancestry and the cached Linux test baseline before edits. Next increment extends existing aggregate security reports with post-integrity ledger outcomes and bounded numeric timings. Shared integrity/classification/AV/probe work overlaps: timings will identify wall versus cumulative/inclusive spans and will not be added into a misleading total. No per-read timing overhead, filenames, new service or weaker eligibility is intended. Then build a deterministic reduced synthetic workload and measurement harness before considering the capacity-checked 150 GB case. The 15% target remains unmeasured. No deployment/merge/restart of production is authorized by this work.The owner clarified that security and performance are both requirements, and the ordinary experience must remain nontechnical. Implemented counts on the existing post-integrity ledger path, with timing clocks at stages/entries rather than per read; no extra file traversal. UI uses a plain File check summary and a separate Technical timing details disclosure. Initial backend/race/source policies and private-findings browser regression passed. Now adding a disposable reduced-workload controller using the real broker/client/workers and exact archived-main processor. Its generated noise JPEG/PNG, tiny HEIC/video templates and sampled resource counters will be explicitly labeled as functional harness trials, not 150 GB realism or exact peak RSS. No benchmark target or protection/performance tradeoff is assumed from small fixtures.
Published
edee4a3b47(edee4a3), draft PR #19, targeting unchanged main0a47f4f. Checkout is clean/synced. #12 remains unfinished/open. This continuesdcc2e5eauthenticated reuse; no merge/deployment/production interruption occurred.The owner's additional steering remains binding: security and performance are both requirements, and ordinary users should get nontechnical results/next actions. Technical detail belongs behind an explicit disclosure. We have not declared the current cost acceptable merely because the security checks work.
Implemented:
Measured results, three paired runs per final profile:
Checks actually passed:
Next work: profile file-count versus byte-size effects and steady-state AV cost; replace tiny repeated HEIC/video templates with realistic encoded data and genuine oversized videos; obtain more precise CPU/peak RSS/logical and block-I/O accounting on the target host. Then run representative repeated baseline/layered trials and the capacity-checked 150 GB case. Preserve both security and practical performance: do not silently turn off image parsing, skip applicable AV or raise unbounded concurrency to manufacture a target result. Any proposed change needs measured cost and explicit protection rationale. Broader oversized-format eligibility and final native image/release/coordinated installation checks remain outstanding. docs/security-measurements.md records semantics, actual numbers and limitations for resumption.
Owner supplied a real multipart archive and authorized realistic local analysis testing. Continuing from
edee4a3. First prepare capacity-checked, read-only input aliases outside Git and a bounded isolated metadata-only inventory; avoid another full-size archive copy. Keep originals unchanged, acquire read-sharing locks during tests, and keep all file identities/metadata/private paths out of shared output and issue comments. Real-data test artifacts remain private/local; shared reports contain numeric aggregates only. Adapt the existing benchmark's test-only input transport as needed without relaxing production mount policy or scanner eligibility. Start with inventory and a controlled analysis pair before repeated full workload claims. No destination/import, production restart, merge or gate waiver is authorized by this testing step.Read-only real-input preflight completed: 3 ZIP parts, 155,354,774,733 compressed bytes, 67,013 inventoried regular files; 16 exceed 1 GiB and 3 exceed 2 GB. This is metadata-only inventory, not an integrity/security pass. Originals stayed unchanged. Windows sharing-lock tests block writers, and actual Docker/broker baseline plus managed-signature analysis passed on a tiny generated fixture through the copy-free input adapter. Production mount policy/code is unchanged; only the explicit test transport maps the supplied collection's opaque aliases as read-only binds. Starting one real baseline/layered analysis comparison under existing worker caps and the acknowledged large-media analysis policy; coverage gaps remain explicit and no import is performed. Private mapping/findings stay outside Git/in a private test report volume. Repeated expensive runs are deferred until the initial comparison is valid. Current managed test definitions are daily 28167/main 63/bytecode 339.
Private real-input harness is committed/pushed as
f0ecca9010(f0ecca9); branch clean/synced, main unchanged at0a47f4f. Production parsing/mount policy and workers are unchanged. docs/private-archive-testing.md records the explicit test-only RO-bind adapter, Windows hard-link/read-sharing protections, outside-Git artifacts, bounded inventory, report capacity checks, scoped cleanup and intentional private findings retention. Three native Windows tests passed, plus Linux backend/vet/races and a real tiny-fixture baseline/managed-signature pair through the adapter (0.328/8.069 seconds; compatibility check only). Nine CI policy tests, staged source/workflow/document contracts, Python compilation and redacted staged secret scan passed. No private archive, filename, metadata or local path was committed.The first actual 155 GB comparison is still running in its baseline verification phase, now reading the third ZIP part. It is using existing one-CPU/two-GiB baseline limits, with observed process high-water RSS around 36 MiB and no full archive copy. Read-only aliases and Windows sharing handles remain held; private phase logs and a local resume marker identify the existing controller. Do not launch a duplicate on resumption: inspect the current private run/report and controller first. The script will attempt one security-enabled analysis after a complete baseline and stop at a hold/failure rather than repeatedly rescanning. Temporary report storage was checked separately; originals and private identities remain local. Windows-to-Docker bind I/O differs from Linux deployment storage, so this first trial establishes compatibility/holds and a local observation, not a target-host performance claim. No completed real-data timing or 15% target result is claimed yet; no destination/import, deployment, merge or production interruption.
The first real-data main baseline completed successfully in 3,001.854 seconds through the documented read-only Windows bind path. All three ZIPs completed integrity verification; analysis reported 33,410 media entries and 26,533 preliminary metadata candidates. These are export-entry/filename-candidate counts, not unique assets or importer predictions. Security-enabled analysis has now started automatically on the same held read-only archive set under the existing 1 GiB/allow-large-media analysis policy. No completed security timing/coverage verdict exists yet. One local baseline observation is not a three-pair or target-host performance result. Keep the existing private controller running; do not start duplicate real-data scans. No private identities/metadata/paths or destination access are involved in shared output.
The first real-input comparison stopped safely; the prior baseline-only timing remains 3,001.854 seconds. The security run held after 345.392 seconds: 184 AV-clean entries, one incomplete engine result, and 66,828 entries not evaluated. No downstream metadata worker, reusable clean evidence, permit or import was authorized. This is not a completed layered performance result and cannot establish overhead against the baseline.
A bounded isolated rescan of only the held 2,501,536-byte JPEG reproduced a broken-format heuristic. Its fixed upstream reason concerns JFIF marker placement. The existing bounded structural probe passed, which does not establish AV completion or that the file is safe; the hold remains. Investigation is limited to that member before any expensive full-collection retry, with original bytes unchanged. No AV alert or coverage requirement has been disabled. All real-run reader/controller/network cleanup was confirmed and Windows sharing handles/aliases released. The small private findings volume and local reports remain intentionally retained; private member names, paths, metadata and engine replies are excluded from this issue.
Next: correlate the selected JPEG header layout with the pinned engine parser and, if justified, a bounded diagnostic-only decode; preserve the hold and improve understandable fixed reason reporting. Do not launch duplicate 155 GB scans or infer a false positive from structural checks alone. Private resume state now records finished-held. #12 remains open and #19 draft; no destination access, deployment, merge or production interruption.
Real-input diagnosis/reporting continuation is committed and pushed as
c6ea848063(c6ea848), clean/synced on the existing issue branch; main remains0a47f4f. PR #19 body now reflects the completed baseline and security hold rather than an active comparison. Sanitized machine-readable observations and limitations are versioned in docs/benchmarks/private-input-compatibility-20261028.json and docs/security-measurements.md.The selected JPEG's JFIF header is in segment 5; the pinned engine reports a broken-format placement heuristic. An isolated, bounded diagnostic decode of only that held member passed, as did the existing structural probe. A tiny generated JPEG reproduces the same real-engine disagreement without private content. This does NOT establish a clean scan, safe file or false positive: the actual collection remains held with 184 AV-clean / 1 held / 66,828 not evaluated. No full-collection retry or fallback occurred. The baseline-only 3,001.854 s and early-held 345.392 s are not a completed performance pair. No 15% claim is possible. Originals stayed unchanged; all reader/container/network cleanup was confirmed. A plain private local review page identifies the held member for the owner, while shared data excludes identities, metadata, paths, hashes and raw engine replies.
Fixed SCAN_FORMAT / SCAN_LIMIT / SCAN_ENCRYPTED categories now preserve understandable hold reasons through the broker and ordinary failure summary. Per-file outcomes stay incomplete/held with unvisited members retained and no successful receipts/authorization. Generated structural/decode tests, journal accounting and broker no-downstream tests cover that distinction. Diagnostic-only decoding is limited to a selected held member, 16 MiB encoded / 40 million pixels, in a resource-limited offline worker; no normal-validation decode or production scanning policy was added/weakened.
Validation: final full Linux AMD64 backend tests/vet and focused API/backend/runner races; real offline engine normal JPEG / generated misplaced JFIF / harmless detection / size / recursion / 640 MiB member cases; targeted managed-definition reproduction; three Windows read-sharing/cleanup tests; nine CI policy tests; document/source/workflow and staged redacted secret checks. Source policy: 59 production files / 12,892 lines / 51 existing reviewed findings, no loading errors/suppressions, unchanged baseline and passing govulncheck policy. No browser code changed. Native image/ARM64/release gates and existing native findings remain open.
Resume here: no real benchmark/controller is active. Private resume state is finished-held; do not launch another 155 GB scan merely to hit the same member. Resolve format-policy compatibility first while retaining explicit incomplete coverage and the original bytes; decide safe handling/quarantine within the existing isolation model rather than treating decode success as AV success. Then finish real oversized-media coverage and repeated full pairs/precise target-host accounting, destination preview/import regression and native release work. Private small findings and isolated test definitions remain retained locally for investigation. #12 open / #19 draft; no merge, deployment, production interruption or destination access.
Owner approved the next-step plan in the active conversation. Continuing #12/PR #19 under #14/comment 332 from
c6ea848; latest main fetched/rebase checked unchanged at0a47f4f, clean starting tree, baseline validation running before implementation. ClamAV 1.5.4 and upstream main retain the same JFIF placement heuristic, so no upgrade-only fix is established. Preserve the pinned engine alert and original media; no claim that decode success establishes clean AV.Proposed bounded assessment change: internal/malware/scanner.go may finish shared-stream integrity checks and assess later members after recognized format/encryption/engine-limit findings, with a fixed maximum of 128 such findings and unchanged global worker/concurrency/CPU/memory/operation bounds. Engine transport/read errors, timeouts, integrity failures and detections remain terminal. No retries, rescan fallback, extraction or container-per-file operation. internal/malware/journal.go must preserve held AV decisions after a successful CRC drain instead of overwriting them as allowed; later CRC failures must override integrity/decision accurately. Any held member withholds collection receipts, reusable clean evidence and all downstream processing. A finding-limit stop preserves remaining members as not evaluated. This is logical private quarantine in the existing sealed upload/report system, not copying or automatically omitting files from import.
Extend existing scanner/journal/broker/API tests for mixed outcomes, draining/CRC, later detections/outage/cancellation/storage errors, the cap and authorization refusal; document exact semantics and nontechnical summaries. After bounded generated and targeted tests pass, use the authorized RO archive harness for a single new assessment to discover later compatibility/oversized-media outcomes, rather than repeat a known fail-fast test. Remaining complete performance pairs, precise accounting and destination/native release gates are not waived. No deployment, merge, production interruption or destination access is authorized by this continuation.
Bounded assessment is published as
5098dd6472(5098dd6), clean/synced; main remains0a47f4f. Recognized format/encryption/non-time engine-limit warnings retain per-file holds while the shared stream finishes integrity and later files are assessed. Assessment stops at finding 128; time exhaustion, unexpected replies, outages/read failures, detections and archive/file-policy failures remain terminal. The journal preserves AV holds after verified ZIP integrity. Later CRC/expansion/archive-layout errors cannot be hidden by an earlier warning. No held collection exposes successful receipts/evidence or downstream authorization. No scanner alerts, dependency pins, mounts or global limits were weakened.Passing checks: full Linux AMD64 backend/vet; focused API/backend/runner races; mixed-outcome/directory/corrupt-suffix/expansion/later-layout/detection/outage/cancellation/time-limit/storage-loss/cap/forged-clean coverage; real engine normal/generated misplaced-JFIF/detection/size/recursion/large-member cases; current managed-signature mixed held/clean fixture under normal offline unprivileged 1 CPU / 5 GiB limits (35.40 s); canonical production scanner build; three Windows input-lock tests; nine CI policy tests; source/document/workflow contracts and redacted staged secret scan. Source policy: 59 production files / 12,951 lines / 51 existing reviewed findings, no loading errors/suppressions, unchanged baseline and passing govulncheck policy. The unchanged importer download stalled during baseline rebuild; an existing local toolchain/importer image provided the full offline baseline without skips or pin changes. Native CI for this commit is queued: AMD64/ARM64 source jobs waiting for runners, downstream/release jobs waiting on prerequisites.
The authorized single follow-up real-data assessment has now finished held in 505.509 seconds. It progressed beyond the original JPEG: 294 evaluated, 292 allowed/AV-clean, 1 retained format hold, 1 terminal TYPE_MISMATCH block and 66,719 not evaluated. No metadata worker or import authorization. Original archives unchanged; all scoped reader/controller/network cleanup confirmed and sharing handles/aliases released. The original JPEG warning was not relabeled clean. Private inventory/joined terminal finding/review data remains outside Git and in its small retained test findings volume. Test definitions were refreshed and confirmed current: daily 28167/main 63/bytecode 339. No real benchmark/controller remains active; private resume marker says finished-held.
Next targeted diagnosis: a 5,447,496-byte member declares an AVI extension while bounded detection identifies video/mp4. That does not yet prove a valid MP4, so inspect only that selected member under existing limits before changing policy. Consider narrowly justified supported-container aliases only with actual structural and AV verification; never accept an executable/unsupported object by renaming. Do not repeat all 155 GB merely to discover the same mismatch. This remains an incomplete compatibility assessment, not a full performance pair; first main-only baseline was 3,001.854 s and no overhead/15% result exists. Native/image/destination/performance acceptance remains open. No destination access, deployment, merge or production interruption.
Selected-member diagnosis confirmed the AVI-named member is a bounded-probe-supported MP4 container and received a clean AV reply under current managed definitions. Originals unchanged, isolated reader cleaned up. Its old TYPE_MISMATCH decision remains in the historical run; no retroactive authorization is issued.
Implementing a narrow AVI-name / MP4-content alias requiring BOTH normal eligible AV and the existing bounded ISO structural probe. The probe drives the same stream into AV through one backpressured pipe and one joined goroutine, with the existing entry deadline; no second decompression, full-file buffer/copy, decode, extractor, new service or container-per-file path. Executable/unsupported/malformed AVI-name content stays blocked. Oversized candidates retain strict rejection or explicit acknowledged partial coverage and required structural checks. All JPEG format holds remain in force.
The new observed file-policy stop also justifies extending assessment to conclusive small per-file rejections while preserving their blocked/held decision. Proposed conservative bounds: at most 16 MiB declared per rejected member and 64 MiB total worst-case rejection draining, sharing the existing 128-finding cap. Larger rejects, exhausted budgets, unsafe ZIP/layout/read/integrity failures, time exhaustion, unknown engine replies, outages and detections stop assessment. This avoids repeatedly hashing a 155 GB collection just to discover the next small rejected member, without decompressing arbitrarily large already-rejected objects. No collection containing any such finding obtains receipts, reusable successful evidence or an import permit. Add mixed/budget/source-error/cancellation/early-engine-reply/race tests before another full assessment.
Policy 4 is committed/pushed as
723d9f3073(723d9f3), clean/synced, main unchanged at0a47f4f. It narrowly supports AVI-named MP4 only with existing bounded ISO structure and complete eligible AV on the SAME decompression stream, via one backpressured pipe and joined AV goroutine. Oversized candidates retain strict/acknowledged-partial rules. Renamed executables, fake headers, malformed/unsupported containers, early clean replies and source/AV failures cannot acquire clean coverage. The real selected 5,447,496-byte candidate passed the new shared pipeline under managed signatures, originals unchanged. No retrospective authorization of the older blocked run.Conclusive small file-policy rejections now retain their block/hold while finishing integrity and assessing later members, capped at 16 MiB declared per rejected entry / 64 MiB aggregate conservative draining, sharing the existing 128-finding stop. Larger rejects or exhausted budgets stop. Unsafe ZIP paths/layout, actual integrity/read failures, unknown AV replies, outages, time exhaustion and detections remain terminal. All JPEG holds remain. Any held/rejected member withholds collection receipts, successful reusable evidence and permits. Policy version is bumped to 4 as required for eligibility changes: authentic signed policy-3 evidence is rejected and needs explicit fresh analysis; historical findings remain readable. No automatic retry, full-size copy, extraction, decoder/service, worker privilege or alert bypass was added.
Validation: full Linux AMD64 backend/vet and focused API/backend/runner races; source CRC/read/pipe errors, cancellation/outage/early reply, early/late container metadata, fake/executable/fragmented/oversized aliases and rejection-drain budgets; older signed-policy refusal. Real offline engine and fresh managed-signature tests pass (managed mixed/alias fixture 21.71 s). The actual required-mode Docker broker/scanner assessed 4 synthetic files as 1 blocked / 1 held / 2 allowed and issued NO evidence, permit or metadata output; existing kernel isolation, partial-consent/reuse, mutation/replay checks also pass (32.61 s). First local controller attempt omitted its required RO definitions view and failed closed; corrected complete setup passed with worker policy unchanged. Actual pinned importer preview/upload of a generated AVI-named MP4 passed against a disposable synthetic destination, not a real Immich server or private-media upload. Canonical production scanner built. Source policies: 60 files / 13,056 lines / 51 existing reviewed findings; no loading errors/suppressions, unchanged baseline and passing govulncheck. Three Windows lock tests, nine CI policy tests, source/docs/workflow and redacted staged-secret checks pass. No browser code changed.
A single new policy-4 real-input assessment is NOW ACTIVE through the RO harness, using the same 3 ZIP parts / 155 GB / 67,013 files and current test definitions. Private resume marker records this revision and the existing controller; do not launch a duplicate. Final outcome/coverage/cleanup is not yet known. Until terminal cleanup, retain original Windows sharing handles/opaque aliases. Prior
5098dd6assessment stopped safely at 294 files (292 allowed / 1 held / 1 mismatch block / 66,719 unvisited) in 505.509 s; its sanitized historical data is versioned separately. No full performance pair, overhead/15% result or precise target-host resource claim exists. Original JPEG compatibility/review, complete oversized coverage, destination/native/image/release gates remain open; native CI has been waiting for runners. No deployment, merge, production interruption or real destination access.The policy-4 real assessment at
723d9f3finished held in 398.780 seconds: 1,637 evaluated, 1,628 allowed/AV-clean, 1 held JPEG, 8 type-policy blocks, and 65,376 not evaluated. It stopped at a 36,022,136-byte noncatalog-named member under the 16 MiB per-rejected-entry drain limit; the fixed limit behaved as intended. No metadata worker or import authority. Originals unchanged; all scoped readers/controllers/networks cleaned up and read-sharing aliases/handles released. No real assessment remains active. Private resume state says finished-held and a local owner review page lists the nine findings; identities stay out of this issue.A single bounded diagnostic worker reused one daemon to inspect all nine already-recorded findings (16-target / 64 MiB each / 256 MiB total / four-minute bounds). The original JPEG still has incomplete AV coverage despite structural evidence. The other eight received clean AV replies and passed applicable bounded structure on unchanged bytes. Diagnosis grants no receipts/permission and does not relabel the historical blocked run. No sample/metadata was sent to a third-party scanner.
Five observed mismatches are between supported names within the same image/video family (four MP4 containers under a different supported video suffix, one JPEG under another supported image suffix). Extending the existing shared-stream proof requirement to known same-family aliases only, with another policy version bump and old-authority invalidation. Unknown/executable names, cross-family aliases and formats without an implemented probe remain blocked. Three noncatalog-name cases remain blocked; owner clarification about whether those are expected importable media or intentionally unsupported/test inputs has been requested privately. Do not broadly admit unknown filenames or weaken validation to finish a benchmark.
Stage observations for this incomplete run: compressed hashing 301.980 s; AV 62.111 s; selected structure 28.591 s; inventory 0.728 s; engine setup 8.027 s. AV/structure/shared I/O overlap, so these cannot be added or treated as CPU cost. There is still no complete layered analysis pair, 15% result or target-host throughput claim. Next: finish known-alias tests/docs/publication, retain the custom-type blocks pending clarification, then choose a justified next assessment rather than repeatedly hashing all 155 GB to hit the same known blocker. Native/release/destination work remains open; no deployment, merge, production interruption or real destination access.
Published
be5dd3e367(be5dd3e), clean/synced in draft PR #19. Fetched/rebase checked; main remains0a47f4f. Policy 5 extends mandatory shared-stream structural proof to selected aliases under already-supported image/video names within the same detected family. Eligible AV still consumes every byte. Unknown/executable/script filenames, cross-family changes and formats without a bounded probe stay blocked. No decoder/service, second decompression, full temporary copy, raised rejection budget, AV alert bypass or new worker privilege. Authentic older-policy evidence cannot authorize execution; fresh analysis remains explicit.Validation passed: full Linux AMD64 backend/vet; focused API/backend/runner race checks; malformed/truncated aliases, exact complete AV bytes, held/flagged results, unsafe names and previous evidence/stream/budget regressions; real offline ClamAV and fresh managed-signature fixtures (29.29 s). Actual required-mode Docker isolation/authority/partial-consent/reuse/mutation/replay checks passed (34.80 s), including blocked/held inputs receiving no metadata output or authority. Actual pinned importer preview/upload tests pass for generated AVI/MPG-named MP4 and PNG-named JPEG against a disposable synthetic destination. Canonical production scanner builds. Source policies pass: 60 files / 13,082 lines / 51 existing reviewed findings, no loading errors/suppressions, baseline unchanged. Source/docs/workflow and redacted staged-secret checks pass. No browser code changed. Native CI on this SHA has 13 pending contexts: both source architectures waiting to run, subsequent jobs blocked by required conditions. Those gates have not passed or been waived.
Historical real result is now versioned in docs/benchmarks/private-input-policy4-20261028.json:
723d9f3assessed 1,637 of 67,013 files in 398.780 s (1,628 allowed, 1 JPEG held, 8 blocks, 65,376 not evaluated), stopping at a 36,022,136-byte unsupported-name member under the unchanged 16 MiB rejection-drain limit. One bounded batch diagnosed the nine recorded findings: eight clean AV replies, nine structural passes, JPEG still incomplete. Diagnostic results grant no authority. Originals unchanged; scoped readers/controllers/networks removed; aliases/locks released. No full real assessment is active. No real destination access, merge, deploy or production interruption.Resume next:
Private review/report/resume files remain outside Git. Ignored local helpers include issue12-private-review.py, issue12-diagnose-batch.py and policy4 progress tooling. Select the finished-held run by revision
723d9f3, not merely newest directory. Do not launch duplicate readers. Detailed design, measurement limitations and validation are in docs/security-measurements.md, docs/malware-security-review.md and docs/private-archive-testing.md. #12/#14 stay open; PR #19 stays draft.Owner clarified in chat that the archives were directly downloaded from Google Takeout and they do not know the unusual formats. The prior source/test-fixture clarification is resolved; owner format identification is no longer a prerequisite. This is a real compatibility case to investigate, without treating Takeout origin as a safety guarantee.
Checked Google's primary documentation: https://support.google.com/photos/answer/3024190 documents embedded original media metadata, secondary JSON for additional Photos metadata and archive_browser.html format information. https://support.google.com/photos/answer/6193313 documents upload formats; it is not a complete Takeout filename specification. No official explanation for the two observed noncatalog suffixes was found. Generic third-party extension directories and unrelated similarly named tools are not evidence about these bytes.
A new bounded local context audit found 32 entries under the two suffixes, all within Google Photos folders, each with an exactly named JSON sidecar whose title matches its basename. All contain photo-time and Google Photos origin fields. Prefix candidates: one PNG and 31 ISO media files. This supports genuine export-content compatibility, not complete format/integrity/AV validation. Only THREE of these 32 entries were in the previous full-byte diagnostic; do not extend those clean AV/structure results to the other 29. Sidecars are untrusted metadata, never authorization evidence.
Read just 150,745 bytes of selected prefixes/sidecars in one offline non-root RO-input worker: 0.5 CPU / 256 MiB / 32 processes, 60-second internal and 90-second outer bounds, capped targets and reads, no socket/credentials/capabilities, extraction, full hashing or media decoding. Originals unchanged; exact reader removal confirmed and sharing locks/aliases released. Private diagnostic JSON stays outside Git. No full assessment remains active and no authority was issued.
Documentation/checkpoint published at
ed5563333c, clean/synced, main unchanged at0a47f4f. Documentation/source/workflow contracts and redacted staged-secret checks pass; product source/policy 5 is unchanged from testedbe5dd3e. No claim of new backend/engine/native CI passes from this documentation-only commit. Detailed bounds/results in docs/malware-security-review.md. PR #19 remains draft. No merge/deploy/production interruption or real destination access.Next: develop/test bounded auxiliary and noncatalog media admission AND explicit importer handling. The pinned importer does not import arbitrary suffixes, so silently admitting them could still lose expected media. Preserve current holds until that policy is implemented; do not add a blanket unknown-extension fallback, automatic renaming, raised drain budgets or AV bypass. Existing JPEG compatibility, complete real-library/performance/destination/native release acceptance remains outstanding. The prior pending owner-format question in checkpoint 474 is superseded by this evidence and clarification.
Starting owner-authorized Takeout compatibility increment after clean fetch/rebase on unchanged main
0a47f4f, existing product baselinebe5dd3e. Scope: narrowly admit the two observed noncatalog suffix/content pairs only with existing full-stream PNG/ISO structure PLUS complete eligible AV; treat these as non-exempt content so oversized AV exceptions cannot be acquired through these unfamiliar names. Add bounded AV/ISO handling for the pinned importer's recognized motion-photo companion suffix. No arbitrary-extension fallback, metadata-as-authority, higher rejection budgets or AV bypass.Import support is separate: the pinned importer currently records the two observed suffixes as non-assets, absent from its asset/skip totals. Add a narrow reviewed hook that records them as unsupported discarded items in its existing aggregate/private affected-item reports. Preview/import must show explicit unsupported-file warnings, preserve original names/ZIPs, and avoid a complete-success presentation when files were left out. This increment will NOT convert/rename them or claim they were imported. The existing supported-media preview/import remains intact; originals are retained for later format recovery. Increment the importer modification version and matching source/notice/build artifacts together.
Validation plan: establish cached baseline; positive exact same-byte stream tests plus wrong-content/truncated/malformed/detection/outage/strict-and-partial size cases; older signed-policy rejection; actual pinned importer generated mixed supported/unsupported multipart preview/import against disposable synthetic destination with private identities preserved and no raw aggregate leaks. Retest isolated required-mode scanner/managed signatures, source policies and changed importer source packaging. Then bounded real-member confirmation, not another full 155 GB hash pass to hit known blockers. Full-library/JPEG/native/release/performance acceptance remains open. No real destination access or deployment/merge/production interruption.
Published
b9444fce7f(b9444fc), clean/synced in draft PR #19. Main fetched/rebase checked before implementation and publication, unchanged at0a47f4f. PR #19 comment 332 (the owner's full 14,403-character architecture write-up) was already incorporated and was rechecked on this turn; it remains controlling scope alongside #14 and subsequent owner steering.Policy 6 narrowly admits the two observed noncatalog suffix/content pairs and the known .mp motion-photo companion name only with complete eligible AV plus bounded PNG/ISO structure on the SAME decompression stream. These names are non-exempt: neither strict nor allow-large-media can grant them oversized AV exceptions. Other unknown names, wrong content and malformed/truncated candidates remain held/blocked. No arbitrary-extension fallback, metadata-as-authority, AV alert bypass, raised rejection/finding budgets, extra service, ordinary decoder, extraction, whole-file copy or new worker privilege. Older signed-policy authority is invalidated; fresh analysis remains explicit.
Importer behavior is now truthful: modified upstream 0.32.0-bridge.5 records the two unfamiliar filename cases as UNSUPPORTED_FORMAT final discards before its previous non-asset fast path. Existing numeric totals and full private affected-item reporting retain them. A mixed preview/import returns IMPORT_UNSUPPORTED_FILES with a plain files-left-out warning. The same-options preview and existing explicit confirmation still permit supported files; logged errors and process/auth/network/item failures take priority and cannot qualify as a successful warning preview. This increment does NOT add upload support for those filenames, rename/convert files, rewrite ZIPs or silently claim their photos were imported. Native media, sidecars and intentional motion-companion import behavior are unchanged. Keep original exports for recovery.
Checks passed: full Linux AMD64 backend/vet; focused API/backend/runner races, with an additional malware race run after diagnostic-only reason reporting changed; exact complete same-byte AV/structure, bad/truncated/wrong-type content, detection/incomplete/outage and both size modes refusing exemption; older signed-policy refusal. Actual modified importer multipart preview AND import against a disposable synthetic destination process one supported photo, count both unfamiliar files as skipped unsupported, preserve exact private names, upload only the supported photo, and keep aggregate output free of those names. Failure-priority tests cover incomplete processing/auth/network/process failure and logged ERROR overriding the unsupported warning. UI build and focused advanced-import browser suite pass, including explicit skip warning/counts, default-closed technical controls, unchecked confirmation, matching preview, reload/busy/offline/cross-session protection and 320px layout. The new warning screenshot was visually reviewed. An initially incomplete browser mock was corrected to the real diagnostics envelope and a selector was scoped away from the duplicate technical-report text; no product behavior was weakened.
Real offline engine fixtures pass; fresh managed-signature fixtures including the new name cases pass (27.37 s). Actual required-mode Docker isolation/authority/partial-consent/reuse/mutation/replay checks pass (35.92 s) with canonical policy-6 scanner and processor images. Both canonical images build, processor runtime reports importer bridge.5. Source policies: 60 production files / 13,100 lines / 51 existing reviewed findings, no loading errors/suppressions and unchanged baseline. Modified importer binary vulnerability policy passes. Matching-source checksum, original license, vendored/build/notice material, added helper/tests and actual walk/report hooks verified; CI now enforces the added matching source. Nine CI policy tests, source/docs/workflow contracts and redacted staged-secret checks pass. A formatting guard caught an unformatted newly added test during an intermediate local controller build; corrected final controller/full checks pass. These local checks do not waive native/image/release gates or existing image findings.
Bounded private real-member results: initial batch plus one reason-reporting refinement/recheck each selected 13 members / 55,146,069 declared bytes (nine earlier findings and four motion companions), one daemon per batch under unchanged 16-target / 64 MiB each / 256 MiB total / four-minute bounds. All 13 classify. Eight return clean AV and pass structure, INCLUDING the three previously blocked unfamiliar-name files. The original JPEG still has an incomplete AV heuristic reply despite passing structure. Four motion companions fail the existing structural probe with MEDIA_STRUCTURE_INVALID; their producer closes the shared pipe, so AV does NOT complete for those four. They are not four additional AV heuristic findings. This is a conservative compatibility hold, not proof the clips are corrupt. No structure or AV result was bypassed to get a green run. Diagnostic results issue no authority and cannot retroactively authorize historical held runs or the other 29 unfamiliar-name files.
Originals unchanged; exact scoped readers removed and sharing aliases/locks released after both batches. Raw results remain outside Git in the existing finished-held policy-4 run folder, selected by revision
723d9f3, not newest directory. Private filenames include policy6-layout-diagnostic-private.json/log, with older batch results preserved. Ignored local helpers: issue12-policy6-layout-batch.py and issue12-policy6-private-batch.py; test image immich-bridge-scanner:issue12-policy6-layout-tests adds only diagnostic reason reporting to the tested production source. Do not start duplicate readers. No full 155 GB assessment is active.Next: diagnose the four companion layouts using bounded numeric/topology evidence (for example empty-track/sample-table or addressing behavior), then add justified structural compatibility and malformed equivalents rather than dropping checks; preserve the original JPEG hold. Broader native-format handling, all 32 unfamiliar-name entries, complete oversized/full-library coverage, repeated main/layered timing pairs with target-host resources, full security-enabled destination regression and native AMD64/ARM64 image/security/release gates remain open. No complete overhead/15% result. Latest native CI has pending contexts; no runner/release gate override. No real destination access, production interruption, merge or deployment. #12/#14 and PR #19 remain open/draft. Detailed validation and limitations: docs/malware-security-review.md, docs/file-validation-policy.md, docs/results.md and docs/importer-build.md.
Starting the owner-approved next #12 increment from
b9444fcafter clean fetch/rebase on unchanged main0a47f4f. Scope first: bounded diagnosis of the four motion-photo companion structure holds, using numeric check/topology evidence in one offline read-only worker, without new full archive hashing or normal media decoding. Preserve JPEG and companion holds. Keep diagnostic instrumentation outside product source; only add a justified structural compatibility change after identifying the rejected invariant and testing malformed equivalents. Existing size/AV/structure/finding/drain/concurrency/resource/authorization limits stay intact. Cached baseline validation is being established before product edits. No real destination access, deployment, merge or production interruption; #14 and PR #19 comment 332 still govern.Bounded numeric diagnosis narrowed all four companion holds to the exact same timing-table invariant: one sample in the final timing run has delta 0, after every earlier sample was accounted for. Complete selected bytes were read; no media payload/identity was published. The blanket delta!=0 rule is too strict for this terminal case. Primary FFmpeg maintainer discussion quotes the format requirement allowing a zero duration for the last sample (https://ffmpeg.org/pipermail/ffmpeg-devel/2021-November/288184.html); Android's writer explicitly emits a one-sample final duration entry (https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/MPEG4Writer.cpp). Google origin is still not safety evidence.
Implementing a narrow structural correction: only a video track's single FINAL sample may have zero delta, following at least one positive-duration sample, with exact complete sample accounting. Earlier/multiple/all-zero timing, zero-count runs, inconsistent sample totals, unsafe references/chunk bounds and unsupported layouts stay rejected. No timestamp repair or byte rewriting; AV still must complete for these non-exempt names, and a new policy version will invalidate older authority. Test generated positive/malformed counterparts and all four real candidates through the unchanged bounded AV pipeline before any import claim. Baseline backend/vet/races passed; main remains unchanged. JPEG hold, importer unsupported-name reporting and all resource/budget/permission boundaries remain intact.
Published
d588fc4d06(d588fc4), clean/synced in draft PR #19. Clean fetch/rebase and cached backend/vet/race baseline preceded product edits; main remains0a47f4f. #14 and the full PR #19 comment 332 architecture steering still govern. No other issue work or deployment occurred.The four motion-companion holds are resolved in the selected full-byte diagnostic. The blanket zero-delta rejection was too strict: the format allows the final sample's delta to be zero, including singleton non-video tracks. An initial video-only correction exposed the subsequent valid singleton-track and Android metadata layouts; it was not published as a successful solution. Final policy 7 permits zero only in a ONE-sample FINAL timing run, with exact sample totals, and requires positive-duration video somewhere in the complete movie. Earlier/multiple zeros, zero-count rows, inconsistent tables and movies without positive video duration still fail. This supersedes the more restrictive video-only proposal in comment 495. No timestamp repair or byte rewrite.
Android's documented mett writer uses one application MIME C string or two identical strings where normal sample-entry fields would be. Recognition is restricted to meta/mett, one description and one ALREADY VALIDATED self-contained data reference, terminated application subtype tokens up to 255 bytes, and at most 512 description bytes. No arbitrary reference inference, external data, malformed/mixed/trailing strings, unsupported handler or encrypted-layout bypass. Normal codec, timing, sample/chunk address/overlap and all count/memory/deadline checks remain. MIME text is format recognition, never safety evidence or a location to fetch. Sources: https://ffmpeg.org/pipermail/ffmpeg-devel/2021-November/288184.html and https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/MPEG4Writer.cpp (writeMetadataFourCCBox and final sample entry).
Policy version bumped to 7; older authentic signed authority requires explicit fresh analysis. No automatic rescan or retrospective authorization. Full eligible AV still consumes original bytes on the same backpressured pipeline, not normalized bytes. Companion/unfamiliar filename cases remain non-exempt in BOTH size modes. Importer bridge.5 and its explicit unsupported-file reporting are unchanged; this does not import/rename unfamiliar filenames. No dependency/service, ordinary decoding, extraction/temp copy, added decompression, new configuration/privilege, relaxed alert, raised rejection/finding budget or extra normal worker.
Final selected real check: same 13 members / 55,146,069 declared bytes, all 13 structurally pass, 12 return clean AV, zero probe-terminated AV checks. ALL FOUR former companion holds now pass full AV plus structure. The original JPEG remains the sole incomplete AV heuristic reply despite passing structure. No engine warning was converted to clean. Original ZIPs unchanged, exact readers removed and Windows sharing aliases/locks released. The diagnosis issues no receipts/evidence/permits and establishes nothing about the other 29 unfamiliar-name entries, every companion or the entire library. No full 155 GB assessment is active.
Checks passed: full Linux AMD64 backend/vet and focused API/backend/runner races; generated early/late metadata and both track orders; standard/legacy/doubled metadata layouts; complete same-byte AV with detection/incomplete/outage still held; malformed timing/counts including exact three-sample cases; external/bad refs, chunk ranges, wrong handlers/types/description counts, MIME termination/mismatch/trailing bytes, 255/512 bounds, absence of positive video; strict/non-exempt and acknowledged native partial-size policy; authentic older-policy refusal. Additional final bound/positive-video tests pass a focused malware race run. Real offline engine fixtures pass; fresh managed-signature motion/Android fixtures pass (30.59 s). Actual required-mode Docker isolation/held+blocked authority/partial-consent/reuse/mutation/replay passes with canonical policy-7 scanner/processor (33.80 s); both canonical images build. Source policy: 60 production files / 13,130 lines / 51 existing reviewed findings, no loading errors/suppressions, unchanged baseline and passing govulncheck. Source/docs/workflow and redacted staged-secret checks pass. No new UI/importer-package/image-advisory check is claimed; unchanged prior evidence stays in its prior checkpoint. Latest native CI has 13 pending contexts (source jobs waiting, downstream gates blocked), not passed or waived.
Private resume: select the finished-held run by revision
723d9f3, not newest directory. policy7-metadata-diagnostic-private.json/log is the FINAL successful selected companion check; older policy7-diagnostic-private records the earlier insufficient video-only attempt. Numeric trace iterations are in motion-*-trace-private files. Raw identities/results remain outside Git. Ignored helpers: temp/issue12-motion-trace/ (diagnostic-only instrumented copies, NEVER ship) and temp/issue12-policy7-metadata-batch.py; final test scanner tag immich-bridge-scanner:issue12-policy7-tests. The selected-member limits remain 16 targets / 64 MiB member / 256 MiB total / four minutes. Numeric trace readers used 0.5 CPU / 512 MiB / 32 processes / 60 seconds; AV retained normal offline non-root one-CPU scanner limits. All readers cleaned; no active job or duplicate reader to resume.Next: bounded classification/coverage of remaining native/auxiliary-format entries and wider real-member validation, then a justified complete assessment while preserving the JPEG hold. The metadata-only name catalog still has other unvalidated formats; keep its private histogram/identities out of public issues and do not broadly admit arbitrary names or rerun 155 GB hashing just to hit known blockers. Resolve the JPEG coverage/recovery workflow without disabling alerts or relabeling incomplete AV. Complete real oversized/full-library coverage, repeated main/layered performance/resource pairs, full security-enabled destination regression and native AMD64/ARM64 image/security/release gates. No complete overhead/15% result. #12/#14 remain open and PR #19 remains draft; no real destination access, production interruption, merge or deployment. Exact rules/checks are versioned in docs/malware-security-review.md and docs/file-validation-policy.md.
Starting the next owner-approved #12 increment from clean
d588fc4, fetched/rebased on unchanged main0a47f4f. First perform a bounded prefix audit of remaining noncatalog native/auxiliary entries using the saved inventory, read-only original ZIP aliases and one offline unprivileged reader. No new full-library hashing, extraction, decoding or repeated full assessment to reproduce a known blocker. Establish cached backend/vet/race baseline before product edits. Use actual detected formats, applicable primary format/importer sources, complete eligible AV and explicit importer behavior to choose narrow support. Preserve JPEG holds, all size/worker/finding/drain/authorization boundaries, and private identities. No blanket arbitrary-extension admission, silent skip/rename, extra service or real destination/deploy/merge/production interruption. PR #19 comment 332 and #14 continue to govern.Owner steering added in chat: AFTER scanning is actually complete, ready and validated, add security-scan configurability to the administrator panel. This is explicitly a later phase; do not build the panel now or use configuration work to bypass unresolved scanning acceptance.
Required future scope includes whether scanning runs, security posture, blocking versus reporting, and other useful administrator tuneables. Candidate controls to assess after readiness: ordinary/oversized coverage policy and thresholds; scan/entry timeouts; bounded concurrency/CPU/memory budgets; signature update schedule and freshness; findings/diagnostic retention and reporting. These are planning candidates, not approved arbitrary command/image/socket/network controls. Design typed, validated settings with safe defaults, clear effective-policy/status feedback and auditability. Policy changes must invalidate affected reusable evidence/permits; reporting mode must accurately show detections/incomplete coverage rather than claim files clean. Separate authorization and resource boundaries remain enforced.
Sequence remains: finish and validate the scanning engine, compatibility/security/error/worker isolation, representative large-input performance and required readiness gates first; then scope and implement admin controls with the existing admin settings architecture. This instruction supersedes the earlier no-new-security-configuration-screen scope only for that later phase. No new issue/chat/automation, deployment or current admin-panel implementation is requested by this checkpoint. Current bounded native/auxiliary-format work continues.
The bounded signature audit identified 41 remaining entries (only 167,936 prefix bytes read) as native icons, JPEG-based MPO images, MP4 under other video names, and one large BMP case. Prefix identification is not safety/CRC/AV proof; originals unchanged and reader cleanup confirmed. Current Immich source advertises MPO but not the other observed suffixes, and the pinned importer uses the connected server's media catalog.
Implementing policy 8 scope: non-exempt native ICO and JPEG-based MPO recognition (ICO directory sanity remains within the same 4096-byte sample), and complete AV+existing ISO structure for the two observed video-name/MP4 pairs. No arbitrary-extension fallback or oversized exemption for any of these cases. Include MPO as a preliminary filename candidate while keeping destination/importer capability authoritative. Extend the existing importer discard hook so observed unimportable categories cannot disappear from skip totals; MPO stays importable only if advertised by the server. No renaming/conversion or additional security configuration UI.
The large BMP case remains unsupported/held: its prefix alone cannot establish a coherent complete format, and its internal declared file length does not match the ZIP member. Do not broadly admit it, ignore that finding, or raise production rejected-entry/drain limits for a green result. A separate bounded full-AV diagnosis can inform recovery but grants no authority. Diagnostic batching may accommodate a single member up to 128 MiB while KEEPING its existing 256 MiB aggregate/16-target/four-minute limits, worker resources and ordinary AV ceiling; this is not a production policy or rejection-budget change. Validate ordinary native/alias positives and malformed/detection/outage/size negatives, actual importer advertised/unadvertised behavior and private/aggregate reports, then recheck selected real members. Admin controls explicitly remain deferred per owner steering 506/507.
Published
4163df9ccfto the existing WIP PR #19. Fetched/rebased before work and again after the validated commit; main remains0a47f4f. Working tree is clean. #14 and PR comment 332 continue to govern. This is a completed compatibility increment, not completed scanning/performance/release acceptance.Policy 8 adds non-exempt native ICO/MPO and the two explicit video-name/MP4-content pairs. Native files require complete ordinary AV; ICO directory checks fit the same 4096-byte prefix, MPO does not use a single-picture JPEG probe, and video aliases require existing shared-stream ISO structure plus full AV. Unknown names remain blocked. No production ceiling, rejection-drain/finding budget, partial-coverage exception, worker privilege/resource/concurrency boundary or engine alert is relaxed. Older signed policy evidence requires fresh explicit analysis.
Importer 0.32.0-bridge.6 now reports six explicit unsupported categories instead of losing them as non-assets. Original identities stay private; aggregate counts and plain files-left-out warnings stay truthful. MPO is a preliminary candidate and imports only if the destination advertises it. The actual importer test caught upstream catalog discovery happening after adapter initialization; a narrow Browse-entry refresh clones the map, refreshes only MPO and rebuilds its filename collector before use. Shared defaults and all other catalog entries stay unchanged. Source packaging, notices and CI verify both Browse/walk hooks. No renaming, conversion or ZIP rewriting.
Checks actually passed:
Private real-member result: a 167,936-byte prefix audit selected 41 remaining entries. Three serial full-AV diagnostic batches cover 246,020,330 declared bytes. All 41 receive clean AV replies; 40 meet classification policy, including four ISO structural passes. The large BMP remains unsupported, with its inconsistent internal declared length, despite clean AV. The original JPEG coverage hold also remains; its alert is enabled. No diagnostic result issues receipts/permits or retroactively authorizes earlier held runs. The diagnostic-only member cap is 128 MiB; 16 targets / 256 MiB aggregate / four-minute context and existing worker bounds remain. Production 16 MiB/member / 64 MiB aggregate rejected-entry draining is unchanged. Original ZIP identities/timestamps remain unchanged, exact readers are removed and aliases/locks released. Private raw results and mappings remain outside Git. No full-library assessment is active.
Resume from this commit, read latest #12/#14/PR comments first, and preserve the real-input holds:
No real destination access, deployment, restart, merge, issue closure or production interruption occurred. PR #19 remains WIP and #12/#14 remain open.
Starting the owner-approved continuation from clean
4163df9, fetched/rebased on unchanged main0a47f4f. Cached full backend/vet/focused race baseline passes before edits. Read latest #12/#14 comments and PR #19 comment 332. First complete bounded full-byte checks of the remaining unfamiliar-name files, using saved private inventory, exact member identity and read-only original aliases with the current policy-8 diagnostic image. Determine wider companion scope from bounded numeric inventory rather than launching unbounded work. Keep all existing JPEG/unsupported holds, production budgets and scan alerts; no repeated full-library assessment merely to reach known blockers. Then document controlled recovery that preserves originals and prevents rejected bytes from reaching the importer; implementation must be concrete, tested and consistent with the existing worker/evidence architecture. Admin controls stay deferred until scanning readiness. No real destination, deployment, merge, production interruption or issue closure.Published
c08e677acbto the existing WIP PR #19. Started from clean4163df9, read current #12/#14 comments and PR #19 comment 332, fetched/rebased and established the cached full backend/vet/focused race baseline before edits. Fetched/rebased again after the validated commit; main remains0a47f4fand the working tree is clean. Production code, validation policy 8 and importer bridge.6 are unchanged.Real-input progress: all 29 previously unvalidated unfamiliar-name members now complete full AV and required structure. Another 64 of 424 motion-photo companions were selected deterministically across three archives and the size range, including per-archive minimum/maximum sizes. All pass. Six serial batches checked 93 members / 477,794,438 declared bytes, without changing 16 targets / 128 MiB member / 256 MiB batch / four-minute context or existing offline one-CPU/5 GiB worker bounds. This completes selected validation of the observed unfamiliar-name set when combined with prior checks, but does not validate all companions, all oversized media or the complete library. Original identities/timestamps unchanged; exact readers removed and aliases/locks released. Raw replies/identities remain private. No receipts/permits, full timing pair or 15% result comes from diagnosis. The original JPEG and unsupported large BMP remain held, with alerts and production budgets unchanged.
The actual required-mode Docker test now completes real processor/modified-importer preview AND upload against a disposable internal synthetic destination, for both a generated clean photo and an acknowledged oversized generated video. The destination verifies exact original asset bytes; preview uploads zero assets and each import uploads one. Structured reports have one processed asset and no discards/errors/pending items. Clean coverage stays clean; partial coverage stays partial through verification and execution. Verification creates no processor, and preview/import creates exactly one processor with NO new scanner. Missing permits, unacknowledged partial verification and consumed-permit replay start neither worker. Private logs/item completion survive removal; aggregates exclude filenames/account text/tokens/evidence/permits. Existing actual hold/block, content mutation, isolation, cancellation, orphan and cleanup checks still pass. These are synthetic destination integration checks, not a real-provider or complete app/browser/pilot claim.
Checks passed on the changed test/documentation tree:
Recovery scope decision is pending the owner's answer in chat. The current design explicitly says "ZIP rewriting is out of scope" and blocks an original ZIP set on any held/rejected member. Existing approval to continue scanning work does not itself authorize replacing that boundary. A concrete proposal is now committed in docs/security-recovery-proposal.md: optional separate owner-selected recovery copy, original export/findings preserved, explicit private exclusion list, exact member identities, existing isolated processor/global slot, bounded output capacity/cancellation/cleanup, then fresh COMPLETE analysis, new evidence and matching preview before any import. Raw compressed member copying can avoid decompression/recompression for packaging, but still adds source/copy I/O, approximately another compressed-export-sized storage allocation and a fresh scan. Unvisited entries are not approved; source ledger rows grant no authority. Copy creation is NOT IMPLEMENTED or enabled. No real archive was rewritten, extracted, copied in full or imported. The alternative remains corrected/re-exported archives. Admin controls stay deferred until scanning readiness.
Private resume: retain the finished historical assessment selected by revision
723d9f3, not the newest run folder. New private files are wider-selection-private.json, policy8-wider-batch-1..6-private.json/log and policy8-wider-diagnostic-private.json. Ignored local helpers are temp/issue12-wider-selection.py and temp/issue12-wider-batches.py. Exact test controller images are immich-bridge:issue12-workflow-tests and immich-bridge:issue12-workflow-race, with logs temp/issue12-security-destination-final.log and temp/issue12-security-destination-race.log. They use canonical policy-8 scanner/processor and the retained private definitions volume, with freshness verified by the real code. No full real assessment or reader remains active.Next:
No real destination access, production interruption, deployment, restart, merge or issue closure occurred. #12/#14 stay open and PR #19 stays WIP. Exact evidence/limits are versioned in docs/malware-security-review.md and docs/security-evidence.md.
Owner approved complete-assessment-first with reusable per-file evidence; recovery remains NOT approved and deferred. Starting from clean
c08e677, fetched/rebased on unchanged main0a47f4f, with cached full backend/vet/focused race baseline passing before edits. Implement safe continuation after isolated file rejections/detections and independently bounded integrity failures; skip rejected payload remainder when drain budget would be exceeded, explicitly retaining incomplete integrity rather than weakening checks or wasting decompression. Unsafe ZIP layout/paths, scanner outage/unexpected replies/timeouts, storage/global limits still stop safely. Persist complete per-file statuses and shared-stream content digests, bind the finalized private manifest and exact input/policy/image/definitions to a separate signed assessment capability. A held assessment must never become an execution permit or authorize the original ZIP set. Completed unchanged checks can be authenticated without another AV run; future selective use is separate and no recovery/copy/import bypass is implemented. Update policy/evidence invalidation and test continuation, sticky detections, skipped vs verified integrity, corrupt entries, resource stops, manifest tampering/forgery/scope/expiry/restart and actual required Docker no-rescan refusal. No new service, decoder, ordinary extra extraction, production interruption, real destination or admin UI. Record proof/results/next steps here.Policy-9 foundation published at
5469cc659fin WIP PR #19; main remains0a47f4f. This follows the owner's new approval: complete assessment and authenticated per-file evidence FIRST; recovery remains explicitly unapproved/deferred, and admin controls remain after scanning readiness.The scanner now continues after per-file detections/recognized holds and independently bounded CRC/deflate/length failures. Strict verification still rejects corruption. The 16 MiB/member and 64 MiB aggregate rejected-data drain budgets are unchanged: larger/over-budget rejected members are closed with integrity NOT completed and no content digest, while independent later entries continue. Declared skipped bytes still charge global expansion; actual expansion is capped by member declaration plus one. The old 128-finding stop is removed; existing entry/report/time/worker limits, terminal unsafe layout/path/read/outage/timeout conditions and zero automatic retries remain. Detections are sticky. Any held/rejected original collection still gets no successful execution receipts or permit.
Completed file digests share the existing content stream, with no second decompression. A bounded three-file manifest seal and separate signed assessment bind exact source ZIP hashes/identities, scope/operation/run, completion metadata, policy/limits/images/definitions and expiry. A current owner can explicitly authenticate the saved report without media reads, workers or another AV pass. This does not clear a finding or become an analysis/execution capability. The proof is private in storage/transport; routine queries, JSON, notifications and support omit it. Future selective retry/import/recovery consumers are NOT implemented. The complete-assessment docs explain exact trust and reuse boundaries.
Finished held assessments now say they finished with items needing review and warn that starting again repeats work. Private details distinguish skipped integrity and expose content digests only under technical disclosure. A previously hidden reader bug retaining an inventory's NOT_EVALUATED reason on clean results is fixed and tested.
Validation: full Linux AMD64 backend/vet and focused API/backend/runner races; per-file continuation, more than 128 findings, fixed rejection budgets, skipped-vs-verified integrity, corrupt members, strict refusal, digest/manifest changes, forgery/context/expiry/restart and analysis/permit domain substitution tests; owner-only no-scan API verification and flagged preview refusal. Real offline ClamAV fixtures and fresh managed signatures pass (22.52 s). Final actual required-mode Docker suite with race-instrumented controller passes (56.72 s), including an over-budget rejected file followed by valid members, held-manifest authentication with NO worker start, refusal as execution evidence, clean/partial preview-upload, mutation/replay/isolation/cancellation/cleanup. Source policies pass: 63 production files / 13,622 lines / 51 existing reviewed findings, no loading errors/suppressions or baseline expansion; govulncheck passes. App/runner/scanner/processor images build; canonical runtime uses pinned Node 24 UI build. UI lint/build, private-findings and advanced-import browsers, visual mobile review, nine CI policy tests, source/docs/workflow/privacy and redacted staged-secret checks pass. Importer stays bridge.6; no new importer advisory/source-package or final-image/native release clearance is claimed.
A single full owner-authorized READ-ONLY real-set assessment is now running on this committed revision, policy allow-large-media, one normal bounded scanner, fixed private definitions and no real destination. It uses the existing private benchmark harness with canonical policy-9 scanner/processor and matching tests image. All 67,013 entries are inventoried; no complete result or performance claim exists yet. Originals are held under read-sharing locks until exact reader cleanup; no archive rewriting, full copy, extraction, recovery or import. No other expensive AV worker should be started. The scope remains analysis/assessment only, with original JPEG/BMP holds preserved.
Private resume: the active run is under the dedicated policy9-5469cc6 private workspace outside Git; do not confuse it with the historic assessment selected by
723d9f3. Current chat shell session 25687 owns the Python harness and locks. Do NOT launch a duplicate or release aliases until reader cleanup is confirmed. Bounded progress helper temp/issue12-policy9-progress.py probes the append-only index without reading media or whole reports; counts are provisional, not percentages or final evidence. Final report/proof/findings remain in that run's private folder/scoped volume. Test-specific assessment proof is saved privately and is not production import permission.Next: finish this one full-set run, verify complete/partial statuses, original identity and exact cleanup, review only fixed aggregate findings publicly and private names locally, then record measured wall/resources and remaining limits. A held complete assessment is not a completed import or a successful 15% timing pair. Do not auto-repeat the full run. Native architecture/image/release and broader real-provider/performance acceptance remain open. Recovery and admin configuration must not start from this checkpoint. No deployment, restart, merge, production interruption or issue closure.
Owner explicitly reaffirmed in chat: recovery must NOT start until everything else is complete. This is not recovery approval. Local implementation/security/container/browser checks have passed, but the full policy-9 real-set assessment is still active and performance/native release gates remain outstanding. Continue those scanning checks first, preserve the original holds, do not implement recovery/copy/rename/selective import, and do not ask to start that work while readiness is incomplete. The active single-scanner run and exact resume instructions remain in comment 527; no duplicate AV run or production interruption.
Readiness checkpoint following the owner's reaffirmation in comment 531: everything has NOT passed completely. Recovery remains unapproved and deferred; admin controls still follow scanning readiness. No source changes were made during this continuation; published HEAD remains
5469cc6and the working tree is clean.The original single read-only real-set assessment remains active, without a restart or duplicate scanner. The most recent lightweight index probe found 43,254 saved outcomes out of 67,013 inventoried entries. These are provisional saved decisions, not a count of clean files or final completion. One resource snapshot showed exactly one worker, approximately one CPU core and 1.43 GiB of its 5 GiB memory bound; this is not a peak measurement. The assessment has already taken substantially longer than the historical baseline analysis, so performance readiness is unresolved. No current paired overhead percentage or bottleneck conclusion is claimed before the final timings. The native CI state for this exact HEAD remains 13 pending checks; none was waived, cancelled or replaced with a production runner.
The main Python harness and original-file sharing locks are still owned by shell session 25687. The prior active-run location and scope in comment 527 remain authoritative. Only the auxiliary lightweight polling loop was stopped; the assessment itself was NOT stopped. Do not launch a duplicate, remove source aliases or release locks while its reader is active. The harness has its existing scanner/controller deadlines and scoped cleanup. Check session completion and the finalized private report before treating the run as finished.
A bounded post-run aggregate reviewer is prepared in ignored temp/issue12-policy9-final-review.py. It does nothing before the unique private report is finalized. After the harness exits, it mounts only the retained findings volume read-only, with no media mount/network, independently verifies the framed three-file manifest digest and exact completion metadata, reconciles inventory/index/result ranges and decision totals, checks digest-versus-integrity consistency, and reports only fixed aggregates. It also verifies exact worker/controller/mapping/network removal and released source aliases. The test broker's saved assessment verification is recorded separately and is not production import authority. The helper passed Python compilation; its real finalized-manifest execution remains pending. No private media names, content hashes, capability tokens or host paths are published.
Next: finish the existing pass, run that bounded review, record complete versus unvisited/unfinished checks, preserve every hold/block, confirm unchanged originals and cleanup, then use actual stage/resource evidence for performance work. Broader workflow/real-provider and native architecture/image/security/release acceptance remain separate unfinished gates. No recovery work, admin configuration, real destination access, deployment, restart, merge or closure occurred. #12/#14 remain open and PR #19 stays WIP.
Resumed from comment 532 after the original full-set run ended. Actual result on
5469cc6: security_hold, assessment_complete=false; 47,588 / 67,013 entries have durable results and 19,425 remain unvisited. Totals: 47,271 allowed decisions (47,264 clean AV and seven explicitly unscanned oversized-media exceptions), four blocked and 313 format-related holds; zero automatic retries. The terminal aggregate reason is ZIP_INTEGRITY and must be diagnosed before attributing it to corruption, time exhaustion or a particular member. No downstream processor or import ran.The bounded independent report reviewer now passes on the finalized run: framed manifest seal and exact completion match; inventory/index/result ranges and counts reconcile; every allowed file has a completed content digest, and no incompletely integrity-checked file has one. The test broker authenticated assessment-only evidence; this remains diagnostic proof, not production permission. Originals unchanged, exact worker/controller/mapping/network cleanup confirmed and aliases/locks released. No real reader is still active. Raw identities, hashes and capabilities remain private.
Measured security run: 10,791.876 seconds (about three hours) for an INCOMPLETE assessment. Recorded overlapping stages: antivirus 9,855.401 s; structure 2,495.749 s (includes shared AV wait); compressed-source hashing 643.613 s; classification 50.093 s; startup 8.022 s; inventory 0.639 s; report writing 4.615 s. These cannot be summed, and this failed/held run is not a complete paired end-to-end benchmark. Sampled scanner CPU 8,916.448485 s and sampled anonymous-memory peak 2,113,576,960 bytes are lower-bound observations. Windows bind I/O/cache and I/O attribution limitations remain; no achieved 15% target is claimed.
Continuation scope: diagnose the exact terminal boundary with read-only metadata and small bounded selected checks, inspect the larger format-hold category without suppressing alerts, and use measured evidence to guide performance work. Do not repeat the full 155 GB assessment merely to reproduce its stop. Clean fetch/rebase onto unchanged main
0a47f4fand the cached full Linux AMD64 backend/vet/focused-race baseline have passed before further product changes. Recovery and admin work remain deferred. #12/#14 stay open and PR #19 stays WIP; no deployment, real destination, merge or gate waiver.Published
078103836d(Preflight archive layouts and handle small ZIP64 descriptors) to existing WIP PR #19. Started from clean5469cc6, read latest #12/#14/PR comments, fetched/rebased on unchanged main0a47f4fand established passing cached backend/vet/focused-race baseline. Fetched/rebased again after the validated commit and pushed normally; working tree is clean. PR body and versioned validation record now describe the final implementation and actual results.The full policy-9 assessment result from comment 534 is final, NOT an active run: 47,588/67,013 saved decisions in 10,791.876 s, 19,425 unvisited, 47,264 clean AV, seven explicit oversized exceptions, four blocks and 313 JPEG format holds; zero automatic retries and no downstream processor/import. Independent seal/completion/index/result/counter review passes, originals are unchanged and all exact readers/controllers/mappings/networks/aliases/locks were cleaned. The original session 25687 no longer exists. Historical per-file findings remain diagnostic; they do not authorize the original collection or policy-10 execution.
The terminal stop was a descriptor-width parsing incompatibility in the third archive, after the first two parts' 47,588 entries. Read-only metadata diagnosis confirmed both small explicitly local-ZIP64 members with wide descriptors and central-only ZIP64 size records with narrow local descriptors. Policy 10 handles both without relaxing CRC/size/range/overlap/encryption checks: local ZIP64 or large sizes require wide descriptors; central-only small sizes require exactly one matching width and an exact end at the next referenced local record/directory. Ambiguous widths and concealed descriptor suffixes remain blocked. Explicit integer bounds also remove old static warnings. No data, filename, media alert or AV eligibility rule was changed.
Every archive layout is now preflighted during existing metadata inventory BEFORE antivirus startup; content traversal rechecks its exact inputs. A later layout failure preserves inventory from all readable directories but starts no engine, creates no file results and issues no content receipts/assessment authority. Cancellation remains bounded. New input opens use os.OpenInRoot. Validation policy is bumped to 10; old signed evidence cannot bypass the changed policy.
Final corrected metadata-only test passes ALL three original archives / 67,013 entries in 79.346 s, one offline non-root CPU/768 MiB reader, with no content hashing, decompression, antivirus or authorization. Originals unchanged, exact reader cleanup and aliases/locks released. A prior corrected-parser pass took 83.977 s; these are metadata times, not end-to-end performance comparisons. No new full 155 GB AV assessment was launched.
Bounded JPEG follow-up: all 313 saved SCAN_FORMAT holds identify as JPEG. A four-member size-spread sample / 18,026,116 declared bytes passes classification but fails the existing JPEG structure checker; that terminates the shared diagnostic pipe, so its four AV outcomes are explicitly NOT_COMPLETED, not clean. The diagnostic batch stopped there. A separate 743,353-byte member again receives a ClamAV broken-format hold and fails structure, while a bounded Go JPEG decode passes and JFIF is segment five. A decoder displaying a file does not establish safety or clear AV. These are selected observations, not a verdict on all 313. All holds remain, raw replies/identities stay private, originals unchanged and readers removed.
Validation on the final source:
Measured AV remains the dominant inclusive stage (9,855.401 s); structure includes shared AV wait and cannot be added independently. Sampled CPU/anonymous memory are lower bounds, and cgroup I/O misses substantial Windows bind reads. The incomplete held run cannot serve as a complete timing pair against historical main analysis. No 15% target or full-library readiness is established. Exact sanitized results/limits are committed in docs/benchmarks/real-assessment-20261030.json and docs/malware-security-review.md; protocol and --layout-only use are documented.
Private resume: select the unique finished run under the dedicated policy9-5469cc6 workspace, not the newest generic local-run folder. Its report.json, assessment-aggregate-review.json, review-files-private.json, test proof/findings volume, policy10-jpeg-batch-1-private.json/log and policy10-jpeg-small-private.json/log retain raw evidence outside Git. Ignored helpers temp/issue12-policy9-final-review.py, temp/issue12-policy10-jpeg-diagnosis.py and temp/issue12-policy10-jpeg-small.py explain exact scopes. The JPEG batch helper intentionally stops on holds; do not interpret that expected failure as a clean sample. Final matching images are immich-bridge:issue12-policy10-tests and immich-bridge-{app,runner,processor,scanner}:issue12-policy10; scanner test image is immich-bridge-scanner:issue12-policy10-tests. Final logs use temp/issue12-policy10-final-tests.log, -final-isolation.log, -final-real-fixtures.log and -source-final/ . Metadata final run is under the separate policy10-layout-verified workspace. No real reader/AV run is still active. Test definitions remain fixed and must pass actual freshness checks before future diagnosis; refresh only outside active assessments, preserving generation binding.
Next:
No real destination, production interruption, deployment, restart, merge or issue closure occurred. #12/#14 stay open; PR #19 stays WIP.
Continuing from published
0781038and comment 537 with the owner's approval. Latest #12/#14/PR steering read; clean fetch/rebase onto unchanged main and cached full backend/vet/focused-race baseline pass. First scope is a bounded test-only JPEG marker/layout audit of already-held selected members, inside the existing offline restricted worker. Reuse the small diagnostic payload already read (at most 16 MiB/member, 64 MiB/batch, 16 targets, four-minute context); no image decode, AV repeat, archive rewriting or authority is needed for this audit. It should explain the exact format/structure mismatch before any compatibility policy change. Then profile representative shared-read/AV costs under existing limits. Keep every hold, alert and performance/security boundary; no blind full-set rescan. Recovery and admin controls remain deferred, with no real destination, deployment or production interruption.Published
aade4509dc(Preflight entry metadata and measure held media safely) to existing WIP PR #19. Started from clean0781038, read current #12/#14/PR steering, fetched/rebased on unchanged main0a47f4fand established passing cached full backend/vet/focused-race baseline. Fetched/rebased again after the validated commit and pushed normally. Working tree is clean; PR body reflects the final behavior and the active full assessment. Owner approval remains scanning/assessment first; recovery and admin configuration remain deferred.JPEG diagnosis now explains the selected strict-structure failures without an AV repeat: one offline CPU/512 MiB test reader audited four already-held members / 18,026,116 bytes, with 16 MiB/member, 64 MiB/batch, 16-target and four-minute bounds. Every payload matches the saved completed content digest and its ZIP CRC/length. Three have 12,123–16,901 bytes after the JPEG end marker, with no recognized immediate image/video/XMP/padding/vendor-footer candidate. The fourth has no recognized JPEG end marker and only a vendor-footer candidate. No primary-image MPF marker was found. No AV daemon, image decoder, extraction, output media or authority is created by this audit. Original identities unchanged, exact reader/aliases/locks removed. These observations do not establish infection, corruption or safety for all 313 JPEG holds. The earlier decoder success does not clear the complete-file structure/AV requirements. Unknown trailers and missing markers remain held; no general trailer allowance was added. Official Motion Photo/Ultra HDR/SEF references and exact caveats are linked in the committed diagnosis record.
Bounded AV component profile: seven selected members / 51,979,839 unique bytes cover two sidecar sizes, two clean JPEG sizes, a motion companion, native video and a held JPEG. One normal offline CPU/5 GiB worker, 2 GiB temp, fresh fixed managed definitions and four-minute context. Targets are capped at eight / 96 MiB unique / 64 MiB each, with at most 480 MiB AV bytes; actual total AV input was 259,899,195 bytes and all member reads 311,879,034 bytes. It performs integrity, first current AV, then two warm current/coalesced pairs in opposite order. The source bytes AND bytes supplied to AV match saved digests on every pass, all engine replies and structural outcomes are identical, and definition identity is unchanged. The held JPEG remains incomplete under every variant. Source errors (including real unexpected EOF) and cancellation have generated regression coverage.
Sums of per-file means: integrity 0.780927 s wall / 0.082916 s container CPU; first current AV 5.290041 s / 4.963526 s; warm current 0.9211445 s / 0.582694 s; warm coalesced 0.8753555 s / 0.501618 s. Warm data chunks fall from 2,265 to 798, but the wall-time saving is only about 46 ms. Cold coalesced cost and whole-library benefit were NOT measured. Coalescing remains a TEST-ONLY experiment and is NOT promoted or exposed as a server tuneable. Diagnostic hashing/clocks, warm clean-result caching, uncontrolled Windows/VM caches and host contention are explicit limitations. This is not the 15% end-to-end acceptance benchmark. Combined with the prior whole-run timings, the first-pass CPU cost points to AV computation rather than report writes as the important target-host measurement. No coverage, alert or resource ceiling is reduced to manufacture a speedup.
Production increment: move existing path/duplicate-name/type/hidden-directory and declared total expansion checks into the already-existing whole-set preflight before AV. The exact path/type predicate is SHARED with later content checking, which still validates exact names/layout, actual expansion, CRC and lengths. This changes when existing failures are discovered, not which files may proceed. Readable directories retain complete inventory on a later metadata failure, but zero file results/engine startup, no source receipts and no assessment authority. A partial/corrupt/unsafe input never becomes allowed through metadata success. Policy stays 10 because the rules are unchanged; immutable image binding still invalidates stale capabilities after rebuilding.
Final validation:
The diagnosis and sanitized measurements are versioned in docs/jpeg-holds-and-av-components.md and docs/benchmarks/av-components-20261030.json. Main source/evidence records link them. Private names, hashes, raw replies, paths and metadata values remain outside Git and these comments.
ONE corrected full read-only original-set assessment is NOW ACTIVE on committed
aade450, using allow-large-media and the ordinary one-CPU/5 GiB scanner, six-hour scanner / 24-hour controller deadlines, no worker network and fixed private definitions. All 67,013 entries are inventoried; at the latest probe zero per-file outcomes were saved during initial source binding. One snapshot shows exactly one worker, 13.40% CPU and 1.062 GiB/5 GiB, 12 processes; this is not a peak or final metric. No new baseline pair or end-to-end acceptance is claimed. All previous holds/alerts remain; no original copy, rename, rewriting, recovery, extraction or real destination/import is involved.IMPORTANT ACTIVE-RUN RESUME:
Next: finish this existing assessment, independently review completeness/coverage/statuses/timings/original identity and exact cleanup, then publish fixed aggregates without private identities. Continue meaningful target-host/repeated full performance and remaining broader workflow/real-provider/native image/security/release gates; a fixed-generation local test is not deployed periodic-updater acceptance. Do not weaken holds or start recovery/admin phases while readiness is unfinished. No deployment, production interruption, restart, merge or issue closure. #12/#14 remain open and PR #19 remains WIP.
Active assessment checkpoint on unchanged clean
aade450: session 67292 is still running and the latest bounded index probe records 31,252 / 67,013 saved file decisions (earlier in this continuation: 28,918). The report is not finalized. These are provisional decisions, not a count of clean files; final coverage, timings, original-identity and cleanup verification are still pending. No source edits, image rebuilds, competing scans, recovery/admin work, original-file mutation or production actions occurred during this continuation.Read-only readiness review reconfirmed the configured FreshClam interval (12 checks/day), database testing, offline scanner separation, freshness checks and generation-bound evidence. Existing source/tests refuse changed definitions without silently starting a replacement scan. This source review is not a newly executed updater test or deployed periodic-update acceptance. The active assessment deliberately uses its fixed test generation; do not change it mid-run. Actual HEAD still has 13 native checks pending: both source jobs are waiting to run and dependent suites/release gate are blocked by their required conditions. No gate was waived or cancelled.
The full resume/checks in comment 544 remain authoritative: one active fixed-build read-only pass, private parent policy10-aade450, normal worker/deadline limits, and shell session 67292 owning original-file sharing locks. Do not duplicate or interrupt it, release aliases before reader cleanup or rebuild its tags. Use temp/issue12-current-assessment-progress.py for bounded index-only progress. After actual completion, use temp/issue12-current-assessment-review.py to authenticate the finalized bounded manifest and reconcile results/digests/cleanup without reading media again, then record fixed aggregates here. The older policy9-5469cc6 assessment is finished and must not be confused with this run. Recovery and administrator scan controls remain deferred until readiness; #12/#14 stay open and PR #19 remains WIP.
Plain-language progress checkpoint: the corrected scan has now progressed beyond the previous stopping point (47,588 entries) and has saved results for 63,292 of 67,013 files. There are 3,721 files without results at this snapshot. The same original pass is still running; these counts include held/blocked decisions and are not an all-clear. No final result or cleanup verdict exists yet.
Next step: finish THIS running pass, then check that every file received a result, review the saved holds/blocks, and verify processing time, resource use, unchanged originals and exact cleanup. No replacement scan or recovery operation is needed to do that review. Local implementation/security checks retain their recorded passing results; all 13 native automated checks remain pending. Scanning readiness is not yet declared.
Session 67292 and the policy10-aade450 private workspace remain the active scope from comment 544. Keep its sharing locks and fixed image/definition generation intact until reader cleanup; do not duplicate or interrupt the pass. Recovery and admin controls remain deferred. No source/image changes, competing scans, deployment, merge or closure occurred during this status check.
The corrected full assessment has FINISHED on unchanged
aade450, with a saved decision for every file: 67,013/67,013, zero unvisited. It took 13,929.850 seconds (about 3 h 52 min). The completion metadata, framed manifest seal, inventory/result index, counters and content-digest rules independently reconcile. Assessment-only authentication passes; this grants no import permission. Originals are unchanged; exact workers/controller/mapping/network cleanup and source-alias release are confirmed. Session 67292 is finished; the old ACTIVE-run instructions are superseded. No repeat full scan is needed to inspect these results.Results: 66,302 complete clean AV results; 16 explicit oversized-media AV exceptions (27,582,744,740 bytes, mandatory checks passed, not represented as AV-clean); 688 JPEG SCAN_FORMAT holds; seven file-type blocks (one unsupported bitmap, one bitmap named as JPEG, five MOV names detected only as generic binary). No known-malware detection is recorded. Four already-rejected entries exceed the bounded rejected-data drain budget, so their ZIP-content integrity remains not_completed and they have no content digest; this does not mean they were unvisited. Every allowed file has a completed digest. All holds/blocks remain in force; no processor/import started.
The owner explicitly authorized overnight post-completion investigation without another confirmation. I am reviewing these fixed categories and bounded representatives from the sealed private manifest, starting with markers/container headers and existing diagnostics rather than repeating AV for the library. Private identities, hashes, metadata and original names stay out of public comments/Git. Google origin is useful provenance, not a substitute for a completed format/security check. No conversion, renaming, ZIP rewriting, recovery-copy/selective import or admin UI is authorized/started.
Private resume parent: policy10-aade450. The ignored helper temp/issue12-current-assessment-review.py verifies the final reports without media reads; private review-files-private.json is the current finding population (do not use the older policy9 population). Timing bottleneck: AV 12,429.961 s; archive verification 13,862.336 s includes shared work and is not additive with AV/structure. Whole-ZIP hashing 910.245 s; classification 67.109 s; structural stage 2,865.063 s includes shared AV. Resource samples: scanner CPU lower bound 11,287.333321 s, sampled anonymous-memory peak 2,105,249,792 bytes. Cgroup I/O excludes much Windows bind traffic; these are not total-I/O/true-peak claims. No paired end-to-end 15% result or native-release clearance is claimed. Next: explain held/blocked formats from bounded private evidence, document the findings and remaining acceptance gates. Recovery/admin remain deferred; PR #19 stays WIP and #12/#14 remain open.
The full scan is finished, and the requested post-scan analysis is published in
545e29a01e. Every one of the 67,013 files has a saved decision; none was left unvisited. Runtime was about 3 h 52 min. Originals are unchanged and cleanup is confirmed. There is no recorded known-malware detection, but this collection is still held.Results: 66,302 complete clean antivirus results; 16 explicit oversized-media exceptions (other required checks passed, antivirus coverage omitted); 688 JPEG format holds; seven file-type blocks. Completed assessment does not mean all-clear or permission to import.
Why these findings appeared:
A fresh Takeout download can preserve format quirks in stored media. Google's export/metadata guidance does not establish that these files satisfy our particular structural rules or have complete antivirus coverage. The evidence points to compatibility and format questions that need specific handling, rather than justification for disabling alerts.
Next implementation step: add narrowly tested recognition AND structural handling for self-contained legacy QuickTime, preserving complete applicable antivirus and existing sample/reference checks. Separately investigate representative JPEG trailers/composite layouts with bounded diagnostic reads. Do not restart the full library scan merely to inspect saved findings. No recovery, rewriting, conversion, selective import or administrator controls were implemented.
Plain findings and next steps · Exact sanitized measurements
Technical evidence, checks and resume
The completed assessment was on
aade450/ policy 10;545e29aadds only test diagnostics and documentation. Production scanner code, alerts, policy and bounds are unchanged. Clean fetch/rebase onto unchanged main0a47f4fand passing backend/vet baseline preceded the increment. Final full Linux AMD64 backend/vet, malware package tests, focused diagnostic races, nine CI policy tests, documentation/workflow/tracked-material contracts and redacted staged-secret checks pass. No new native/final-image/deployed-updater/real-provider clearance is claimed. PR #19 remains WIP.Independent framed manifest/completion/index/counter/digest reconciliation and assessment-only proof pass. Every allowed file has a completed digest. Four originally rejected entries retain integrity not_completed/no digest because of bounded rejected-data draining. Supplementary diagnostics passed CRC/length on all seven blocked entries, without modifying the sealed report, clearing holds or generating authority. All five movie candidates contain one mdat then one moov, no ftyp, two dhlr/alis handlers and two header-only flag-1 alis references each; the current probe explicitly returns MEDIA_STRUCTURE_UNSUPPORTED. Do not admit arbitrary aliases or external references based on this observation.
Private parent policy10-aade450 is finalized; session 67292 no longer runs. temp/issue12-current-assessment-review.py authenticates the retained report without media reads. Current private review-files-private.json is the population to use. Do not substitute the older incomplete policy9 population. New opt-in TestLocalFormatMetadataAudit uses existing classification/movie walkers, fixed output categories, RO locked inputs, 1 CPU/768 MiB, 16 targets, 128 MiB/member, 256 MiB/batch and a four-minute context; no AV/decoding/extraction/authority. Marker-only samples remain under tighter existing limits. All exact diagnostic readers and aliases are removed. Raw identities, hashes, metadata and source names remain private.
AV remains the largest inclusive stage (12,429.961 s); stage times overlap. This held security-only run is not complete ingestion or a current paired 15% benchmark. Performance acceptance and broader/native/image/updater gates remain. The owner authorized this analysis without additional confirmation; recovery and admin controls remain deferred. No real destination, deployment, restart, merge or closure.
Owner approved the examination-only archive direction in chat: package the 695 held/blocked files with readable/downloadable private reports, keeping the actual manifest, filenames, media hashes and paths out of public issues and Git. This supersedes the earlier blanket deferral ONLY for this examination export. Approved-remainder recovery, rewriting/sanitizing media, selective import and administrator controls remain deferred. The 688 JPEG antivirus results are incomplete and the seven type blocks were not AV-scanned; the bundle will explicitly be unverified content, never described as malware-cleared.
Start scope: implement and test a bounded offline export using existing ZIP safety checks and sealed saved findings, then produce the actual private bundle for owner examination. Copy exact selected file contents with CRC/length and available saved-digest verification, preserve source-part/folder relationships, include plain-language and machine-readable reports privately, and publish no partial archive. No whole-library AV repeat, real destination, import authority or clearing of holds. Original inputs stay read-only. Record aggregate counts/tests and handoff here; no actual per-file manifest or samples will be committed or posted. Clean fetch/rebase onto unchanged main and full backend/vet baseline passed before implementation.
Owner steering to retain for the LATER admin-controls phase: when server security checks are enabled, show a plain-language user-facing message that analysis may take time, that users can leave/return while it continues, and that they can receive an email when analysis has finished and results are ready for next steps. Integrate with the existing verified-address/opt-in notification preferences and truthful completion states (including files needing review), rather than promising delivery when notifications are unavailable or implying every completion is import-ready. This is recorded for later implementation with admin controls; it does not expand the current examination-bundle work or bring admin controls forward.
The owner-approved private examination bundle is ready. Published tooling/docs:
c2752ff00b; main remains0a47f4f. Actual filenames, per-file reports/manifests, hashes, original paths and the archive are NOT in Git or public comments. The bundle and separate readable report are provided only through the owner's private local workspace/chat.Included: all 695 held/blocked format files (688 SCAN_FORMAT, six TYPE_MISMATCH, one UNSUPPORTED_TYPE), 2,366,783,766 original content bytes. ZIP size is 2,368,015,440 bytes. It contains the files under source-part-separated original folder relationships, README.txt, REPORT.txt and REPORT.json; both reports are also available separately without extracting media. Reports preserve the original check statuses and add separate copy-integrity evidence. No file is described as antivirus-cleared: 688 retain incomplete AV and seven retain not_run. All existing holds remain.
Generation took 112.046 s in one offline non-root 1-CPU/2-GiB worker. Selected contents share one read for stored-ZIP output, CRC/length and hashing; no recompression, decoder, loose-media extraction or AV starts. Saved manifest/completion, whole-set metadata/layout rules and exact source selection verify. 691 completed saved content digests match; four previously rejected files now pass supplementary copy CRC/length and get separate copied digests without altering their original saved incomplete-integrity status. Originals retain their size/mtime/file identity, and exact worker/alias cleanup passes.
A separate offline RO verifier with no original-media mount verifies every new ZIP payload and embedded/standalone report in 15.38 s. Final code's partial-artifact verifier and required count-matched success marker also pass against the same bundle in 13.760 s; no duplicate source export or whole-library rescan was needed. Final tooling integrates that verification before publishing fixed names. Capacity/writer bounds and confirmed cleanup precede publication; Docker cleanup outages retain sharing locks while exact cleanup retries.
This increment implements private OPERATOR examination tooling, not a deployed/in-app download action. Application owner-download/admission/retention integration, approved-remainder recovery and selective import are separate future work. Current limits are 10,000 files/16 GiB selected content/32 MiB per report, bounded ZIP overhead, a 1 GiB free-space reserve and a 30-minute worker context. Software writer limits are not a filesystem quota against a compromised worker. Operator bundles are retained until owner removal. No authority, collection registration, import, recovery/sanitization, real destination, deployment, restart, merge or closure occurs.
Checks: clean fetch/rebase and full backend/vet baseline; final full Linux AMD64 backend/vet; malware package tests and exporter-focused races; generated refusal/selection/integrity/cancellation/output-bound cases; two Python safety tests, three Windows input-adapter tests, nine CI policy tests, documentation/workflow/tracked-material contracts and redacted staged-secret scan all pass. Production scanner/authorization/UI/import rules are unchanged. Native/release/performance/scanning readiness gates remain; PR #19 stays WIP.
Examination design, actual aggregate results and boundaries. Reproducer: tests/export_local_review.py plus the opt-in TestLocalReviewArchiveExport/TestLocalReviewArchiveVerify in a matching malware test image. The private finalized policy10-aade450 assessment remains the selection source. No background scan/export remains active. Next: owner examines the private report/bundle; continue narrow legacy QuickTime and JPEG format diagnosis using targeted files, preserving holds. Later admin-controls reminder from comment 561 is now also recorded in docs/malware-scanning.md: security-enabled analysis-duration/leave-return/optional completion-email guidance, respecting current preferences and truthful next-step states.
Owner is reviewing the private bundle and reports that the first 20 manually opened files display normally. Starting a parallel-in-time (single agent) bounded diagnosis of the examination set: group JPEG header/trailer patterns across the small review bundle, inspect remaining unsupported containers and compare candidate structures against primary format/engine sources. Use the existing private bundle/report and read-only isolated readers; no whole original-library AV repeat, automatic previews, production-policy relaxation or import. Keep all actual names, per-file manifests, payload/metadata values and hashes private; public evidence will remain fixed categories/counts. Clean fetch/rebase onto unchanged main and backend/vet baseline precede diagnostic code changes. Viewing success is useful compatibility evidence, not full-file security clearance. All existing holds remain.
The complete review-set investigation is published at
0db7a1ce85; main remains0a47f4f. The owner manually confirmed wanted photos/videos display normally and traced the one unwanted game package to an inadvertent desktop backup. Game packages remain excluded from import/manual-review findings even with an embedded bitmap prefix. No originals were deleted. A generated regression proves existing .vpk classification gives no media/oversized exception; production rules are unchanged.A bounded offline census covers all 695 review members / 2,366,783,766 bytes in 18.767 s; every copied digest/CRC/length matches, no AV/decode/preview/full-original-library read. All 688 JPEG headers violate the engine's accepted JFIF-position cases. Exact groups: 654 share the aecDebug trailer marker; 24 end exactly at JPEG EOI; four contain no EOI byte pair anywhere; four have post-EOI Samsung footer candidates; one has another JPEG component; one has zero padding. Common markers are not a validated trailer schema or permission to ignore suffixes. A trial length-framing hypothesis did not cover complete debug trailers and is not promoted.
Metadata-only inspection in 7.213 s finds 679 extended-XMP sets with complete ranges, matching lengths/identifier digests, references from standard XMP and well-formed reconstructed XML. This explains many extra header segments. It is not validation of every metadata schema, embedded payload, trailer or codec, and the saved report does not retain each raw signature to claim identical engine replies across all 688. Successful viewing is compatibility evidence; the four missing markers remain a separate exact-byte finding without a claim of visible damage or Takeout corruption.
The wanted JPEG-named bitmap has a consistent whole-file uncompressed BMP layout. The unwanted package starts with a valid BMP then substantial other content, explaining the prefix-only detector result. The five wanted movies retain the legacy QuickTime prefix/reference compatibility barriers. Supplementary actual AV on these six wanted non-JPEG files / 137,755,316 bytes now reports six clean results in 22.618 s. Their original type blocks had prevented AV. An initial attempt correctly refused aged definitions before scanning; a separate managed updater with NO media mount refreshed a separate test generation (daily 28170/main 63/bytecode 339, database test passed). This is a one-shot refresh, not deployed periodic-update acceptance. Normal 1 CPU/5 GiB scanner limits apply; no JPEG AV repeat or whole-library rescan occurs. Original saved statuses, holds and execution authority are unchanged.
Private FORMAT-ANALYSIS.txt and FORMAT-ANALYSIS.json beside the bundle map each file to its exact group and supplementary findings. Actual filenames, original paths, manifest rows, hashes and metadata/payload values are NOT committed or posted. The specific bitmap identity requested in chat was disclosed only privately. All scoped readers/updater/aliases are cleaned; original bundle remains unchanged.
Next implementation sequence: narrow complete BMP alias and self-contained legacy QuickTime validation with negative fixtures; then JPEG metadata/trailer framing and proof of engine coverage under format heuristics using source review and safe generated canaries. The 24 no-trailer JPEGs are a smaller starting group. Do not disable broken-media alerts globally, strip/rewrite bytes before AV to manufacture a clean reply, or infer clearance from a decoder. The common debug trailer still needs a complete bounded format rule; missing markers/vendor components/additional JPEG/padding need explicit treatment. Keep one-pass normal ingestion and existing resource limits.
Checks: clean fetch/rebase and full backend/vet baseline; final Linux AMD64 full backend/vet, malware tests, focused census/metadata/package-regression races, nine CI policy tests, documentation/workflow/private-material contracts and redacted staged-secret scan pass. Only opt-in test diagnostics, a regression for existing classification and sanitized docs/aggregates change. Production scanner/alerts/authorization/UI remain unchanged; native/release/performance/scanning readiness gates remain. Approved-remainder recovery, selective import and admin controls stay deferred. No real destination, import, deployment, restart, merge or closure; PR #19 remains WIP.
Findings, primary sources and evaluation plan · Sanitized exact aggregates. Resume with the existing private bundle and format-census/metadata-framing/wanted-AV private location records; never substitute an actual per-file report for the sanitized artifact in Git. New opt-in tests are TestLocalReviewFormatCensus and TestLocalReviewMetadataFraming; existing TestLocalBatchMemberDiagnostic performs the supplementary six-file check. Fresh test-generation pointer is ignored temp/issue12-review-definitions.json; retain the older generation for historical evidence. No diagnostic reader, scanner or updater remains active.
Owner steering recorded in documentation-only
8bf59b791b: a bitmap named as JPEG must remain in manual review with a specific format/name mismatch explanation, even when the bitmap layout and diagnostic AV pass. This supersedes the earlier BMP-alias recommendation in comment 570 and the previous plan. Suggested wording: "This file is named as a JPEG but contains bitmap data. It may not work correctly as a JPEG. Review or replace it before importing." Do not silently rename, convert, accept or delete it. Existing production behavior already holds this case.MOV validation is possible without requiring conversion: extend narrow legacy QuickTime recognition and self-contained reference handling, preserve complete bounded container/track/sample checks and applicable AV, and reject external/ambiguous/unsupported data. A bounded independent probe is optional only where the existing checker needs it; full decoding/transcoding is not the normal path.
Recommended JPEG direction remains proving supported metadata/component framing and actual engine coverage with source review and safe generated tests, starting with the 24 no-trailer cases. Do not globally disable broken-media alerts, ignore arbitrary suffixes or turn a viewer success into security clearance. Undocumented trailers, missing markers and incomplete coverage need explicit handling. The UX target is automatic handling of supported files, one clear result and a small private exception list; approved-remainder/selective import and optional repair/conversion remain separate future work. No implementation of that future behavior or clearing of current holds is claimed.
Documentation links/format checks pass; no production code, tests, media, actual per-file manifest or user filenames changed/published in this increment. Main unchanged, PR #19 remains WIP; no deployment, import, merge or closure.
Owner approved the current plan in chat. Starting implementation with bounded legacy QuickTime support, then the simpler JPEG/engine-coverage fixtures. Preserve the steering from comment 576: BMP-under-JPEG mismatches and game packages remain manual-review/unsupported cases, with no automatic rename/conversion/deletion. Recovery of the approved remainder, selective import and admin controls remain deferred.
MOV scope: use bounded real container evidence as a candidate, require the existing full-stream movie probe and complete applicable AV, admit only exact header-only self-contained legacy alias references in QuickTime context, and preserve all sample/offset/timing/codec/resource checks. Add negative fixtures for fake prefixes, external/payload-bearing aliases, malformed/range/timing cases and AV failures; validate the five private wanted candidates without another whole-library scan or real import. Bump the validation policy for changed eligibility and invalidate old reusable evidence. Follow with safe generated JPEG canaries/source checks before changing any handling of format warnings. No global disabling of broken-media alerts or transformed-byte AV shortcut is authorized.
Clean fetch/rebase onto current main (unchanged
0a47f4f) and existing backend/vet baseline precede code changes. Actual file identities/manifests remain private; public handoffs use fixed counts/codes only. No deployment, restart, merge or issue closure.Published
bf4073d237on the existing WIP branch. Main remains0a47f4f. The approved first compatibility increment is implemented and locally verified; overall scanning readiness is still unfinished.MOV result: policy 11 recognizes bounded actual legacy QuickTime framing without requiring optional
ftyp, then requires complete structure plus applicable AV. Exact header-only flag-1 self-containedalisreferences/data handlers and fixed version-0 unsigned 8-bit mono/stereo PCM descriptions are supported. Full timing/sample/offset/nonoverlap/codec/resource checks remain. Fake prefixes, external/payload aliases, malformed descriptions/ranges, warnings/errors and strict/partial size behavior have generated regressions. No decode, conversion, full-file copy, extra decompression or service. BMP-under-JPEG mismatches and game packages stay in manual review/unsupported handling, with no rename/conversion/deletion.All five private wanted movies / 137,459,582 bytes pass combined actual AV and complete structure in 31.696 s, with copied-content digests and ZIP CRC/lengths matching, unchanged bundle and confirmed exact reader/alias cleanup. One offline non-root worker uses 1 CPU / 5 GiB / 2 GiB temporary limits. Startup/hashing and uncontrolled host load are included; this is not an end-to-end throughput/15% result. No original-library AV repeat or real import occurs. Private readable/JSON MOV-VALIDATION-UPDATE reports are beside the examination ZIP. Actual media names, original paths, manifest rows, hashes and metadata stay out of Git/public comments. Saved policy-10 reports remain unchanged; old reusable evidence cannot authorize policy 11. A fresh analysis is explicit, never automatic.
Incremental definitions bug found and fixed: required-mode integration initially refused the real refreshed managed generation because Freshclam's rebuilt CLD has the documented exact
X:Xouter-header placeholders. Identity now recognizes that pair only for CLD in the trusted managed volume. CVD placeholders, malformed/partial pairs, ambiguous/missing databases remain refused. Header/name/size/mtime stay bound; even same-size header updates with preserved timestamps invalidate identity. Actual ClamAV database loading/testing, freshness, read-only mounts and before/after generation checks remain. This does not enable custom databases, weaken age checks or change updater scheduling. The actual required-mode integration now passes with that refreshed generation.JPEG finding: safe generated ClamAV 1.4.6 tests detect a raw test signature before/after the troublesome JFIF header and after EOI. A stronger file-hash canary for a supported Photoshop thumbnail is detected standalone and inside an ordinary JPEG, but missed after late JFIF while only the format hold is returned. Source review matches an early parser exit. Therefore a structural/viewer pass plus suppressed format warning cannot establish complete supported-component coverage. Opaque compressed ZIP canaries are not extracted even in ordinary JPEG controls, so they are recorded separately rather than treated as successful nested coverage. No actual infection is claimed and no JPEG hold, alert or engine setting is relaxed.
Checks actually run: final Linux AMD64 full backend/vet; focused API/security/runner races; separate definitions races; actual offline real-engine fixtures including generated large media (13.01 s) and the new JPEG control (0.07 s); all four canonical application/runner/processor/scanner image builds; required-mode Docker integration with race controller (47.61 s, disposable synthetic destination), clean/partial preview-upload, held refusal, authenticated no-rescan paths, mutation/replay/isolation/cleanup. Source gosec policy passes: 63 production files / 13,821 lines / 41 existing reviewed findings, zero suppressions/baseline expansion; source govulncheck passes. Nine CI policy tests, documentation/workflow/privacy contracts, staged exact-private-identity exclusion and redacted staged-secret scan pass. Worktree clean, fetch/rebase unchanged, push complete. No scanner/updater remains active.
Next: use the existing private reports/bundle to inventory supported metadata and embedded components specifically in the 24 no-trailer JPEGs; build bounded positive/negative component fixtures and prove coverage on original bytes before changing eligibility. Preserve one-pass ingestion, raw AV, explicit byte/count/deadline limits, and hold unsupported components. The recurring undocumented debug trailers, missing EOI and vendor/composite formats remain separate work. Do not disable broken-media alerts globally or scan rewritten photos to manufacture clean replies. Opt-in diagnostics include TestLocalReviewMetadataFraming, TestLocalFormatMetadataAudit and TestLocalBatchMemberDiagnostic; the latter can now assert saved copied-content digests. TestRealClamAVJPEGFormatCoverage reproduces the parser coverage gap with generated content only.
Native CI has 13 pending checks on this exact published head; no native/release waiver, new browser/real-provider/final-image advisory clearance, complete performance acceptance or scanning-readiness claim. Approved-remainder recovery, selective import and admin controls remain deferred, including the recorded later security-enabled analysis-duration/leave-return/completion-email UX. No deployment, restart, real destination, merge or issue closure. PR #19 remains WIP.
Implementation and primary sources · Sanitized measurements.
Continuing owner-approved work from comment 580. Scope: privately inventory the 24 no-trailer JPEGs and their metadata/components, add bounded positive/negative structural/component coverage tests, and only change eligibility if complete original-byte/applicable-component AV is established. Keep late-JFIF holds until that proof; no global alert disabling, transformed-byte clean shortcut, full-library rescan or decoder/transcoding service. BMP mismatches/game packages, undocumented trailers, missing EOI and composite/vendor cases retain their review decisions. Preserve one-pass limits and original reports, with no public actual file identities/manifests. Approved-remainder recovery, selective import and admin controls remain deferred. Clean fetch/rebase and baseline checks precede implementation.
Published
f40f4d0a50. Main remains0a47f4f. This is an opt-in diagnostic/test prototype, not a production JPEG exception; all existing holds, policy 11, saved assessments and execution boundaries remain unchanged.Next concrete action: implement a strict supported JPEG metadata/component profile in the existing pass, starting with the three panorama candidates. Complete EXIF/ICC, unknown-marker, recursive-component, integrity/error/outage and authorization handling before tolerating only the precise JFIF-position warning. Apply relevant encoded-component coverage to ordinary JPEGs too. Retain raw AV, detection priority, original warning/compatibility evidence, private results and byte/count/deadline limits. A changed eligibility rule requires a policy bump and invalidation of old evidence. The 21 camera-debug cases need separate semantic decisions; trailers/missing markers stay separate. No additional confirmation is needed for that already-approved next work.
Findings: all 24 no-trailer files have extended XMP. Three contain unpadded base64-encoded secondary JPEGs whose complete structures pass. Twenty-one contain 42 HDRP camera-diagnostic wrappers; bounded decoding verifies a single gzip member/checksum/end in each. Inside the camera notes, 255 encoded field slots are observed: 252 decode to small generic binary data, while three optional debug values remain unresolved. Neither unknown debug slots nor binary labels establish corrupted/infected photos or safe complete schemas. The implementation reference and exact bounded limitations are documented; no new program/service/dependency is added.
Actual coverage: an added generated canary is missed when base64-encoded in an ordinary JPEG's XMP, yet detected after component decoding. The late-JFIF equivalent remains format-held. Thus ordinary engine-clean and format-warning cases both need thought about encoded components; no actual infection is claimed. The earlier supported-thumbnail and raw-signature controls remain.
The final private prototype processes 24 originals / 68,622,039 bytes from one forward ZIP-member read each, sharing bytes with original AV, full JPEG structure, copied-content digest and ZIP CRC/length checks. It retains only bounded headers (observed maximum 2,447,352 bytes). After original AV finishes, one existing daemon sequentially checks 339 component representations / 11,781,068 bytes, all clean. All 24 originals still return the JFIF-position warning; three unresolved slots are not counted as scanned. Wall time 24.138 s, offline non-root / 1 CPU / 5 GiB / 2 GiB temporary limit. Definition freshness and before/after identity pass. Exact reader cleanup, locked-alias release and unchanged source pass; no active scanner/updater remains. Startup/hashing/caches/host load are included: this is not complete-library throughput, production readiness or paired 15% performance acceptance.
Important integration constraint: capture metadata while the existing original stream runs, then scan decoded components after its AV request joins. Starting a second AV request from the shared reader/probe callback can deadlock the single-thread daemon while it awaits more original bytes. Do not add an extra extraction/full-photo buffer or per-file worker. The diagnostic demonstrates the bounded ordering; production still needs its strict profile and fail-closed integration.
Private JPEG-COMPONENT-UPDATE.txt/json reports beside the examination ZIP map these results to the actual files. Full private schema/payload reports and saved identities remain outside Git/public comments. Only sanitized aggregate measurements and generated fixtures are published. Original policy-10 reports are preserved; diagnostics issue no execution or recovery authority.
Checks actually run: clean fetch/rebase and full backend/vet baseline; final Linux AMD64 full backend/vet/malware tests; focused component/HDRP/header-capture races; actual ClamAV 1.4.6 generated coverage test; private bounded one-pass batch; nine CI policy tests; docs/workflow/private-path contracts; staged exact-private-identity exclusion and redacted staged-secret scanning. Negative fixtures cover chunk boundaries, malformed/range/size/depth/entity/expansion/count/CRC/suffix/multistream failures, cancellation and component visitor errors. Production source/dependencies are unchanged, so prior production-image/source-policy/integration evidence is not presented as newly rerun. Native/release/provider/image/browser/performance gates remain; current native status is recorded separately below.
Approved-remainder recovery, selective import and admin controls (including the recorded later duration/leave-return/completion-email guidance) stay deferred. No real destination/import, original-library rescan, alert disabling, rewritten-original clean shortcut, conversion, deployment, restart, merge, closure or gate waiver. PR #19 remains WIP.
Findings, limits and production next step · Sanitized exact measurements.
Native status on this exact published head: 13 checks pending; no waiver or acceptance is implied.
Continuing the owner-approved production JPEG work from comment 587. Mandatory fresh fetch found new main
7663404(#25 retained-ZIP re-runs/progressive workflow), so resolving and validating the rebase comes first. All 28 #12 commits are now rebased locally; scanner/format changes are preserved, with workflow/UI conflict resolutions retaining new main behavior, security findings and partial-coverage consent. The new analysis endpoint must omit old private security evidence from ordinary reads and clear it atomically when starting a new operation. New operation identity must also invalidate old partial-coverage acknowledgement even if the coverage digest is identical, with the acknowledgement reachable outside collapsed setup sections. Full backend/vet baseline and UI lint/build pass; targeted backend/races/browser tests are running. A failed preliminary build against the mid-rebase tree is not counted as validation. No JPEG production eligibility change has begun yet. No private identities, deployment/restart, real import, new native waiver or recovery/admin work. Next remains strict panorama metadata/component coverage after the integrated baseline is sound.Published the mandatory integration checkpoint
36b230a3ad, rebased onto current main7663404. All 28 #12 commits are preserved; exact-expected-head lease protected the feature-branch update, and a local pre-rebase evidence ref is retained. Main and the deployed pilot were not changed. The old MOV/XMP evidence snapshots map to d5d351f/9ebf734; original assessment/media bytes and historical reports are unchanged.The new protected retained-ZIP re-run handler omits private security evidence from its ordinary read and clears it in the atomic new-operation reset. A seeded regression proves reset while uploads/history/notification outcomes remain. UI report queries now follow current operation identity; partial-coverage consent is bound to operation plus coverage digest, so identical results from a re-run still need fresh consent. Consent remains visible outside collapsed setup sections and resets before each new preview/import. Private findings and technical attempt summaries coexist with #25 progressive sections, active Stop controls, settings and history.
Checks: full Linux AMD64 backend/vet and focused API/security/runner races; UI lint/build plus canonical Node-24 application/runner/processor/scanner builds; six synthetic browser suites (migration workflow, advanced options, private findings, task activity, email and mobile ZIP rail), including 1280/390/320px re-run/consent coverage. Actual required-mode Docker integration with race controller passes in 83.35 s, covering clean/partial preview-upload, held refusal, authenticated reuse, mutation/replay/isolation/cleanup. Source policy passes: 64 production files / 13,872 lines / 41 reviewed findings, zero suppressions/baseline expansion; source govulncheck passes. Nine CI tests, docs/workflow/privacy and redacted staged-secret checks pass. A preliminary mid-rebase build is excluded from successful evidence; the final clean baseline passes.
Native state on this exact head: pending (13 checks). No native/release/provider/image-advisory waiver, original-library rescan, actual user import, deployment/restart, merge or closure. No JPEG eligibility change yet: production policy remains 11. Continuing next with the already-approved strict panorama metadata/component rules on this verified baseline, retaining unknown/debug/trailer/missing-marker/mismatch holds. Recovery/selective import/admin controls stay deferred.
Integration details.
Published policy-12 panorama checkpoint
e143297b44on the security feature branch, with all 30 prior security commits rebased onto current maina279a1b, followed by workflow/locale integration and expanded engine controls. Exact-expected-head lease protected the rebase update; the later test-only commit used a normal push. Local pre-rebase evidence refs are retained. Main and the deployed pilot are unchanged.Result: the three real panorama photos now pass the complete recorded scanner path. The raw original JFIF-position warning is retained separately from
JPEG_PANORAMA_V1/clean_compatibleevidence. Full original AV/integrity/structure shares one source read; bounded Exif/XMP checks precede sequential embedded-JPEG AV. Known fields/types/counts, directory/reference ranges, exact redundant text copies, packet/geometry/table bounds, unknown metadata, recursion and failure paths have explicit rules. No original rewriting, conversion or global alert relaxation. Policy 12 invalidates older reusable evidence; historical saved assessments remain unchanged.Fresh actual-engine mixed recorded test on the rebased scanner: 24 no-trailer JPEGs / 68,622,039 bytes; three allowed panoramas, 21 held camera-debug photos, every private result verified, no collection receipt or import authority. Three embedded images / 3,663,846 bytes are separately scanned. Latest scanner wall time 28.990 s / isolated command 30.347 s; prior panorama-only test issued a test-archive integrity receipt in 24.218 s scanner wall time. Cold startup/load/caches prevent treating these targeted timings as complete-import overhead acceptance. Original inputs are unchanged; exact readers are removed before locked aliases are released. No original-library rescan or actual import.
The additional final pinned-engine test crosses ordinary/late-JFIF layouts with clean/encoded-canary contents. Raw ordinary AV reports clean despite the encoded harmless signature; production component checks block it in both layouts. Clean controls preserve ordinary versus compatibility evidence. Detection, other warnings/limits, outages, bad/short input and cancellation cannot obtain the exception.
Private readable PANORAMA-VALIDATION-UPDATE.txt and detailed JSON are beside the examination ZIP. The private manifest, names, original paths, content digests and metadata values are outside Git/public comments. Public documentation contains aggregates and fixed codes only. Normal private UI wording explains the extra photo checks; codes/counts are folded under technical evidence.
Current main integration retains guided stages, SSO/dependency fixes and operation-bound report/coverage consent. Preview and Import each expose fresh consent outside collapsed setup; identical-result reruns need new consent. Finished tasks remain available through history while active Stop controls stay visible when navigating backward. Scanner/updater use fixed C locale / UTC, checked against hostile controller locale variables.
Local checks pass: full backend/vet and focused API/security/malware/runner races; four canonical images; real ClamAV fixtures including large media and embedded controls; actual required-mode Docker integration with race controller 61.18 s (synthetic clean/partial preview-upload, held refusal, authenticated reuse, mutation/replay/isolation/cleanup); UI lint/build and seven synthetic browsers, including 1280/390/320px guided consent/re-run coverage. Source policy: 69 production files / 15,301 lines / 43 reviewed findings, zero suppressions. Two new exact MD5 entries are format-identification only, with SHA-256/HMAC authorization unchanged; numeric findings were fixed in code. Source govulncheck, zero-advisory npm install audit, nine CI tests, docs/workflow/privacy, redacted staged secrets and exact exclusion of 2,035 private identity/digest strings pass.
Native checks on this exact head: pending (13 checks). PR #19 stays WIP; no native/release/provider/image-advisory/performance waiver. Next: investigate supported semantics for the 21 camera-debug files without treating successful decoding or unknown binary AV alone as full validation; then handle vendor/debug trailers, missing markers and composites. Preserve mismatch/game-package review decisions. Recovery/selective import and later admin controls remain deferred, including security-enabled duration/leave-return/completion-email guidance. No deployment/restart, real destination, merge or closure.
Strict profile · Validation and remaining work · Sanitized benchmark.
Beginning the owner-approved validation of the 21 no-trailer camera-debug JPEGs from the policy-12 handoff. Current main remains
a279a1b; the feature worktree is clean ate143297. First inspect bounded private decoded camera-note structures and the three unresolved optional field slots against primary format implementations, then add focused malformed/limit/detection controls and any justified supported validation rules. Existing holds and original-warning evidence stay in place unless complete coverage is demonstrated and changed eligibility is versioned. Preserve one forward source stream, bounded sequential work, offline worker isolation and private reports. No media identities/manifests/digests in public Git/issues. Recovery/selective import/admin controls and deployment/restart remain outside this increment.Published
941c440573on the security feature branch, based on unchanged maina279a1b. This increment adds opt-in diagnostic validation, generated negative tests and documentation; production policy remains 12 and all 21 camera-debug photos remain held. The three approved panoramas keep their existing policy-12 rules.Plain-language finding: nothing checked here establishes that these photos are corrupt. Their holds are caused by camera metadata outside the supported profile. Three previously reported decoding failures were a line-boundary mistake in our diagnostic parser, not damaged encoded data; that mistake is fixed and regression-tested. All 255 preview buffers now decode and fit a bounded inferred raw-raster layout. That layout remains an inference rather than a complete vendor specification.
All 21 narrow ICC profile candidates and bounded Exif directory graphs pass, with no unclaimed nonzero Exif bytes. Three standard Exif fields (47 instances) still need explicit production type/count/value rules. All 18 embedded Exif JPEG thumbnails pass complete framing and separate AV. The 21 binary camera logs pass bounded protobuf wire checks, but two byte-valued leaves per file remain semantically unresolved (42 leaves / 1,072 bytes total). They receive separate AV too. A generic parse, plausible numeric interpretation or negative AV does not establish the missing field declarations.
Fresh actual-engine validation: 24 original members / 68,622,039 bytes, including three panorama controls, followed by 465 component representations / 13,658,523 bytes, all negative for malware detection. These cover decoded wrappers, reconstructed gzip, expanded notes, preview buffers, unresolved leaves, Exif thumbnails and ICC profiles. Original AV, full JPEG framing, saved-content comparison and ZIP integrity share one forward source read per member; bounded sequential component work begins after original AV joins. Original raw engine JFIF-position warnings remain recorded. This diagnostic does not revoke the existing panorama approvals or issue camera compatibility/collection authority.
Final isolated command: 30.806 s, including startup, hashing and private snapshots under variable load/cache state; not a paired end-to-end overhead acceptance. The offline non-root worker retains one CPU / 5 GiB memory / 2 GiB temporary space, a three-minute context, 512 component requests and 256 MiB component bytes. Expansion is checked before reconstructed gzip AV. Fresh managed definitions and before/after identity checks pass. Inputs remain unchanged; exact reader removal precedes locked-alias release, and no worker remains active.
Checks actually run: full Linux AMD64 backend/vet and focused API/security/malware/runner races; all real-ClamAV fixtures, including a new generated encoded-raster canary that original ordinary/late-JFIF AV misses and separate raster AV detects; nine CI tests; docs/workflow/privacy contracts; exact exclusion of 2,035 private identity/digest strings; redacted staged-secret scan; whitespace checks. Generated negatives exercise encoding/size/layout limits, Exif cycles/overlap/references, ICC Unicode/types/ranges/curves, protobuf truncation/overflow/groups, cancellation, component budgets and visitor errors. A recorded production-path regression confirms successful diagnostic checks grant no compatibility or receipt to unknown metadata.
Private readable CAMERA-METADATA-VALIDATION-UPDATE.txt and detailed JSON are beside the examination ZIP. They include per-file findings for local examination; full raw snapshots remain private. No names, original paths, metadata values, content digests or manifest rows were added to public Git/issues. Historical reports and assessments remain unchanged; the new report supersedes the earlier three-slot decoding interpretation.
Next step: define and test supported treatment of the unresolved camera-log leaves, then integrate reviewed HDRP/raster, the three standard Exif fields and narrow ICC rules as one versioned production change. Require complete recorded decision-path, generated detection/failure and real mixed-file validation before changing eligibility. Unknown schemas and other errors stay held. Then return to remaining vendor/debug trailers, missing markers and composites.
Native status on this exact head: pending (13 checks). PR #19 remains WIP. No new production-image/browser/provider/source-policy, complete-library performance or release clearance is claimed; prior scoped evidence remains in comment 636. No original-library rescan, recovery/selective import, admin controls, deployment/restart, actual import, merge or closure. The deployed pilot is unchanged. #25's separately completed owner acceptance does not waive these security gates.
Findings and primary references · Sanitized measurements · Validation record.
Continuing the owner-approved investigation of the two unresolved camera-log fields for the 21 held photos, from checkpoint
941c440. Current main was fetched and remainsa279a1b, already included in the feature branch; prior exact-tree backend/race baseline passes. Compare bounded private binary leaves with independently named text metadata and primary producer/reader sources; add discriminating generated controls and record any inferred versus proven meanings separately. No unknown-data approval based on generic framing or negative AV. Preserve original warning evidence, one source read, bounded isolated work and private reports. Existing holds remain until complete production coverage is defined and tested. Recovery/admin/deployment/import remain deferred.Published
1e6a5d6c3fon the security feature branch, based on unchanged maina279a1b. This continues the owner-approved investigation from comment 657. All changed Go files are test-only; production policy remains 12 and the 21 photos remain held. Main and the deployed pilot are unchanged.The two fields now have a specific, well-corroborated candidate interpretation: numbers describing how the camera combined exposures into the finished photo—frame merge weights and frame merge factors. They fit packed 32-bit floats alongside a frame-weight exponent. In 18 photos, both arrays, their exponent and the number of merged frames agree with the camera's named readable notes, allowing only the rounding used in those notes. The other three lack all four readable lines; they are explicitly recorded as absent, not as matches or damaged data. All 21 pass the same bounded binary candidate. This supersedes the earlier undifferentiated opaque-leaf interpretation; it is not an authoritative complete Google declaration or production clearance.
The comparisons distinguish this from the earlier plausible-double interpretation, whose element count does not match the named arrays/count. Only 10 readable cases also match the different maximum-possible-weight line, whereas all 18 match actual merge weights. No filenames, source paths, private values, content digests or manifest rows are published.
Checks added: exactly one outer merge record; complete known inner fields with exact multiplicity/wire types/packed extent; finite values, bounded counts/ranges, equal array sizes and reference-frame rules. Unknown/duplicate fields, nonminimal varints, truncation, extra bytes, nonfinite/out-of-range values and excessive work fail. Text comparison uses fixed decimal rounding intervals and rejects partial, duplicate, malformed, reordered-value, extra-token or contradictory lines. Candidate bounds are a supported research subset, not claimed vendor maxima. Other outer camera-log fields still need complete typed production handling.
A new actual-ClamAV control proves why these rules must remain layered: harmless signature bytes can also be valid bounded numbers, with matching forged readable notes. Raw ordinary/late-JFIF JPEG AV misses the encoded signature; separate decoded-leaf AV detects it. Neither numbers nor matching notes authenticate metadata or replace component scanning. Recorded production tests still refuse compatibility/authority for camera candidates.
Actual-data check: 42 previously extracted private metadata snapshots / 6,590,205 bytes; 21 candidate passes, 18 four-detail matches, three absent sets, zero contradictory readable sets. Final isolated command 2.885 s with one CPU / 256 MiB memory / 16 MiB temporary space, offline non-root/read-only worker, before/after plus worker SHA-256 checks, Windows read-sharing locks and exact reader removal before locked-alias release. This reads no original photos/archive and makes no new actual-media AV claim; the previous 465-component managed-engine result remains historical scoped evidence. Timing is not complete-import overhead acceptance.
Validation: fresh full Linux AMD64 backend/vet and focused API/security/malware/runner races; all real-ClamAV fixtures, including the numeric and prior raster/JPEG/large-media/limit controls; coverage-guided generated-only fuzz, 24,284 executions over a requested 15 seconds; nine CI tests; docs/workflow/tracked-path contracts; exact exclusion of 2,035 private identity/digest strings; redacted staged-secret scan; whitespace checks. Early diagnostic iterations were corrected and excluded from final successful evidence. No new production-image/browser/provider/source-policy or release/performance clearance is claimed.
New private readable CAMERA-MERGE-FIELDS-UPDATE.txt and detailed JSON are beside the examination ZIP, including per-file matched/absent status. Prior reports/assessments and source files remain unchanged.
Next step is implementation of one complete, versioned camera profile: finish typed handling of the rest of the camera log and HDRP notes, then integrate the reviewed merge fields, bounded raw previews, standard Exif fields/thumbnails and narrow ICC subset into the existing JPEG path. Share retained-header/expansion/component/deadline budgets and scan decoded components sequentially after original AV joins. The planned treatment records missing verbose notes as optional corroboration, while still requiring the same complete binary profile and mandatory AV; partial or contradictory notes fail. Absence alone never grants approval. Require full recorded decision-path, generated detection/outage/budget/failure and real mixed-file tests before changing eligibility or issuing compatibility evidence. Unknown schemas and other warnings/errors remain held.
Native checks on this exact head: pending (13 checks). PR #19 stays WIP. No original-library rescan, recovery/selective import, admin controls, real import, deployment/restart, merge, closure or gate waiver.
Findings and supported-treatment details · Sanitized measurements · Validation record.
Beginning the owner-approved complete camera-profile integration from
1e6a5d6/ comment 666. Current maina279a1bwas fetched and is already included; clean-tree canonical backend/vet/focused race baseline passes (cached exact tree). Implement typed bounded camera-log fields, HDRP notes/raw previews, reviewed Exif and narrow ICC validation through the existing JPEG path, shared resource budgets and sequential component AV after original AV joins. Missing verbose merge notes remain absent optional corroboration; contradictory/partial notes fail. Version eligibility/evidence once the complete profile and generated/recorded/real mixed-file checks pass. Preserve original warnings, private reports, unknown-data holds and existing panorama behavior. No recovery/admin/import/deployment/restart/merge/closure.Local policy-13 integration checkpoint (not yet published): the first complete managed-engine recorded test passes all 24 targeted photos (21 camera, three existing panoramas), original AV warnings retained, 480 component responses / 13,659,321 bytes checked, saved findings verified and all source hashes matching the private examination copies. The test-only archive receives its integrity receipt; there is no importer job or original-collection approval. Two newly promoted numeric conversions are being fixed rather than added to the source allowance. The complete production audit also found three lowercase encoded camera-note records previously treated as text, plus structured burst tags; these now have explicit bounded parsing and separate decoded coverage. No opaque-text exemption was added. Final actual-engine negative controls pass locally; broader source/race/fuzz/isolation/UI checks and a final exact-tree recorded run remain before publication. Main/deployed pilot unchanged; recovery/admin work and import/deployment remain deferred.
Published
bd677ba900on the security feature branch, based on unchanged current maina279a1b. The 21 targeted camera photos now pass the complete supported checks without conversion or changes to their bytes. The three panorama controls still pass. This is implemented, locally tested and published; it is not deployed and does not complete all of #12.Policy 13 adds
JPEG_CAMERA_V1: explicit bounded camera-log and additional-note schemas, HDRP/gzip/raw-preview decoding, the reviewed standard Exif fields/thumbnails, narrow ICC validation and structured burst metadata. Original AV/integrity/framing/hashing shares one forward source read; sequential component AV begins after the original request joins. Header/expansion/wire/preview limits plus a shared 64-request / 32-MiB per-file component plan bound work. Unknown schemas, arbitrary byte containers, recursive content, other warnings and failed checks stay held. No engine alert is disabled and no rewritten original is scanned to manufacture a clean reply.Readable camera notes remain optional corroboration: present sets must match all named details within their fixed printed rounding intervals; duplicate/partial/contradictory sets fail. Entirely absent notes are explicitly recorded as absent. The complete audit found three additional lowercase encoded records the earlier exploratory parser had treated as text, plus structured burst tags. Those now have explicit bounded decoding/validation and AV rather than an opaque-text exemption. This adds 15 requests / 798 bytes beyond the earlier diagnostic; historical reports remain unchanged and the new report supersedes its incomplete nested-field interpretation.
Final actual managed-engine recorded test: 24 originals / 68,622,039 bytes, with all 21 camera files and three panoramas allowed. 480 component requests / 13,659,321 bytes complete without detection. Original JFIF-position warnings remain recorded separately from
clean_compatibleand the applicable profile code. Eighteen camera files record matched notes; three record absent notes. Every saved outcome/source hash matches the private examination copy, and the result manifest verifies. The test-only archive receives its integrity receipt; there is no importer job or approval for the original collection.Final scanner wall time 21.120 s / isolated command 21.918 s, including startup under variable host/cache conditions. This is not a paired complete-import overhead comparison. Existing offline non-root one-CPU / 5-GiB / 2-GiB-temp limits, fresh unchanged managed definitions, source immutability and exact reader removal before locked-alias release all hold. No original-library rescan or real destination is involved.
Checks actually run: full Linux AMD64 backend/vet and focused API/security/malware/runner races; all real-ClamAV fixtures, including clean ordinary/late-JFIF camera profiles and harmless signatures encoded in previews, packed numeric leaves, the additional note and normalized XML labels; generated schema/type/count/bound/nonfinite/unknown/duplicate/truncation/gzip/Exif/ICC/Bag/rounding/short-read/outage/cancellation/recursion/mixed-refusal cases. Valid numbers and matching forged notes still require AV. Generated-only guided fuzz passes 154,269 executions over a requested 20 seconds, without private seeds.
Four canonical images build. The final actual required-mode Docker workflow with race controller passes in 46.09 s, using a reproducible generated camera JPEG through analysis, authenticated saved-check reuse, preview and synthetic upload. Acknowledged partial coverage, held refusal, mutation/replay/isolation/cleanup pass as well. This uses a synthetic Immich endpoint, never a real account or owner media. The old mixed diagnostic entry point now forwards to the current profile test rather than asserting historical holds.
UI lint/build and the private-findings mobile browser pass: ordinary wording stays “extra photo checks completed”; matched/absent notes are inside optional technical details. Source policy: 76 files / 16,626 lines / 43 reviewed findings, zero suppressions and no new allowance entries. Two promoted numeric casts were fixed in code. Source govulncheck, zero-advisory npm audit, nine CI tests, docs/workflow/private-path contracts, exact exclusion of 2,035 private identity/digest strings, redacted staged-secret scan and formatting checks pass. No native/release/provider/image-advisory/performance gate is waived.
New private readable CAMERA-POLICY13-VALIDATION-UPDATE.txt and detailed JSON are beside the examination ZIP. Per-file identities, paths, metadata values, content digests, original manifests and raw findings remain outside public Git/issues. Original media, ZIPs and historical saved reports remain unchanged. Policy 13 invalidates older reusable eligibility and requires a fresh explicit analysis when this feature is available; there is no automatic rescan fallback or original-collection approval.
Next: use the existing private format census to select the largest remaining held JPEG family, then add bounded trailer/missing-marker/composite validation and generated/actual detection/failure controls before another versioned eligibility change. The deliberately unwanted game package and bitmap/content mismatch retain manual-review decisions. Keep recovery/selective import and later admin scanning settings/guidance deferred until scanning readiness.
Native checks on this exact head: pending (13 checks). PR #19 remains WIP. Main/deployed pilot unchanged; no actual import, deployment/restart, merge, issue closure or broader-readiness claim.
Implemented camera policy · Sanitized measurements · Validation record.
Beginning the owner-approved next held-JPEG family, from
bd677ba. Rebase decision: check/integrate main now, before further scanner changes. Fresh origin/main remainsa279a1band is already an ancestor (0 main-only / 35 feature-only commits); Git rebase origin/main reports up to date. The published Go 1.27.3/package updates are already included; no history rewrite or package downgrade is needed. A local pre-step backup ref is retained and canonical Linux AMD64 backend/vet/focused race baseline passes. Next investigate the 654 recurring aecDebug trailers against producer source and bounded private metadata, with generated malformed/limit/detection controls. A repeated marker is not a schema or safety proof. Production policy stays 13 and existing trailer holds remain until complete validation is justified and tested. No private identities/values/manifests in public Git/issues, original-media changes, recovery/admin work, real import, deployment/restart, merge or closure.Trailer investigation checkpoint: an offline one-CPU / 256-MiB reader inspected only 654 known trailer ranges (15,206,525 bytes), with no full-library reread, AV or decoder. Independent bounded Go parsing accounts for 421 canonical variable-length envelopes (331 with focus data, 90 without); 233 focus layouts remain unrecognized. This explains the failed historical fixed-16-bit hypothesis. Outer framing does not validate payload contents: all 654 production holds remain under policy 13. Public camera code documents opaque debug buffers, not a complete schema for this observed wire format. Generated bounds/truncation/unknown-content/race checks pass locally; broader current-tree checks are running. New readable/JSON private findings are beside the examination ZIP. No identities, payloads or manifests are posted. Next: settle focus framing and typed payloads, check headers independently, then prove full bounded original/component coverage before changing eligibility. The fetch/rebase check found no new origin/main commits; all published Go/dependency updates are already included.
Published
4189520ecb: a bounded diagnostic and regression checkpoint for the next largest held JPEG family. This explains more of the attached camera data; it does not clear the 654 files yet.Rebase decision: check/integrate now. A clean-tree backup and fresh fetch confirmed published
origin/mainremainsa279a1bd77c0f600f705856077b6a6312457b87b, already an ancestor of the pre-increment security headbd677ba(35 feature-only, zero main-only commits).git rebase origin/mainreported up to date. Go 1.27.3 and main's published package updates are already included. Full baseline tests passed before edits. Main was checked again after publication and is unchanged. No history rewrite or force push was needed; unpublished changes cannot be included by this check.Finding: the old fixed-16-bit size hypothesis was wrong for this family. A new Go diagnostic follows canonical variable-length sizes after the exposure, optional focus, and white-balance markers, advancing only by declared lengths. It accounts for 421 complete outer envelopes: 331 with focus data, 90 without. 233 focus layouts remain unrecognized. Marker text inside a block cannot move a boundary, and unknown focus data is not skipped to the next familiar label.
Even a completely framed envelope can contain arbitrary bytes. Typed payload semantics, embedded representations and applicable antivirus coverage remain unfinished for all 654. Public camera producer code carries opaque private debug buffers but does not establish a complete schema or producer identity for this observed format. No generic numeric-data assumption, magic-prefix allowance, metadata stripping, conversion or warning suppression was introduced. Production policy remains 13; all 654 holds remain. The previously validated 21 camera photos and three panorama controls are separate and unaffected. These findings do not establish corruption or infection.
Bounded private work: an offline non-root one-CPU / 256-MiB reader copied only 654 known trailer ranges / 15,206,525 bytes in 4.783 s from the stored examination ZIP. It verified current range lengths, preceding image-end markers and recorded prefixes. The earlier complete verified census remains the integrity basis: this is not a new full-content CRC/digest check. Source sharing locks, read-only aliases and exact reader removal before alias release passed. The independent Go snapshot diagnostic took 3.325 s including command/cleanup, with 1-MiB/trailer and 700-file / 32-MiB batch limits. No AV, decoder, preview, original image-stream reread or original-library rescan ran. These small diagnostic timings are not complete-import performance acceptance.
Checks actually run: canonical Linux AMD64 full backend/vet/focused API/security/malware/runner races both before and after the new diagnostic; all malware tests and an explicit new diagnostic race test. Generated cases cover canonical one/two/three-byte sizes, every truncated prefix, marker text inside payloads, unknown/reordered/duplicate blocks, trailing junk, zero/overlong/overflowing/incomplete lengths, limits and cancellation. An executable-like payload deliberately remains only
framing_complete_payloads_unvalidated; this helper has no allow verdict or authorization path. Nine CI tests, docs/workflow/private-path contracts, formatting/diff checks, exact exclusion of 2,035 private identity/digest strings and redacted staged-secret checks pass. Only test/docs/aggregate files changed. Earlier policy-13 runtime engine/image/source/UI/workflow evidence remains applicable to unchanged runtime code; those suites were not repeated for this diagnostic-only increment.New private readable DEBUG-TRAILER-INVESTIGATION.txt and detailed JSON beside the examination ZIP map every affected file to its findings. Names, paths, metadata values, raw snapshots, content digests and manifest rows remain outside public Git/issues. Originals and historical results remain unchanged.
Next steps, in order:
Native checks on this exact head are pending (13 checks). PR #19 stays WIP. Main/deployed pilot unchanged; no real import, deployment/restart, merge, closure, performance acceptance or gate waiver.
Investigation and source limits | Sanitized measurements | Validation record.
Continuing from
4189520after reading the latest #12/#14 comments and PR #19 steering. Fresh fetch/rebase is still up to date with origin/maina279a1b. Scope: identify the alternate focus layout and typed debug payload framing using existing small private trailer snapshots, then evaluate full header compatibility separately. No repeated whole-library scan. Add independent bounded diagnostic/negative checks before considering production support; all 654 holds and policy 13 remain until complete validation and AV coverage are demonstrated. Keep private filenames, bytes, values and per-file mappings outside public Git/issues. Recovery/admin work, real import, deployment/restart, merge and closure remain deferred.Published
30fdec7e3d. The outer layout is now accounted for in all 654 targeted trailers. The 233 previously unexplained focus blocks use a compressed wrapper. This is a diagnostic checkpoint; payload semantics and full security coverage remain unfinished, so all 654 retain their production holds under policy 13.What we learned
Bounded private validation
The Go snapshot command reproduces the historical 421/233 result with its original helper, then accounts for all 654 with the wrapper-aware helper. It takes 4.286 s including cleanup, under one CPU / 256 MiB, with no original media ZIP mounted. Limits include 1 MiB per trailer/per-file payload total, 700 snapshots / 32 MiB per batch, 128:1 gzip expansion, 4,096 root fields/block, bounded output and deadlines. An exploratory recursive-shape inventory failed without a complete report; no inferred nested schema or acceptance evidence is taken from it.
The independent header test takes 6.84 s under one CPU / 512 MiB, retaining 73,721,923 header bytes and reading 75,051,008 bytes including bounded read-ahead. The final 4,096-byte read may include a small amount of image data; no complete image or original-library reread occurs. Windows sharing locks/read-only aliases, unchanged-source checks and exact reader removal before alias release pass. Earlier complete verified census integrity remains the basis. There is no fresh full-content digest/CRC check, private-data AV, pixel decode, conversion, preview or import in these diagnostics. These timings are not full security overhead or achievement of the 15% target.
Checks actually run
Full canonical Linux AMD64 backend/vet/focused API/security/malware/runner races pass after the unchanged-main baseline. All malware tests and targeted diagnostic races pass. Generated cases cover plain/compressed/absent focus, every truncation, marker text, canonical/zero/overlong/overflowing/incomplete lengths, gzip CRC/size/suffix/second-member/expansion failures, malformed root wire and field budgets, unresolved byte fields, visitor failure and cancellation. A complete generated camera-profile JPEG with a trailer receives no receipt or compatibility authority with either raw-clean or warning responses.
Actual ClamAV 1.4.6 generated-only controls preserve the clean original's JFIF warning and detect the harmless compressed canary in both the original and expanded focus payload for this fixture. The control therefore does not assume all suffixes are hidden from original AV. Synthetic definitions are disposable and separate from managed private-data definitions; this is not a private-data scan or freshness claim. Nine CI tests, docs/workflow/private-path contracts, formatting/diff checks, exact exclusion of 2,035 private identity/digest strings and redacted staged-secret checks pass. Only tests/docs/aggregates changed. Runtime dependencies/policy are unchanged; earlier runtime source/UI/image/managed-engine/workflow checks were not repeated or waived.
New private readable DEBUG-PAYLOAD-INVESTIGATION.txt and detailed JSON beside the examination ZIP map every affected file to both diagnostics. Raw bytes, metadata values, filenames/paths, digests and manifest rows remain outside public Git/issues. Originals and historical reports remain unchanged; the new report supersedes the earlier unresolved-focus interpretation.
Next, in order:
Fresh fetch/rebase remains up to date: published main
a279a1bis already included in pre-increment security head4189520(36 feature-only, zero main-only commits), with Go 1.27.3/package updates present. Main is unchanged on the final API check. No history rewrite or force push. Native checks on this exact head: pending (13 checks). PR #19 stays WIP. No real import, deployment/restart, merge, closure, readiness/performance claim or gate waiver.Current investigation | Sanitized measurements | Validation record.
Continuing from
30fdec7after the latest #12/#14 comments and PR #19 steering. Fresh fetch/rebase is up to date with maina279a1b(0 main-only / 37 feature-only commits). Scope: locate the exact schema gaps in the 396 camera-log header variants, retain bounded private metadata snapshots to avoid repeated archive reads, and relate supported named metadata evidence to trailer fields where justified. Add generated bounds/unknown-field/refusal checks; diagnostic wire shapes do not authorize contents. All 654 holds and policy 13 remain until complete typed validation and AV coverage are demonstrated. No private identities/values/manifests in public Git/issues, original changes, recovery/admin work, real import, deployment/restart, merge or closure.Published
5bb88ce6a6. The 396 camera-log failures now have explicit schema-gap findings, and a newer printed table explains the note-comparison failures. This is still research/test support: production policy stays 13 and all 654 original holds remain.Findings
Maximum merge weightis a different quantity from the olderMerge frame_weights. An equality hypothesis matched 166 files coincidentally and disagreed in 209. It is rejected before production use. The final diagnostic parses the maximum column's bounded decimal grammar but never claims its binary correspondence is verified. Every table receives an explicitly partial status, even where numbers coincide. No invented equality should become an upload requirement; future support must treat each quantity according to its own representation.Bounded private work
One sharing-locked, read-only metadata pass saves 654 header snapshots / 73,721,923 bytes, reading 75,051,008 bytes including bounded look-ahead, in 13.77 s. Snapshot output is capped at 128 MiB. Unchanged-source checks and exact reader removal before alias release pass. Subsequent iterations mount only those snapshots, not the media ZIP: 354,706 log bytes, with a final retained-header command/cleanup time of 10.286 s under one CPU / 512 MiB. Log, batch, selected-note text, field, depth and deadline limits apply. This avoids repeated original-archive reads while developing the validators. No complete-image/library reread, private-data AV, pixel decode, conversion or import occurs. Earlier complete census verification remains the integrity basis; there is no new full digest/CRC check or complete-import performance acceptance.
Checks actually run
Full canonical Linux AMD64 backend/vet/focused API/security/malware/runner races pass after the unchanged-main baseline. Malware tests and focused gap/table/header/refusal races pass. Generated cases cover missing/unknown fields, count/byte/wire budgets, ambiguous/partial/mixed/duplicate tables, column counts and indices, precision and contradictory factors, unsupported notes, cancellation and private snapshot-output failure. A generated complete camera with these notes, with or without an additional log field, receives no receipt or compatibility authority under raw-clean and JFIF-warning responses.
Actual ClamAV 1.4.6 detects a harmless signature encoded as plausible numeric merge factors, despite forged readable table factors agreeing. Partial text consistency cannot substitute for AV. This uses disposable synthetic definitions and no private media. Nine CI tests, docs/workflow/private-path contracts, formatting/diff checks, exact exclusion of 2,035 private identity/digest strings and redacted staged-secret checks pass. Only tests/docs/aggregates change; no runtime, dependency, UI or eligibility changes. Earlier runtime image/source/UI/managed-engine/workflow evidence is not presented as newly repeated or waived.
New private readable CAMERA-LOG-VARIANTS-UPDATE.txt and detailed JSON beside the examination ZIP map every affected file to the latest findings. Values, actual names/paths, raw snapshots, digests and manifest rows remain outside public Git/issues. Originals and historical reports remain unchanged.
Next, in order:
Fresh fetch/rebase is up to date with published main
a279a1b, already included in pre-increment security head30fdec7(37 feature-only, zero main-only commits). Its Go/package updates remain present; main is unchanged on the final API check. No rewrite/force push. Native checks on this exact head: pending (13 checks). PR #19 stays WIP. Recovery/selective import and later admin settings plus security-enabled duration/leave-return/email guidance remain deferred until scanning readiness. No real import, deployment/restart, merge, closure or readiness/performance waiver.Log/table findings | Sanitized measurements | Validation record.
Continuing the approved next steps from
5bb88ceafter reading the latest #12/#14 comments and PR #19 steering. Fresh origin/main remains included (0 main-only / 38 feature-only commits); rebase is up to date. Implement an explicit bounded candidate for the recorded added-log paths/presence variants, retaining unknown-field/type refusal and numeric representation AV; complete distinct note handling without aliasing maximum/ordinary weights. Use existing tiny decoded-log and retained-header snapshots, not another archive/library read. Keep candidate results separate from production eligibility until full metadata/trailer/AV/budget/evidence paths pass. No private identities/values/manifests in public Git/issues; policy 13 and current holds remain during this work. Recovery/admin controls, real import, deployment/restart, merge and closure remain deferred.Local candidate checkpoint before final publication: all 636 reachable saved logs pass the new explicit bounded field/path/presence validator. Fourteen extended motion records require a u32 only in that named layout; base boolean rules remain. Numeric AV reconstruction is sorted by semantic field number, preserving repeated order, so reordered tags cannot evade coverage. Managed fresh unchanged definitions complete 2,544 representation requests / 684,774 bytes without detection (54.783 s including command/cleanup), covering decoded wire, normalized scalars, weights and factors. No original media or ZIP is mounted in that check; no photo approval/receipt. Generated tag-separated canaries demonstrate raw-wire-only coverage can miss a signature that normalized scalar AV catches. Generated-only fuzz passes 193,270 executions. The last 20 header cases are now narrowed to two plain UTF-8 note blocks with no declared encoded previews, 15 DisableSuggestedAction list properties and three BurstPrimary attributes; these do not yet change production eligibility. A zero-preview text candidate retains full-text AV requirements. Broader current-tree checks and privacy review are being completed; policy 13/all 654 holds remain. No private names/values/manifests are published or originals changed.
Main advanced during candidate validation to
532f520(three importer/stack commits). Preserved the published4e0afefcheckpoint in a local backup and rebased all 39 security commits onto that main. Conflict review retains the exact checked stack patch, duplicate destination-ID handling, collapsed-stack warning behavior, both unsupported Takeout reporting hooks/tests, and both matching-source CI checks. The malware, archive-verification, private-report, module and UI sources are identical to the saved security checkpoint; no private media is rescanned merely for Git history changes. The combined importer is now distinctly labeled bridge.7. Full combined backend/race/importer/image/source/workflow checks are being run before publishing the rewritten branch, using a lease against the exact previously published head. Policy 13/all 654 holds remain; no deployment, real import, recovery/admin work, merge or closure.Published
24db3ffdf4, including the log-schema candidate (rebased1eb37d6, formerly4e0afef) and current-main integration. Production policy remains 13 and all 654 original photo holds remain. This completes the explicit log candidate and combined-main validation checkpoint; it does not approve those photos or finish #12.Log findings and scoped antivirus
All 636 saved logs / 354,706 bytes reachable through the understood XMP layouts pass explicit bounded field/path/type/presence validation. Added records have named schemas and exact table dimensions; unknown IDs, duplicates, partial patterns, nonfinite values, overflows and budget failures refuse. Fourteen extended motion parents select a bounded u32 child only within that complete layout; older boolean constraints remain. Maximum weights remain distinct from ordinary weights, with no invented equality rule.
The candidate reconstructs 297,116 numeric bytes in semantic field-number order, preserving repeated order. Reordering independent wire tags cannot change this representation. Actual ClamAV generated controls detect a harmless signature split across numeric fields after normalization even where raw-wire scanning misses it. Valid numbers and matching readable notes still require AV.
Fresh unchanged managed definitions complete 2,544 representation requests / 684,774 bytes, all without detection or incomplete/error replies: decoded wire, normalized numeric data, weights and factors. Exact consumption/EOF is required for a clean reply. One offline non-root worker uses one CPU / 5 GiB / 2 GiB temporary space and bounded request/byte/deadline budgets; command/cleanup is 54.783 s. No source ZIP or original photo is mounted. This is scoped log evidence, not full photo/header/trailer AV, a collection receipt or complete-import overhead acceptance.
The remaining 20 header cases are narrowed using 1,675,465 retained bytes: 15
DisableSuggestedActionlist properties, threeBurstPrimaryattributes, and two readable UTF-8 note blocks with no declared encoded previews. A zero-preview text candidate retains mandatory full-text AV; declared encoded records cannot take a plain-text fallback. Its generated real-AV canary is detected. The two actual note texts still await integrated private AV; complete property allowlist integration is also pending.Generated negative/refusal controls cover graph/table shapes, required fields, scalar widths, tag order, byte/field/depth/count limits, cancellation, scanner outage and unread clean replies. Complete staged camera fixtures receive no production receipt or compatibility authority under raw-clean/JFIF-warning responses. Generated-only fuzz passes 193,270 executions, requested 20 s / observed 21.052 s, with no private seeds.
Rebase and combined checks
Main advanced during work to
532f5202fa. All 39 security commits were rebased onto it, followed by integration commit24db3ff; the prior published checkpoint is retained locally. Exact main stack helper/tests/patch and LF handling remain, as do both Takeout reporting hooks/tests and matching-source CI checks. Combined importer 0.32.0-bridge.7 has matching binary/source/build/license labels. Actual tests cover collapsed/valid stacks, genuine stack errors, disabled job control, discards, reports and interruption/retry.Production Go, modules, UI, malware/archive/private-report and staged-validator sources are byte-identical to the saved security checkpoint; main's additional API regression tests are retained. Range/history review finds no unresolved markers. Git identity changes do not justify another original-library read. The final fetch confirms main is unchanged and included (0 main-only / 40 feature-only commits). Publication uses an exact force-with-lease against
4e0afef, preserving other remote changes.Full combined Linux AMD64 backend/vet/focused API/security/malware/runner races, importer compatibility, all actual-engine generated fixtures and four canonical image builds pass. Matching-source export/checksum/both hook sets/vendor materials verify; fresh importer binary govulncheck passes policy. The actual required-mode Docker workflow with a race controller passes in 56.14 s with fresh managed definitions, generated camera analysis, authenticated reuse, preview and synthetic upload, held refusal, acknowledged partial coverage, mutation/replay, isolation and cleanup. This uses no real destination or user media; timing includes host/startup effects. Initial local harness setup errors were corrected without a product change, freshness override or security bypass.
Nine CI tests, final docs/workflow/private-path contracts, formatting/diff checks, exact exclusion of 2,035 private identity/digest strings, staged and full-history redacted secret checks pass. Unchanged production source/UI retains earlier source-policy/source-govulncheck/browser evidence; those checks are not represented as freshly repeated. Native/release/provider/final-image-advisory and representative complete performance/readiness gates remain open; local passes do not waive them.
Private reports and next steps
New owner-readable CAMERA-LOG-SCHEMA-UPDATE.txt and detailed JSON beside the examination ZIP map the files to findings. Actual names/paths, scalar values, snapshots, digests and manifest rows remain outside Git/public issues. Originals and historical reports remain unchanged.
PR #19 remains WIP. Recovery/selective import and later admin enablement/posture/block-vs-report/tunables plus security-enabled duration/leave-return/completion-email guidance remain deferred until scanning readiness. No original-library rescan, real import, deployment/restart, owner-task interruption, merge or issue closure.
Bounded candidate | Sanitized measurements | Combined main validation | Validation record.
Native status on this exact published head: pending (13 checks). These native checks remain separate from completed local validation.
Continuing the approved remaining metadata/trailer work from
24db3ffafter latest #12/#14 and PR #19 steering. Develop explicit bounded property/note variants and trailer payload checks using retained private snapshots; preserve unknown-container refusal and require original/reconstructed AV before eligibility. Keep all private values/identities/reports outside Git/issues. Production policy 13 and all 654 original holds remain while candidate checks are incomplete. No recovery/admin work, real import, deployment/restart, merge or closure.Remaining metadata/trailer checkpoint is locally validated and undergoing final publication review. Candidate XML parsing accounts for all 654 saved headers. Fifteen of the 20 remaining cases have complete staged XMP component plans, with 365 managed fresh unchanged-definition AV requests / 7,584,496 bytes completed without detection, including both plain-text notes. Five cases expose further byte-field schemas: two in the numeric log and three richer encoded notes. They remain unsupported; no recursive guesses or omitted required fields. All 654 saved trailers / 1,872 blocks complete 2,105 wire/gzip representation requests / 20,126,895 bytes without detection. Exact root-record correspondence gives zero matches across 652 supported header logs; two unsupported logs are excluded. Trailer nested types still need explicit schemas. Initial broad plan construction stopped at its 256-MiB work bound; focused completion retains that bound and avoids repeating every preview decode. Full backend/vet/races, all actual-engine fixtures, source policy/govulncheck and generated-only guided XML fuzz (204,358 executions) pass. New readable/JSON private reports are beside the examination ZIP. Production policy 13/all 654 holds remain. No private values/identities/manifests published, originals changed, real import, recovery/admin work, deployment/restart, merge or closure.
Published
2cfea71066. The remaining-property work is staged and tested, with more precise private findings. No additional photo is approved: policy stays 13 and all 654 original holds remain.The next action is explicit: establish the types of the newly exposed byte fields in two camera logs and three richer encoded notes, then finish the trailer schemas. A clear AV response or readable wire framing cannot replace those checks.
What completed:
Work remains bounded. One offline non-root worker per batch uses retained snapshots only, read-only fresh managed definitions and existing per-file 64-request / 32-MiB component limits plus batch limits. Clean replies require exact consumption/EOF; outages, early replies and cancellation fail. Metadata command/cleanup is 23.125 s, trailers 32.088 s, under one CPU / 5 GiB / 2 GiB temporary space. No source ZIP, full photo or original library is mounted; there is no fresh complete-file CRC/hash or receipt. Timings include startup/cleanup and are not complete-analysis overhead acceptance.
The initial broad component-plan attempt stopped at its 256-MiB decoded-work bound. The completed check restricts expensive decoding/AV to the twenty cases and parses XML fields in the other headers. The limit was retained. Focused research can read just 1,675,465 snapshot bytes, avoiding repeated preview decoding across the whole set.
Validation actually run: full canonical Linux AMD64 backend/vet/focused API/security/malware/runner races; malware and focused candidate/relation tests; all actual-ClamAV generated fixtures; complete-file production refusal/no-authority controls; and generated-only coverage-guided XML fuzz, 204,358 executions, requested 20 s / observed 21.055 s, with no private seeds. Source policy passes with 76 production files / 16,638 lines / 43 reviewed findings, zero suppressions or new allowances; source govulncheck passes. Nine CI tests, final documentation/workflow/private-path contracts, formatting/diff checks, exact exclusion of 2,035 private identity/digest strings, staged and full-history redacted secret checks pass. UI, dependencies and importer are unchanged; earlier browser/importer/complete-workflow evidence is retained without claiming a new run or waiver.
New private METADATA-TRAILER-CHECKS-UPDATE.txt and JSON beside the examination ZIP map all 654 files to their exact scope. Actual identities, paths, values, snapshots, digests and manifest rows remain outside Git/public issues. Originals and historical reports remain unchanged.
Next, in order:
Fresh fetch confirms main
532f520remains included (0 main-only / 41 feature-only commits); its Go/dependency/stack fixes and combined bridge.7 importer remain. Normal push, no history rewrite. PR #19 stays WIP. Native/release/provider/final-image-advisory/performance/readiness gates remain open. Recovery/selective import and later admin enablement/posture/block-vs-report/tunables plus security-enabled duration/leave-return/completion-email guidance remain deferred. No real import, deployment/restart, owner-task interruption, merge or closure.Current findings and limits | Sanitized measurements | Validation record.
Native status on this exact published head: pending (13 checks). Completed local validation does not waive these native checks.