WIP: Layered ingestion security and measured analysis costs #19
Loading…
Reference in a new issue
No description provided.
Delete branch "issue-12-isolated-malware-scanning"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Large Google Takeout exports need meaningful security checks without repeated full scans or a blanket small-file ceiling. This WIP adds layered validation, isolated managed antivirus, complete private per-file findings and authenticated reuse. Scope follows #12, #14 and comment 332. Overall scanning, performance and release acceptance remains unfinished.
Implemented behavior:
532f520, guided stages, SSO/dependency/locale fixes and operation-bound findings/consent remain integrated. Importer bridge.7 combines main's duplicate/stack fixes with Takeout reporting/private manifests and matching source/license notices.Current staged checkpoint (
2cfea71):One offline non-root worker per batch uses saved snapshots only, fresh immutable-generation managed definitions and bounded per-file/shared requests, bytes, expansion and deadlines. Metadata command/cleanup is 23.125 s, trailers 32.088 s, under one CPU / 5 GiB / 2 GiB temporary space. No source ZIP/full photo is mounted. An initial broad plan hit the 256-MiB decoded-work bound and stopped; the completed check focuses expensive work on the twenty cases while retaining that bound. These timings are not representative complete-import overhead or achievement of the 15% target.
Current validation passes: canonical full Linux AMD64 backend/vet/focused API/security/malware/runner races; malware/candidate/relation and production-refusal controls; all real-ClamAV generated fixtures; source policy (76 files / 16,638 lines / 43 reviewed findings, zero suppressions/new allowances); source govulncheck; generated-only guided XML fuzz (204,358 executions, requested 20 s / observed 21.055 s, no private seeds); nine CI tests; docs/workflow/privacy contracts; formatting; exact private-identity exclusion; staged/full-history redacted secret checks. UI/importer/dependency sources are unchanged; earlier browser/importer/complete-workflow evidence is retained without claiming a new run.
Historical runtime evidence remains separate: combined-main checkpoint
24db3ffpassed four canonical image builds, actual importer compatibility (collapsed/valid stacks, real errors, disabled job control, discards, reports and retry), matching-source export and binary govulncheck, plus the race-controller required-mode Docker workflow in 56.14 s with generated camera analysis/reuse/preview/synthetic upload, held refusal, partial consent, mutation/replay/isolation/cleanup. The policy-13 targeted archive passed all 21 camera photos and three panorama controls with 480 component requests. Earlier whole-set assessment visited 67,013 members, with 66,302 clean AV, sixteen acknowledged oversized exceptions, 688 JPEG holds and seven type blocks. These historical scopes do not approve the original collection under the current feature or establish complete-analysis overhead acceptance.Next: explicit added log/numeric-note byte-field types, then complete exposure/focus/white-balance trailer schemas. Prove complete original/reconstructed AV, hidden-content/error controls, one-read capture, shared budgets and authenticated saved evidence before new versioned eligibility; measure representative complete analysis afterwards. Native/release/provider/final-image-advisory/performance/readiness gates remain open. Bitmap mismatches and game packages retain review decisions. Recovery/selective import and later administrator enablement/posture/reporting/tunables plus security-enabled duration/leave-return/completion-email guidance remain deferred until scanning readiness. No real import, deployment/restart, owner-task interruption, merge, closure or gate waiver.
Current findings and limits | Sanitized measurements | Main integration | Validation record.
Immich-Bridge: Lightweight, Risk-Based Malware Scanning Architecture
Before making any code changes, fetch the latest "main" and rebase the existing issue branch onto it.
Do not begin implementing the scanning changes until the rebase and baseline validation are complete.
Redesign malware scanning to provide meaningful security protection without preventing legitimate Google Photos and Apple Photos exports from being imported.
The current problem is ClamAV's file-size and scanning limits. Large legitimate media files, particularly videos, can exceed those limits. Mandatory successful ClamAV scanning of every file would make normal photo-library imports impractical.
The solution must:
Core principle: Every file must pass mandatory lightweight security validation. Additional scanning should be performed only when applicable or justified by risk.
Tier 1: Mandatory Lightweight Validation
Apply to every archive entry using the existing inspection pipeline.
Validate:
Reject unsupported, executable, or suspiciously disguised files according to the established security policy.
Do not rely on filenames or extensions alone.
Avoid full-file decoding, redundant reads, and additional extraction passes.
Tier 2: Conditional ClamAV Scanning
Retain ClamAV as the primary known-malware detection engine.
For eligible files:
For oversized files:
Scanner failures are not size exceptions.
ClamAV crashes, unavailable services, timeouts, and unexpected scan errors must be handled separately from documented size limitations. Such failures must not silently authorize import.
Tier 3: Selective Media Validation
Use deeper media inspection only when justified by file characteristics or suspicious findings.
For videos, consider "ffprobe" to validate:
Avoid full video decoding and transcoding.
Do not run expensive media validation against every file by default.
Image decoding with "libvips" is deferred unless testing demonstrates a meaningful security benefit beyond existing validation.
Media validation must execute within the existing restricted worker environment with bounded CPU, memory, execution time, and concurrency.
A successful media probe does not establish that a file is malware-free.
Security classification must be separate from the final import decision.
Condition| Required behavior
Valid supported media; ClamAV clean| Allow
Known malware detected| Block
Archive traversal or unsafe entry| Block
Executable disguised as media| Block
Oversized, structurally valid supported media| Allow under explicit risk-based exception
ClamAV unavailable or unexpected failure| Hold and retry; do not silently bypass
Suspicious or inconsistent media structure| Escalate or quarantine
Unsupported or unreadable content| Reject or quarantine according to policy
Implement explicit per-file results, including:
These are proposed logical outcomes; adapt them to existing application conventions rather than creating redundant status systems.
Preserve the distinction between detection, incomplete scanning, and policy authorization.
Do not automatically treat a single infected file as proof that all other files in the archive are infected. Preserve per-file findings and support a safe quarantine decision. Review existing import behavior before choosing whether the overall job is blocked or eligible files can continue.
The solution must remain practical for archives containing tens of thousands of files and hundreds of gigabytes of media.
Requirements:
Use the existing approximately 23,000-file, 150 GB import benchmark as a representative workload.
Performance target: Aim for no more than 15% additional end-to-end analysis time compared with the existing baseline, while retaining mandatory security checks. Treat this as a measurement target, not permission to weaken security.
If the target cannot be met, report the measured bottlenecks and tradeoffs rather than silently reducing coverage.
Preserve the existing validation-worker security boundaries.
Verify that:
Do not introduce additional container infrastructure unless a demonstrated security requirement cannot be satisfied by the existing architecture.
Remember that downstream Immich components will process original media. Validation reduces risk but does not eliminate vulnerabilities in downstream parsers.
Integrate with the existing reporting and persistent logging architecture.
For each file, record:
At the job level, provide:
User-facing reporting: Keep messages understandable to nontechnical users. Explain that some large media files could not be antivirus-scanned but passed other required checks.
Technical reporting: Preserve detailed diagnostics for troubleshooting and security auditing.
Avoid presenting an incomplete scan as a successful clean scan.
Extend the existing automated test suite rather than introducing a separate testing framework.
Functional tests
Security tests
Regression and performance tests
Do not implement the following without a demonstrated need and separate approval:
After completing the mandatory rebase:
Work incrementally. Do not perform an unnecessary rewrite of the existing validation architecture.
This issue is complete when:
Guiding Principle
Build a secure media-ingestion pipeline, not a general-purpose antivirus platform.
Use inexpensive validation universally, ClamAV where technically applicable, and deeper analysis only when justified.
Maintain explicit security decisions, preserve isolation, and never silently convert an incomplete or failed scan into a clean result.
Security must be meaningful, measurable, and proportionate to the actual risks of importing legitimate photo and video libraries.
dc6003743023cfc67c57WIP: Gate archive processing on isolated antivirus coverageto WIP: Layered ingestion security — design and shared-stream foundationWIP: Layered ingestion security — design and shared-stream foundationto WIP: Layered ingestion security with bounded media validationWIP: Layered ingestion security with bounded media validationto WIP: Layered ingestion security and private file findingsWIP: Layered ingestion security and private file findingsto WIP: Layered ingestion security with authenticated scan reuseWIP: Layered ingestion security with authenticated scan reuseto WIP: Layered ingestion security and measured analysis costsf40f4d0a5036b230a3ad36b230a3adcb0437832d4e0afef18724db3ffdf4View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.