Preserve complete private importer logs and flag review concerns

jchapman commented

2026-10-25 06:09:14 +00:00

Owner

This change makes skipped-file explanations and importer evidence available after the temporary processor exits. It also prevents a successful process exit or zero item-error counters from hiding WARN, ERROR or critical log messages. References #10, #11 and the workflow in #14.

Behavior

Preserve complete DEBUG-and-higher importer output from both streams and every final discarded-item identity. Remove total log-byte, line-length and item-sample caps. Preserve duplicate JSON fields and exact numeric metadata.
Keep filenames, paths, original errors and ordinary URL details readable. Redact actual credentials and signed access parameters before storage. Private endpoints require both attempt and collection ownership, with no administrator override.
Index every WARN-or-higher entry. Warnings require review; ERROR/critical entries remain errors and do not qualify a preview for import. A WARN-only preview can proceed only with explicit confirmation of the same destination/archive/options set.
Preserve the existing step-by-step flow. A short review notice links to server-paged details, original messages and filenames, technical records and complete streaming downloads. The separate shareable support report excludes private records.
Retain all private diagnostics until explicit owner acknowledgement that ALL private logs/details will be deleted, or seven days after the attempt ends. Active attempts and ordinary upload cleanup cannot delete the logs. Explicit collection deletion includes those logs in its acknowledgement. Pending deletion and interrupted-attempt review windows recover after restart.
Transfer in acknowledged, retry-safe chunks. Storage/transfer/incomplete-manifest failures stop the attempt, retain available evidence and report incomplete diagnostics. No silent truncation, size-based pruning or automatic import replay.
Correct the pinned upstream completion path that attempted job resume despite --pause-immich-jobs=false. The importer now returns early when job pausing is disabled; the restrictive proxy remains intact. Modified importer and matching source are versioned 0.32.0-bridge.4.

Validation

Completed locally on AMD64 with Docker Desktop:

Go backend tests, vet and focused race checks, including the actual pinned importer, skip mappings/source call-site audit and disabled-job-control fixture.
Complete retention of 46,000 records representing 23,000 images plus metadata, logs exceeding the former 64 MiB cap, 27,003 affected items beyond the former sample, long/non-JSON records, exact names, duplicate fields, every WARN-or-higher flag and full paging.
Signed-in owner/other-user/admin isolation; authenticated range downloads; explicit acknowledgement and expiry; storage/transfer failure, lost-acknowledgement retry, restart recovery and cancellation/final flush.
Disposable browser flow with upload interruption/recovery, log/item/flag paging, original errors and names, inert hostile text, full native log download, support-report privacy, explicit deletion and mobile width. Importer ERROR fixture shows needs-attention despite zero item errors.
Real Docker processor isolation/cancellation and production HTTPS/readiness smoke tests.
UI lint/build; documentation/workflow contracts and nine CI policy tests; source secret scan.
Source, importer-binary and all three image advisory policies, using the existing reviewed baselines. The final source scan has 29 reviewed findings across 32 production files, no new exceptions. Matching-source checksum, notices, vendored source and applied job-resume guard verified.

Remaining checkpoint

This is implemented and locally tested, with source pushed for review. It has not been merged, published as release images, installed on the pilot, or run against a real photo library. A 150 GB import was not benchmarked. Full retention requires sufficient staging capacity; individual very long records still require redaction memory. Previously discarded logs cannot be recovered.

The owner approved proceeding with ARM compatibility assumed for these ordinary Go/file/HTTP changes. Native ARM64 execution remains unverified; no passing native Forgejo release gate is claimed, and release rules are unchanged.

Review this PR, then coordinate an idle installation of app, runner and processor together and run a new owner preview. Keep #10/#11 open through that pilot checkpoint. Do not interrupt active uploads or tasks.

This change makes skipped-file explanations and importer evidence available after the temporary processor exits. It also prevents a successful process exit or zero item-error counters from hiding WARN, ERROR or critical log messages. References #10, #11 and the workflow in #14. ### Behavior - Preserve complete DEBUG-and-higher importer output from both streams and every final discarded-item identity. Remove total log-byte, line-length and item-sample caps. Preserve duplicate JSON fields and exact numeric metadata. - Keep filenames, paths, original errors and ordinary URL details readable. Redact actual credentials and signed access parameters before storage. Private endpoints require both attempt and collection ownership, with no administrator override. - Index every WARN-or-higher entry. Warnings require review; ERROR/critical entries remain errors and do not qualify a preview for import. A WARN-only preview can proceed only with explicit confirmation of the same destination/archive/options set. - Preserve the existing step-by-step flow. A short review notice links to server-paged details, original messages and filenames, technical records and complete streaming downloads. The separate shareable support report excludes private records. - Retain all private diagnostics until explicit owner acknowledgement that ALL private logs/details will be deleted, or seven days after the attempt ends. Active attempts and ordinary upload cleanup cannot delete the logs. Explicit collection deletion includes those logs in its acknowledgement. Pending deletion and interrupted-attempt review windows recover after restart. - Transfer in acknowledged, retry-safe chunks. Storage/transfer/incomplete-manifest failures stop the attempt, retain available evidence and report incomplete diagnostics. No silent truncation, size-based pruning or automatic import replay. - Correct the pinned upstream completion path that attempted job resume despite `--pause-immich-jobs=false`. The importer now returns early when job pausing is disabled; the restrictive proxy remains intact. Modified importer and matching source are versioned `0.32.0-bridge.4`. ### Validation Completed locally on AMD64 with Docker Desktop: - Go backend tests, vet and focused race checks, including the actual pinned importer, skip mappings/source call-site audit and disabled-job-control fixture. - Complete retention of 46,000 records representing 23,000 images plus metadata, logs exceeding the former 64 MiB cap, 27,003 affected items beyond the former sample, long/non-JSON records, exact names, duplicate fields, every WARN-or-higher flag and full paging. - Signed-in owner/other-user/admin isolation; authenticated range downloads; explicit acknowledgement and expiry; storage/transfer failure, lost-acknowledgement retry, restart recovery and cancellation/final flush. - Disposable browser flow with upload interruption/recovery, log/item/flag paging, original errors and names, inert hostile text, full native log download, support-report privacy, explicit deletion and mobile width. Importer ERROR fixture shows needs-attention despite zero item errors. - Real Docker processor isolation/cancellation and production HTTPS/readiness smoke tests. - UI lint/build; documentation/workflow contracts and nine CI policy tests; source secret scan. - Source, importer-binary and all three image advisory policies, using the existing reviewed baselines. The final source scan has 29 reviewed findings across 32 production files, no new exceptions. Matching-source checksum, notices, vendored source and applied job-resume guard verified. ### Remaining checkpoint This is implemented and locally tested, with source pushed for review. It has not been merged, published as release images, installed on the pilot, or run against a real photo library. A 150 GB import was not benchmarked. Full retention requires sufficient staging capacity; individual very long records still require redaction memory. Previously discarded logs cannot be recovered. The owner approved proceeding with ARM compatibility assumed for these ordinary Go/file/HTTP changes. Native ARM64 execution remains unverified; no passing native Forgejo release gate is claimed, and release rules are unchanged. Review this PR, then coordinate an idle installation of app, runner and processor together and run a new owner preview. Keep #10/#11 open through that pilot checkpoint. Do not interrupt active uploads or tasks.

jchapman added 1 commit

2026-10-25 06:09:15 +00:00

Explain skip outcomes and retain private importer diagnostics

Bridge validation / source (amd64) (push) Has been cancelled

Details

Bridge validation / source (arm64) (push) Has been cancelled

Details

Bridge validation / backend (amd64) (push) Has been cancelled

Details

Bridge validation / backend (arm64) (push) Has been cancelled

Details

Bridge validation / isolation (amd64) (push) Has been cancelled

Details

Bridge validation / isolation (arm64) (push) Has been cancelled

Details

Bridge validation / browser (amd64) (push) Has been cancelled

Details

Bridge validation / browser (arm64) (push) Has been cancelled

Details

Bridge validation / production (amd64) (push) Has been cancelled

Details

Bridge validation / production (arm64) (push) Has been cancelled

Details

Bridge validation / supply_chain (amd64) (push) Has been cancelled

Details

Bridge validation / supply_chain (arm64) (push) Has been cancelled

Details

Bridge validation / release-gate (push) Has been cancelled

Details

59cde963d5

Rows
Columns

Preserve complete private importer logs and flag review concerns #15

Behavior

Validation

Remaining checkpoint