fs-mail 2.2.1: stress and longevity report
========================================

RESULT

The final full suite passed: 530 tests, 0 failures, 0 skipped.
Command: python3 -m unittest discover -s tests -q
Recorded elapsed time: 49.113 seconds.

The 2.2.0 baseline first passed its existing 481 tests. New regressions and
long-running workloads exposed defects that were corrected in this release.
The final installed zipapp matches the frozen source modules byte for byte.

All accounts, addresses, passwords, message contents, and server responses used
for testing were synthetic. No real mailbox was contacted and no external mail
was submitted. Real wire tests used temporary TLS servers on 127.0.0.1.

MEASURED WORKLOADS

1. IMAP synchronization and recovery

   FSMAIL_SYNC_SOAK_CYCLES=1000 python3 -m unittest discover -s tests \
       -p test_sync_longevity.py -v

   Four tests passed in 177.766 seconds. The workload comprised:

   - 2,000 idle account polls across two separate account roots.
   - 1,000 cycles mixing local and remote flags, file renames, folder moves,
     local deletion, remote deletion, and new arrivals.
   - 1,000 simulated crashes after a verified IMAP MOVE, each followed by
     database reopening and local recovery without replaying the MOVE.
   - 64 UIDVALIDITY rollovers without applying old flags to a new generation.
   - 5,154 synchronize() calls including initialization and recovery.
   - 446 required initial/new/generation body downloads, with no redundant
     body downloads in the tested scenarios.

   Idle polling left file counts, object hashes, SQLite page counts, and open
   file-descriptor counts unchanged. Transport objects closed after each run,
   SQLite quick_check remained ok, and successful recovery left no unresolved
   operations. Real history grew when messages changed, as intended.

   This extended run began before the final header/provenance edits were
   loaded. Those edits passed focused regressions, the default 24-cycle soak,
   and the final complete suite on the delivered source.

2. Outbox submission and restart recovery

   FS_MAIL_OUTBOX_CYCLES=512 python3 -m unittest discover -s tests \
       -p test_outbox_longevity.py -v

   The final conservative implementation passed all 12 tests in 67.158 seconds.
   Peak process RSS reported by the Linux runner was 30,044 KiB.

   - 512 successful attachment-bearing submissions.
   - 512 uncertain-delivery scenarios explicitly confirmed without another
     SMTP DATA submission.
   - 4,096 main-loop database reopenings: 3,072 fault-recovery, 512 post-send,
     and 512 idle reopenings; four additional reopenings in the move-out test.
   - 2,048 atomic editor saves of a held draft.
   - 2,000 account/endpoint changes in the prompted-credential cache.

   Attachments in accepted archives remained byte-exact after their original
   files changed. Idle runs added no files or file-size growth. File-descriptor
   counts stayed unchanged and SQLite quick_check returned ok.

   Faults included recipient refusal, a lost final SMTP reply, renamed held
   sources, cleanup failure after explicit delivery confirmation, and attachment
   edits before and during DATA. SMTP was an in-process scripted transport.

3. Calendar-length watch scheduling

   PYTHONPATH=.:tests python3 tests/test_watch_longevity.py \
       --soak --days 180 --output docs/watch-soak.json

   The real scheduler traversed 180 simulated days with two account intervals:

   - Personal account: 51,841 runs at a 300-second interval.
   - Work account: 17,281 runs at a 900-second interval.
   - Total account runs: 69,122.
   - Configuration reloads: 259,200.
   - Sleep decisions: 259,201, each positive and no longer than 60 seconds.
   - Instrumented wall time: 7.420 seconds.

   Allocation samples after warm-up were 8,702 and 8,830 bytes. Peak traced
   allocation was 10,118 bytes; 1,930 bytes remained at measurement end. These
   are allocations inside the scheduler exercise, not total process memory.

   Additional tests covered slow account runs, completion-based intervals,
   temporary invalid configuration, recovery, removed accounts, changed
   endpoints, and pruning obsolete typed passwords.

   The clock and account operation were substituted. This is not a claim of
   180 days of real provider connections or continuous machine uptime.

4. Attachment extraction and MIME parsing

   python3 -m unittest discover -s tests -p test_attachment_stress.py -v

   Thirteen new attachment tests passed. They exercised:

   - 280 binary payloads across names and transfer encodings.
   - Interruptions after each of five publication stages, followed by restart.
   - Edited partial files, symlinks, duplicate names, empty files, and limits.
   - 80 messages and 240 attachments across 180 database reopen cycles, with
     stable hashes and counts and SQLite integrity_check = ok.
   - 500 attachments under an approximately 2 KiB account path, yielding a
     received header larger than 1 MiB. UTF-32 editor rewriting, baseline
     adoption, scanning, and reply creation all succeeded.

   The repeatable MIME soak is included as a separate script:

   python3 tests/stress_mime.py --no-tracemalloc \
       --json-output docs/mime-soak.json

   The delivered runner passed 4,096 renders: 256 seeded mutated multipart
   messages over 16 passes. All passes had the same aggregate output digest.
   File descriptors stayed 4 -> 4. Runtime was 22.123 seconds.

   An earlier allocation-instrumented run of the same mutation algorithm and
   message contents also passed 4,096 renders in 104.466 seconds, retaining
   44,087 traced bytes with a peak of 780,278 bytes. The delivered runner pins
   the MIME boundary for reproducibility; the earlier run generated it randomly.
   Its default enables tracemalloc if that measurement is wanted again.

5. Credential helper subprocesses

   Eight new tests use real isolated helper processes. They verify literal
   arguments without a shell, exactly 64 KiB of output, runaway output, helper
   timeout, stdout closing before process exit, nonzero exit, discarded stderr,
   and 40 repeated oversized-output failures without descriptor leakage.

FIXES INCLUDED

Encoded attached email
    Some base64 or quoted-printable message/rfc822 and message/global parts
    previously produced still-encoded .eml files. They now decode into usable
    attached messages. Invalid encoding gives a readable notice; the original
    raw mail remains available.

Attachment filename extensions
    A wholly non-ASCII filename stem could disappear during normalization,
    turning a PDF filename into bare "pdf". A safe fallback stem now preserves
    useful extensions, and duplicate normalized names remain distinct.

Generated attachment lists
    A valid received message with many attachments could generate a header
    larger than the client's own 64 KiB metadata limit. Incoming metadata now
    has a separate 4 MiB bound, consistently applied to replies, baseline
    objects, recovery, and identity scans, including UTF-32 local views.
    Outgoing submission limits remain explicit and stricter.

Unicode identity scanning
    Byte-level blank-line lookalikes inside UTF-16 characters could hide a
    later Local-ID. Separator matches must now align with character boundaries.
    Large header scanning uses incremental searches to avoid quadratic work.

Cross-account attachment provenance
    A readable message saved as an attachment from another account could share
    a Local-ID and falsely block an ordinary deletion. Checks now consider
    Account-ID and Raw-Message provenance. Contradictory edited metadata cannot
    hide an original that still identifies its source raw message.

Long-running credential caches
    Successful watch configuration reloads discard typed credentials for
    removed accounts and obsolete endpoints. Active endpoints retain their
    passwords; temporary invalid configuration does not erase the last valid
    cache. At most two credentials per selected account are retained.

Credential helper output
    The old helper-output limit was checked only after completion, allowing
    excessive buffering until timeout. Output is now bounded while reading.
    Timed-out or oversized-output helpers are stopped and reaped; cleanup also
    closes descriptors. Error messages do not include the credential output.

Malformed recipient comments
    Excessive nested comments could raise RecursionError and abort processing
    later drafts. Nesting is bounded and parser recursion failures become a
    per-file error. A valid later draft can still be processed.

OUTBOX IDENTITY REVIEW

An attempted optimization retired old inode identities after editor saves.
Independent review found that it could release an uncertain draft after a
concurrent rename, or after the draft spent a polling cycle outside outbox and
was edited and returned under another name. The optimization was removed.

The delivered alias-discovery function is identical to 2.2.0. New regressions
verify that these edited sources remain held and that SMTP DATA is not repeated.
Known historical identities remain conservative until the job is resolved.

Consequently, an unresolved draft's journal can grow when it is repeatedly
edited or renamed, and ambiguous inode reuse can produce a hold requiring
explicit resolution. Idle scans do not add history. A durable outgoing identity
format would be needed before safely retiring that evidence; this release does
not exchange duplicate-submission protection for a smaller journal.

DOWNLOAD POLICY AND REMAINING LIMITS

This release still downloads complete messages from all selectable IMAP folders,
subject to per-message and per-run message-count limits. Attachments are already
inside those fetched raw messages. This is incremental after initial download,
but the first synchronization can transfer a large archive.

Selected-folder retrieval, headers-only working files, a total byte budget,
and selective on-demand MIME-part downloading are not implemented in 2.2.1.
The proposed --download-attachments and --download-folder interface is described
in DOWNLOAD-DESIGN.txt; these are design proposals, not current command options.

Raw mail, attachments, accepted submissions, and recovery history are retained.
Disk usage grows with real mail and edits. Error logs are not automatically
rotated. No automatic archive/history garbage collection was added.

The final executable was built and installed to an isolated prefix. Version,
help, local account listing, and offline status succeeded. Its modules matched
source byte for byte; 59 Python files parsed using Python 3.9 grammar. The
manpage rendered with groff without warnings, and both manuals are ASCII.

Tests ran on Linux. A physical Chromebook, native OpenBSD, live providers,
physical power loss, and many months of real network operation were not tested.
The results support the listed behaviors and workloads, not a claim that no
possible bug remains or that ordinary SMTP can guarantee exactly-once delivery.
