Cards from scheduled scans of editor-saved profiles showed the account's
GUID as its name and no Elev/Ansat badge: the profile editor stored bare
id strings, the scheduler passes them straight to run_scan, and run_scan
normalised them to displayName=<id>, userRole="other". Before 1.7.10 the
scan-start profile overwrite kept repairing profiles to named objects,
which is why this surfaced now.
- run_scan: _enrich_bare_users() looks bare accounts up in the directory
(list_users, falling back to lookup_user) for name and role.
- M365Connector.build_role_sku_map(): the SKU map /api/users built inline,
now shared so cards and the account list classify roles identically.
lookup_user() also returns assignedLicenses.
- Profile editor saves {id, displayName, userRole} (manual accounts keep
email/manual), like the 💾 save.
- _accountPill: an account_name equal to account_id counts as missing;
name and role are filled from the loaded user list, so existing cards
display correctly without a rescan.
- Excel export: the scan's account-name map no longer breaks on bare id
strings (.get() on a str raised and dropped the whole map).
Tests: tests/test_account_names.py (10); the run_scan test fails on the
old engine with "Collecting emails — guid-henrik…".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Graph returned 404 for /users/google:<email>/mailFolders: Google-
Workspace-only accounts (id "google:<email>") reached the M365 scan.
- getSelectedUsers() fell back to "every ticked checkbox in the DOM"
whenever no M365 account was ticked — including Google-only rows. The
fallback is now only used before the user list has loaded, and never
includes google: rows.
- startScan() skips the M365 part (with a log line) when M365 sources are
ticked but no M365 accounts are: the server treats an empty account
list as "every user in the tenant".
- run_scan: _resolve_manual_users -> _resolve_scan_users, which now also
drops google: ids (reported as a phase message). Covers scheduled scans
and profiles that already contain Google-only accounts. Never widens to
all users.
Tests: 3 new in tests/test_manual_users.py; the run_scan one fails on the
old engine with exactly the reported Graph call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pressing Stop could look like the scan restarting, and for scheduled jobs
it really did restart the Google stage.
- Scheduler: check the abort flag between stages and never clear the
Google abort after a Stop (that started a fresh all-users Google scan).
A stopped job is recorded as "cancelled" and skips email/retention;
scheduler_done carries cancelled=true.
- SSE: scan_cancelled and file_scan_done now end the current scan like
scan_done, and sse.replay_messages() never replays a finished scan.
Previously a stopped M365 scan (and every file scan) stayed "current",
so each reconnect replayed it and re-showed Stop + progress.
- Frontend: scan_cancelled clears the M365 running flag and only tears
down the UI when no Google/file scan is still running; scheduler_done /
scheduler_error clear all running flags.
- run_scan: Stop during collection now ends the scan (checkpoint, finish,
scan_cancelled) instead of announcing scan_start / "Scanning…".
- Stop button: switches to a disabled "Stopping…" state on click; all
show/hide goes through _setStopBtnDisplay(), which resets it when hidden
or when a new scan starts.
Adds tests/test_stop_scan.py (10 tests).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual accounts were never scanned:
- Sidebar "add user" entries had no platform, and getSelectedUsers()
required platform === 'm365', so they were dropped from every scan.
- Profile-editor manual accounts were stored as "manual:<email>", which
is not a Graph user id, and applying the profile never put them in the
sidebar list, so they could not be selected.
Now the editor looks the email up via /api/users/lookup and stores the
directory id plus name/email (falling back to manual:<email> only when the
lookup fails). Applying a profile adds its manual accounts to the sidebar
and resolves legacy manual:<email> entries, logging any that can't be
found. The scan engine resolves remaining manual:<email> ids (e.g. from
scheduled scans) via the new M365Connector.lookup_user(), skipping and
reporting failures without widening to all users.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Starting an M365 scan wrote the sidebar state back into the active profile
via _save_settings(), replacing its user_ids with the sidebar selection
(minus Google-only users and anything hidden by the role filter). With no
profile active it overwrote profiles[0]. Profiles are loaders, so scan
start no longer writes them; profile_id is audit-only.
The profile editor built its account list from S._allUsers only, so
manually added accounts, users not yet loaded, and users gone from the
directory were not rendered and got dropped on the next save. They now
render as checked "Manual" / "Not loaded" rows, and user_ids "all" is kept
when saved before users load.
Applying a profile cleared the pending user IDs before the un-awaited
Google merge had added google:<email> users, leaving them unticked.
Pending IDs are now kept until all are matched or the user list is final,
and _mergeGoogleUsers() re-applies them. A saved google:<email> also
matches an M365 user merged to platform 'both'.
Adds regression tests for scan start leaving profiles untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Settings → E-mailrapport tab (scheduler.js) saved the SMTP username
as `user` and TLS flag as `starttls`, but every backend reader expects
`username`/`use_tls` (routes/email.py). Result: username was always
empty, server.login() was skipped, and the SMTP server rejected the
send — surfacing as a misleading "authentication failed" message even
with a valid App Password. The bug was latent because Graph is preferred
whenever M365 is connected, so the SMTP path was rarely exercised.
- scheduler.js: send/load canonical keys (username, use_tls). The
send-report modal (scan.js) already used these.
- _load_smtp_config(): normalise legacy user→username / starttls→use_tls
so configs saved before the fix work without re-entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The card badge only rendered when f.account_name was set, and the
group (role) badge was nested inside that same check. But save_item
never persisted account_name — only account_id (a GUID) and user_role.
Live SSE cards carried account_name so badges showed during a scan;
now that the grid loads finalized scans from the DB, the gap is exposed
and both badges vanish for earlier scans.
- Persist account_name (migration 11 + save_item) so future scans show
the user badge. Both M365 and Google cards already carry it.
- _accountPill() in results.js drives the group badge off user_role
alone (shows for legacy rows) and resolves a best-effort user label:
account_name → S._allUsers (id/email) → email-style account_id → omit.
Both card layouts share the one helper.
Legacy rows still lack account_name (never captured), but now show the
group badge and a resolved/email user label where possible.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
get_session_items / get_open_items / latest_scan_id all require
finished_at IS NOT NULL, but the M365 and Google engines return early
on abort (skipping finish_scan) and a process kill mid-scan (deploy,
OOM, crash) never reaches it either. Result on prod: 41/42 scans had
finished_at NULL, so 291 already-saved flagged items were invisible —
the grid showed nothing.
- finalize_orphan_scans(): finalises every finished_at-NULL scan; runs
once at startup before the scheduler (nothing is scanning at boot, so
any unfinished scan is dead). Recovers existing stranded items and
guards against future mid-scan restarts.
- run_scan: finalise the DB scan on the abort early-return too, so a
stopped scan's items stay visible without waiting for a restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The default results view loaded only the latest scan session (±300s
window), so items dropped out of sight once a newer scan started — and
a long scheduled scan could show little or nothing on browser open.
Add get_open_items(): every flagged item with no disposition (or status
'unreviewed') across all scans, deduped by id to the latest finished
scan. GET /api/db/flagged now serves it when no ?ref is given; ?ref=N
still loads a specific past session. Frontend loadHistorySession(null)
routes to a new loadOpenItems() loader. Rename the banner button to
"Open items" (da/de/en).
get_session_items() default is unchanged — export.py and
scan_scheduler.py still rely on latest-session for the current scan's
report/email.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The page-load restore was one-shot and bailed when a completed scan's
replayed scan_phase left a running flag set; sse_replay_done (the other
retry) only fires for a non-empty replay buffer, which is empty after a
restart — so refreshing post-update showed a blank grid despite the
results being in the DB. The watchdog now retries the restore on each
4s poll while nothing is shown and no scan runs, clearing stale flags
first. /api/scan/status also reports google_running separately so a
refresh during a live Google scan is no longer treated as idle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Werkzeug sets its server socket inheritable unconditionally, so the
os.execv restart carried it into the new process as a zombie listener:
one PID listening on both 5100 (never accepted) and 5101 (the real
server). Mark all fds above stderr close-on-exec before exec'ing so
the old socket dies and the new server rebinds the original port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operators can now choose Tesseract language pack(s) per profile via a
sidebar select (#optOcrLang) and profile editor (#peOptOcrLang). Presets:
dan+eng (default), dan, eng, dan+eng+deu, dan+eng+swe, dan+eng+fra. The
ocr_lang option flows from the UI through all three scan engines (M365
files/attachments, Google Drive, Gmail) down to document_scanner.scan_pdf
and scan_image — including the spawned PDF-OCR subprocess worker.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat: video/audio metadata scanning, profile rename fix, route tests
- Scan .mp4/.mov/.avi/.mkv and .mp3/.flac/.ogg/.m4a/.wma (+ 7 more)
for GPS coordinates, artist/author, title, comment — metadata only,
no frame or audio analysis. Uses mutagen (added to requirements.txt).
GPS-tagged phone recordings now flag with gps_location like photos.
- Fix _extract_audio_metadata silently returning empty results:
mutagen.File() first positional arg is `filename`, not `fileobj` —
was passing BytesIO as the filename. Fixed to keyword args.
- Fix profile copy rename not reflected in left column until modal
reopen: _pmgmtSaveFullEdit called loadProfiles() but never
_renderProfileMgmt(). Added re-render and active-row highlight.
- Add TestProfileRoutes (10 tests) covering all profile API endpoints
including a rename regression test. Total: 182 tests.
- generate_fixtures.py now produces 6 audio/video fixtures (14–19):
2 MP3, 2 FLAC, 2 MP4 — 4 flagged, 2 negative cases.
