37 Commits
Author SHA1 Message Date
StyxX65andClaude Opus 5.5 5e0ce6996b TODO: how to verify a web-UI update actually restarted the app
The web-UI update restarts via os.execv in the same process, so systemd's
ActiveEnterTimestamp doesn't change. Check for a "[skus] Loaded" startup
line after the release was pushed instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:57:12 +02:00
StyxX65andClaude Opus 5.5 5e60d17d60 Release 1.7.13
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- Fix: result cards (and the Excel report) showed account IDs instead of
  names, with no Elev/Ansat badge, for scheduled scans of editor-saved
  profiles.
- VERSION, CHANGELOG and manual version strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:43:00 +02:00
StyxX65andClaude Opus 5.5 20156ffc93 Show account names and roles on cards from scheduled scans
Cards from scheduled scans of editor-saved profiles showed the account's
GUID as its name and no Elev/Ansat badge: the profile editor stored bare
id strings, the scheduler passes them straight to run_scan, and run_scan
normalised them to displayName=<id>, userRole="other". Before 1.7.10 the
scan-start profile overwrite kept repairing profiles to named objects,
which is why this surfaced now.

- run_scan: _enrich_bare_users() looks bare accounts up in the directory
  (list_users, falling back to lookup_user) for name and role.
- M365Connector.build_role_sku_map(): the SKU map /api/users built inline,
  now shared so cards and the account list classify roles identically.
  lookup_user() also returns assignedLicenses.
- Profile editor saves {id, displayName, userRole} (manual accounts keep
  email/manual), like the 💾 save.
- _accountPill: an account_name equal to account_id counts as missing;
  name and role are filled from the loaded user list, so existing cards
  display correctly without a rescan.
- Excel export: the scan's account-name map no longer breaks on bare id
  strings (.get() on a str raised and dropped the whole map).

Tests: tests/test_account_names.py (10); the run_scan test fails on the
old engine with "Collecting emails — guid-henrik…".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:40:14 +02:00
StyxX65andClaude Opus 5.5 c55db48894 Document the post-1.7.12 M365+GWS scan report and update checks
TODO.md: open entry for "M365 not scanning M365+GWS accounts after the
1.7.12 update" — symptom, what was verified, likely half-applied update,
server checks to run after an update, what to collect if it recurs,
rollback option, and two possible hardening steps.

routes/CLAUDE.md: note that half-applied updates look like code bugs and
to verify server state before debugging a post-update regression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:44:55 +02:00
StyxX65andClaude Opus 5.5 04efb462af Release 1.7.12
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- Fix: Google-only accounts no longer reach the M365 scan (Graph 404s);
  an M365 scan with no M365 accounts selected no longer scans everyone.
- VERSION, CHANGELOG and manual version strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:30:01 +02:00
StyxX65andClaude Opus 5.5 b5476228ba Keep Google-only accounts out of the M365 scan
Graph returned 404 for /users/google:<email>/mailFolders: Google-
Workspace-only accounts (id "google:<email>") reached the M365 scan.

- getSelectedUsers() fell back to "every ticked checkbox in the DOM"
  whenever no M365 account was ticked — including Google-only rows. The
  fallback is now only used before the user list has loaded, and never
  includes google: rows.
- startScan() skips the M365 part (with a log line) when M365 sources are
  ticked but no M365 accounts are: the server treats an empty account
  list as "every user in the tenant".
- run_scan: _resolve_manual_users -> _resolve_scan_users, which now also
  drops google: ids (reported as a phase message). Covers scheduled scans
  and profiles that already contain Google-only accounts. Never widens to
  all users.

Tests: 3 new in tests/test_manual_users.py; the run_scan one fails on the
old engine with exactly the reported Graph call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:28:05 +02:00
StyxX65andClaude Opus 5.5 b60c4a5c47 Release 1.7.11
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- Fix: selecting a profile ticked (and would scan) every M365-only and
  GWS-only account — regression in 1.7.10.
- VERSION, CHANGELOG and manual version strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:21:14 +02:00
StyxX65andClaude Opus 5.5 37c9cbfd2c Fix selecting a profile ticking every M365-only and GWS-only user
_applyPendingProfileUsers() set u.selected = matches(u), where matches()
was `wanted.has(u.id) || (u.googleEmail && ...)`. For users without a
googleEmail (every M365-only and Google-only user) that evaluates to
undefined, not false — and both renderAccountList() and getSelectedUsers()
treat anything other than false as selected. So applying a profile ticked,
and would have scanned, all of those users in addition to the profile's own.

Coerce the googleEmail branch to a boolean.

Reproduced against a copy of real data: a 1-account profile ticked 56
(37 M365-only + 18 GWS-only extras); after the fix it ticks 1, and every
profile ticks only its own stored accounts.

Regression from bbe9c62 (released in 1.7.10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:15:36 +02:00
StyxX65andClaude Opus 5.5 5df151724a Release 1.7.10: changelog + manual updates
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- VERSION 1.7.9 -> 1.7.10 (patch: fixes and small behaviour changes only).
- CHANGELOG: 1.7.10 entry covering everything since 1.7.9 — Stop fixes,
  profile/manual-account fixes, delete_message, landing card,
  classification data path, scan-thread crash logging, repo move.
- Manuals (EN/DA): version, Stop -> "Stopping…" and scheduled-job stop,
  scans no longer change profiles, manual accounts in the profile editor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:07:01 +02:00
StyxX65andClaude Opus 5.5 606d8e944b Make Stop stop: no replay, no restart, immediate feedback
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
Pressing Stop could look like the scan restarting, and for scheduled jobs
it really did restart the Google stage.

- Scheduler: check the abort flag between stages and never clear the
  Google abort after a Stop (that started a fresh all-users Google scan).
  A stopped job is recorded as "cancelled" and skips email/retention;
  scheduler_done carries cancelled=true.
- SSE: scan_cancelled and file_scan_done now end the current scan like
  scan_done, and sse.replay_messages() never replays a finished scan.
  Previously a stopped M365 scan (and every file scan) stayed "current",
  so each reconnect replayed it and re-showed Stop + progress.
- Frontend: scan_cancelled clears the M365 running flag and only tears
  down the UI when no Google/file scan is still running; scheduler_done /
  scheduler_error clear all running flags.
- run_scan: Stop during collection now ends the scan (checkpoint, finish,
  scan_cancelled) instead of announcing scan_start / "Scanning…".
- Stop button: switches to a disabled "Stopping…" state on click; all
  show/hide goes through _setStopBtnDisplay(), which resets it when hidden
  or when a new scan starts.

Adds tests/test_stop_scan.py (10 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:03:12 +02:00
StyxX65andClaude Opus 5.5 9cf23da907 Point docs at git.fejl40.nu after the move from GitHub
- README / CONTRIBUTING: clone from the Gitea repo.
- SECURITY: Gitea has no private security advisories; report to
  support@fejl40.nu instead.
- RELEASING: release URL on Gitea, plus a note that build.yml still uses
  the GitHub-only action-gh-release step, so no releases are produced on
  Gitea yet (server self-update via git is unaffected).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 09:49:06 +02:00
StyxX65andClaude Opus 5.5 1c79ac15b7 Scan manually added accounts
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
Manual accounts were never scanned:
- Sidebar "add user" entries had no platform, and getSelectedUsers()
  required platform === 'm365', so they were dropped from every scan.
- Profile-editor manual accounts were stored as "manual:<email>", which
  is not a Graph user id, and applying the profile never put them in the
  sidebar list, so they could not be selected.

Now the editor looks the email up via /api/users/lookup and stores the
directory id plus name/email (falling back to manual:<email> only when the
lookup fails). Applying a profile adds its manual accounts to the sidebar
and resolves legacy manual:<email> entries, logging any that can't be
found. The scan engine resolves remaining manual:<email> ids (e.g. from
scheduled scans) via the new M365Connector.lookup_user(), skipping and
reporting failures without widening to all users.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 09:27:56 +02:00
StyxX65andClaude Opus 5.5 bbe9c62ff9 Fix profiles losing their selected users
Starting an M365 scan wrote the sidebar state back into the active profile
via _save_settings(), replacing its user_ids with the sidebar selection
(minus Google-only users and anything hidden by the role filter). With no
profile active it overwrote profiles[0]. Profiles are loaders, so scan
start no longer writes them; profile_id is audit-only.

The profile editor built its account list from S._allUsers only, so
manually added accounts, users not yet loaded, and users gone from the
directory were not rendered and got dropped on the next save. They now
render as checked "Manual" / "Not loaded" rows, and user_ids "all" is kept
when saved before users load.

Applying a profile cleared the pending user IDs before the un-awaited
Google merge had added google:<email> users, leaving them unticked.
Pending IDs are now kept until all are matched or the user list is final,
and _mergeGoogleUsers() re-applies them. A saved google:<email> also
matches an M365 user merged to platform 'both'.

Adds regression tests for scan start leaving profiles untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 09:22:32 +02:00
StyxX65 b77bf70ff5 Fix deprecated smbprotocol.query_info import
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
smbprotocol.query_info is a thin deprecation shim (from smbprotocol.file_info
import * + a DeprecationWarning) scheduled for removal in the library's next
major release. Switched to importing FileDirectoryInformation directly from
smbprotocol.file_info, the actual current location - confirmed present there
in the installed package. Pure rename, no behavior change: all 227 tests
still pass, and the warning is gone from the test output.
2026-09-03 22:59:38 +02:00
StyxX65andClaude Sonnet 5 edc820051e Fix classification/ directory never matching where data files lived
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
The code has always expected classification/m365_skus.json and
classification/google_ou_roles.json (present since the initial commit
in m365_connector.py, google_connector.py, and build_gdpr.py's own
skus_dir variable) - but the actual data files lived in skus/ under a
different name (education.json) or just the wrong directory. Both
loaders degrade to a fallback on failure, so this was never fatal, but
it meant:

- M365 SKU-based role classification silently ran on fragment-matching
  only, never touching the real 13 student + 11 staff SKU list.
- Google Workspace OU-based role classification silently used a
  hardcoded 4-prefix fallback missing real categories the actual
  config has - /Elev, /Pupils (student), /Teachers, /Admin (staff).
  _load_ou_roles() had a bare `except Exception: return <fallback>`
  with no logging at all, so this had zero visibility anywhere.

Fix: moved both files to classification/ (git mv, so history is
preserved), updated .gitignore's exception rule from !skus/*.json to
!classification/*.json, fixed the stale "skus/google_ou_roles.json"
docstring, and added a module logger + proper warning/error logging to
_load_ou_roles() matching the pattern _load_sku_data() already used -
so if this class of bug recurs, it won't be silent again.

Verified: both loaders now log successful loads with the real counts
(13+11 SKUs, 4+6 OU prefixes including the previously-missing
categories), all 227 tests still pass, and a full app boot confirms
the startup warning is gone and the app still serves normally.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-03 22:58:24 +02:00
StyxX65 2260a451a5 Fix deprecated fitz import, log unhandled scan-thread crashes
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
document_scanner.py: `import fitz` is PyMuPDF's deprecated import
alias, scheduled for removal - switched both sites to `import pymupdf
as _fitz`. Pure rename, no behavior change (all 227 tests still pass).

routes/scan.py: the background scan thread caught no exceptions, so
an unexpected crash in run_scan() escaped into Python's default
unhandled-thread-exception hook instead of the app's own logging/audit
trail. The scan lock was already released correctly via `finally` -
now the crash itself is also logged (`_log.exception`, full traceback)
and audit-recorded before that. Confirmed by test_lock_released_when_
run_scan_raises: the PytestUnhandledThreadExceptionWarning it
triggered is now gone, with no change to the test's own assertions.
2026-09-03 22:55:07 +02:00
StyxX65andClaude Opus 4.8 efbbeb7306 Restore M365Connector.delete_message (was an orphaned method body)
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
The def line for delete_message had been lost, leaving its body as
unreachable dead code at the end of _delete() and no delete_message
attribute on the connector. Deleting an Outlook message therefore failed
with "'M365Connector' object has no attribute 'delete_message'". Restored
the method (soft-delete: move to Deleted Items, fall back to DELETE).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 15:43:46 +02:00
StyxX65andClaude Opus 4.8 54f8848e30 Document renderGrid landing-card hiding in static/js/CLAUDE.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 14:49:43 +02:00
StyxX65andClaude Opus 4.8 8a446509c6 Hide landing/last-scan card whenever results render
The live scan_file_flagged handler showed the grid but never hid
#emptyState / #lastScanSummary, so when a scan ran with the landing
card visible, results appeared underneath it until a manual refresh
(which re-ran loadOpenItems and cleared it). Hide both panels in
renderGrid whenever files are present, covering every render path
(live SSE, open-items load, history, filters).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 14:45:50 +02:00
StyxX65andClaude Opus 4.8 d55778ab35 Release 1.7.9: changelog + manual updates
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
Document this cycle's changes: open-items default results view,
interrupted-scan recovery, restored user/group badges, the SMTP
username-key fix, and the new "always send via SMTP" toggle. Stamp
manuals (EN/DA) to 1.7.9.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 11:36:41 +02:00
StyxX65andClaude Opus 4.8 874c3ccec1 Add "prefer SMTP" toggle to skip Microsoft Graph for email
When the M365 connector is connected the app always tries Graph first,
and a Graph 202 ends the send — so report mail to recipients Exchange
silently drops (Google-hosted subdomains of the O365 domain) never
reaches them, even with working SMTP configured.

New prefer_smtp flag gates all three Graph branches (smtp_test,
send_report, _maybe_send_auto_email) so they go straight to SMTP. UI
toggle #st-smtpPreferSmtp in Settings → E-mailrapport, saved/loaded by
scheduler.js, with da/de/en strings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 11:30:45 +02:00
StyxX65andClaude Opus 4.8 526e2b0b78 Fix SMTP auth: settings tab saved wrong config keys
The Settings → E-mailrapport tab (scheduler.js) saved the SMTP username
as `user` and TLS flag as `starttls`, but every backend reader expects
`username`/`use_tls` (routes/email.py). Result: username was always
empty, server.login() was skipped, and the SMTP server rejected the
send — surfacing as a misleading "authentication failed" message even
with a valid App Password. The bug was latent because Graph is preferred
whenever M365 is connected, so the SMTP path was rarely exercised.

- scheduler.js: send/load canonical keys (username, use_tls). The
  send-report modal (scan.js) already used these.
- _load_smtp_config(): normalise legacy user→username / starttls→use_tls
  so configs saved before the fix work without re-entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 11:25:15 +02:00
StyxX65andClaude Opus 4.8 b661a94f98 Restore user/group badges on DB-loaded result cards
The card badge only rendered when f.account_name was set, and the
group (role) badge was nested inside that same check. But save_item
never persisted account_name — only account_id (a GUID) and user_role.
Live SSE cards carried account_name so badges showed during a scan;
now that the grid loads finalized scans from the DB, the gap is exposed
and both badges vanish for earlier scans.

- Persist account_name (migration 11 + save_item) so future scans show
  the user badge. Both M365 and Google cards already carry it.
- _accountPill() in results.js drives the group badge off user_role
  alone (shows for legacy rows) and resolves a best-effort user label:
  account_name → S._allUsers (id/email) → email-style account_id → omit.
  Both card layouts share the one helper.

Legacy rows still lack account_name (never captured), but now show the
group badge and a resolved/email user label where possible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 10:15:19 +02:00
StyxX65andClaude Opus 4.8 29d9168643 Recover unfinished scans so their items aren't stranded
get_session_items / get_open_items / latest_scan_id all require
finished_at IS NOT NULL, but the M365 and Google engines return early
on abort (skipping finish_scan) and a process kill mid-scan (deploy,
OOM, crash) never reaches it either. Result on prod: 41/42 scans had
finished_at NULL, so 291 already-saved flagged items were invisible —
the grid showed nothing.

- finalize_orphan_scans(): finalises every finished_at-NULL scan; runs
  once at startup before the scheduler (nothing is scanning at boot, so
  any unfinished scan is dead). Recovers existing stranded items and
  guards against future mid-scan restarts.
- run_scan: finalise the DB scan on the abort early-return too, so a
  stopped scan's items stay visible without waiting for a restart.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 09:51:22 +02:00
StyxX65 7bf589bf7a Update ZORAXY_SETUP.md 2026-06-22 09:21:08 +02:00
StyxX65andClaude Opus 4.8 68076eba52 Show all open (unactioned) items by default, not just the last scan
The default results view loaded only the latest scan session (±300s
window), so items dropped out of sight once a newer scan started — and
a long scheduled scan could show little or nothing on browser open.

Add get_open_items(): every flagged item with no disposition (or status
'unreviewed') across all scans, deduped by id to the latest finished
scan. GET /api/db/flagged now serves it when no ?ref is given; ?ref=N
still loads a specific past session. Frontend loadHistorySession(null)
routes to a new loadOpenItems() loader. Rename the banner button to
"Open items" (da/de/en).

get_session_items() default is unchanged — export.py and
scan_scheduler.py still rely on latest-session for the current scan's
report/email.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 09:19:55 +02:00
StyxX65andClaude Fable 5 67f66c8441 Document self-update system and related changes in CLAUDE.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-16 12:16:14 +02:00
StyxX65andClaude Fable 5 8bb482925f Release 1.7.8
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- CHANGELOG: cut the 1.7.8 release (dated 2026-06-16); reset Unreleased.
- VERSION: 1.7.7 -> 1.7.8.
- Manuals (DA + EN): bump version stamps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-16 11:56:12 +02:00
StyxX65andClaude Fable 5 f84c8516df Reliably restore last session on refresh after a server restart
The page-load restore was one-shot and bailed when a completed scan's
replayed scan_phase left a running flag set; sse_replay_done (the other
retry) only fires for a non-empty replay buffer, which is empty after a
restart — so refreshing post-update showed a blank grid despite the
results being in the DB. The watchdog now retries the restore on each
4s poll while nothing is shown and no scan runs, clearing stale flags
first. /api/scan/status also reports google_running separately so a
refresh during a live Google scan is no longer treated as idle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-16 11:53:07 +02:00
StyxX65andClaude Fable 5 9fd1aa1f8a Manuals: describe new share-link create flow
After Create the form clears and the new link appears highlighted in
the Active links list, copied from there — not from a preview row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:52:22 +02:00
StyxX65andClaude Fable 5 da356fb310 Release 1.7.7
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- CHANGELOG: cut the 1.7.7 release (dated 2026-06-15); reset Unreleased.
- VERSION: 1.7.6 -> 1.7.7.
- Manuals (DA + EN): bump version stamps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:12:00 +02:00
StyxX65andClaude Fable 5 bdba80e72d Remove stale link preview from share modal after create
The generated-link "Copy link:" row stayed visible after creating,
looking like the form hadn't reset — but the new link was already in
the Active links list with its own Copy button. Drop the redundant
preview row; on create, reset the form and briefly highlight the new
entry in the active list. Removes the now-dead shareNewLinkRow markup
and copyShareLink().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:11:03 +02:00
StyxX65andClaude Fable 5 c26dd7d320 Add Zoraxy HTTPS setup guide, correct SECURITY.md bind address
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:20:33 +02:00
StyxX65andClaude Fable 5 841311a6bd Release 1.7.6
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- CHANGELOG: cut the 1.7.6 release (dated 2026-06-11); reset Unreleased.
- VERSION: 1.7.5 -> 1.7.6.
- Manuals (DA + EN): bump version stamps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:02:44 +02:00
StyxX65andClaude Fable 5 dd19be8bbf Close leaked listening socket on update restart
Werkzeug sets its server socket inheritable unconditionally, so the
os.execv restart carried it into the new process as a zombie listener:
one PID listening on both 5100 (never accepted) and 5101 (the real
server). Mark all fds above stderr close-on-exec before exec'ing so
the old socket dies and the new server rebinds the original port.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 15:01:17 +02:00
StyxX65andClaude Fable 5 c43725ca7f Release 1.7.5
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- CHANGELOG: cut the 1.7.5 release (dated 2026-06-11); reset Unreleased.
- VERSION: 1.7.4 -> 1.7.5.
- Manuals (DA + EN): bump version stamps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:42:06 +02:00
StyxX65andClaude Fable 5 a1712ae178 Make static files revalidate so the UI is fresh after updates
No Cache-Control header meant browsers cached JS/CSS heuristically for
days; after a server update (including the in-app self-update reload)
the backend was new but the frontend stayed stale. SEND_FILE_MAX_AGE
_DEFAULT=0 forces ETag revalidation — 304 when unchanged, fresh file
immediately after an update.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:39:45 +02:00
51 changed files with 1967 additions and 229 deletions

No files matched your search

+1 -1
View File
@@ -5,7 +5,7 @@
*.json
!lang/*.json
!keywords/*.json
!skus/*.json
!classification/*.json
!package*.json
# Be explicit about the most sensitive files
+109
View File
@@ -11,6 +11,115 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
---
## [1.7.13] — 2026-09-28
### Fixed
- **Result cards showed an ID instead of the person's name** — cards from scheduled scans of profiles saved in the profile editor showed the account's Microsoft 365 ID (e.g. `38a9797a-e065-…`) where the name should be, and no Elev/Ansat badge. The profile editor saved accounts as bare IDs, and the scan used the ID as the name. The scan now looks such accounts up for their name and role (using the same Elev/Ansat classification as the account list), the profile editor saves the name and role with each account, and existing cards with an ID as their name now show the name and role from the account list — no rescan needed.
- **Excel report could show IDs instead of names** — for scans started from ID-only account lists, the report's account-name lookup failed as a whole and fell back to the stored values.
---
## [1.7.12] — 2026-09-28
### Fixed
- **Microsoft 365 scan failed with "404 Not Found" for Google-only accounts** — accounts that exist only in Google Workspace were sent to the Microsoft 365 scan (`…/users/google:<email>/mailFolders`), which Graph cannot resolve. This happened when no Microsoft 365 account was ticked (the scanner then took every ticked account, Google-only ones included), and in scheduled scans whose profile contains Google-only accounts. The Microsoft 365 scan now skips Google-only accounts and reports how many it skipped; the Google Workspace scan still covers them.
- **An M365 scan with no M365 accounts selected could scan the whole tenant** — with Google sources ticked and no Microsoft 365 account ticked, the Microsoft 365 part started with an empty account list, which the server treats as "every user". It is now skipped, with a note in the log.
---
## [1.7.11] — 2026-09-28
### Fixed
- **Selecting a profile ticked accounts that aren't in it** — a regression in 1.7.10: applying a profile also ticked every Microsoft 365-only and every Google Workspace-only account in the sidebar, on top of the profile's own accounts. Only accounts present in both M365 and Google Workspace followed the profile correctly. Because a scan uses the ticked accounts, a scan started after selecting a profile in 1.7.10 could cover more accounts than the profile lists. Profiles now tick exactly their own accounts.
---
## [1.7.10] — 2026-09-28
### Changed
- **The Stop button responds immediately** — clicking **Stop** now switches the button to a disabled **Stopping…** state and logs that the scanner is finishing the current item. Previously nothing changed until that item was done (up to a minute for a large PDF being OCR'd), so the scan appeared to ignore the click.
- **Stopping a scheduled scan stops the whole job** — the remaining stages (file sources, Google Workspace) are skipped, the run is recorded as **cancelled** (⏹ in the job history), and no report email or retention run happens for the partial results.
- **Refreshing the page no longer replays the log of a finished or stopped scan** — only a scan that is still running is replayed. Results still load from the database as before.
- **Repository moved to `git.fejl40.nu`** — clone and security-contact details in the documentation now point to the Gitea repository; security reports go to support@fejl40.nu. Server installs that self-update must have their git `origin` pointed at `https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git`.
### Fixed
- **Pressing Stop looked like the scan restarting** — a stopped scan was never marked as finished, so every page load (and the 4-second reconnect check) replayed its events and brought back the Stop button and progress bar. Stopping while accounts were still being collected also announced a fresh "Scanning…" phase before stopping. For scheduled jobs, the Google stage cleared the stop request and genuinely started a new scan of all Workspace users. All three are fixed.
- **Profiles lost their selected accounts** — starting a scan overwrote the selected profile's accounts with whatever was ticked in the sidebar (dropping Google-only users and anyone hidden by the Elev/Ansat filter), and a scan with no profile selected overwrote the *first* profile. Scans no longer change profiles. The profile editor also dropped manually added accounts, and accounts that weren't loaded yet, on the next save, and Google-only accounts weren't ticked when a profile was applied — both fixed.
- **Manually added accounts were never scanned** — accounts added in the sidebar were silently left out of every scan, and accounts added in the profile editor were stored in a form Microsoft 365 couldn't use. The profile editor now looks the account up in Microsoft 365, applying a profile adds its manual accounts to the sidebar, and older profiles are resolved automatically at scan time (unknown accounts are reported and skipped, never widened to all users).
- **Deleting an Outlook message failed** with "'M365Connector' object has no attribute 'delete_message'" — the method had been lost; restored (moves the message to Deleted Items).
- **Results appeared underneath the landing card** — during a scan, results rendered below a still-visible landing/last-scan card until a manual refresh. The card is now hidden as soon as results appear.
- **Licence/OU classification data wasn't found** — the SKU and Google OU role files were read from `classification/` but lived elsewhere, so role classification silently used its fallback. The files are now where the code expects them.
- **Crashes in the background scan thread are logged** — an unexpected error in a scan used to vanish silently; it is now written to the log and the audit log.
### Internal
- Replaced deprecated imports (`fitz` → `pymupdf`, `smbprotocol.query_info` → `smbprotocol.file_info`) ahead of their removal upstream. No behaviour change.
---
## [1.7.9] — 2026-06-22
### Added
- **"Always send via SMTP" option for email reports** — new toggle in **Settings → E-mailrapport**. When the scanner is signed in to Microsoft 365 it normally sends email through Microsoft Graph; Graph reports "accepted" the instant a message is queued, which hides the case where Exchange Online later silently drops it (e.g. a recipient on a Google-hosted subdomain of your Microsoft 365 domain — the message is treated as internal, finds no mailbox, and is discarded, with no delivery and no bounce). Enabling this option makes the manual report, the test email, and the after-scan auto-email all go straight through your configured SMTP server (e.g. Google Workspace `smtp.gmail.com` / `smtp-relay.gmail.com`), bypassing the Graph routing entirely.
### Changed
- **The results grid now shows every open item by default, not just the last scan** — when you open the app (or refresh after a scheduled or manual scan), the grid loads *all* flagged items that still need action — i.e. those with no disposition — across every scan, instead of only the most recent scan session. Items you have already tagged (kept, redacted, deleted, false positive, …) drop out of the view. Re-scans are de-duplicated so each item appears once, showing its most recent state. The session picker still loads any individual past scan, and the history banner button (formerly "Latest scan") is now **"Open items"** and returns to this default view.
### Fixed
- **Interrupted scans no longer lose their results** — a scan only became visible once it was *finalised*, but the Microsoft 365 and Google scan engines skipped finalisation when a scan was stopped, and any scan cut short by a server restart, crash, or out-of-memory kill never finalised at all. Its already-found items were then stranded in the database and invisible in the grid (this is what caused "scan finished but no results shown", especially after the in-app self-update restarts). Unfinished scans are now finalised automatically on startup (nothing is scanning at boot, so any unfinished scan is known to be dead), and a manually stopped Microsoft 365 scan finalises immediately so its partial results stay visible.
- **User and group badges were missing on result cards loaded from the database** — the reviewer's display name was shown live during a scan but never saved, so cards loaded from a past scan (now the default view) lost both the person badge and the Elev/Ansat group badge. The display name is now stored with each item, and the group badge is shown from the saved role even for older items that predate this fix (where a name can't be recovered, the group badge and a resolved e-mail still appear).
- **Email reports sent via SMTP failed with "authentication failed"** — the **Settings → E-mailrapport** tab saved the SMTP username under the wrong field name, so the username never reached the mail server and sign-in was skipped — the server then rejected the unauthenticated message, which surfaced as a misleading authentication error even with a correct password or app password. The setting is now saved correctly, and configurations saved before the fix are migrated automatically.
---
## [1.7.8] — 2026-06-16
### Fixed
- **Blank results grid after a browser refresh (especially after a server restart)** — restoring the last scan session on page load was one-shot: `_sseWatchdog()` called `loadHistorySession(null)` a single time, guarded by `_initialStatusChecked`. If that attempt was blocked — a completed scan's replayed `scan_phase` event leaves a `_*ScanRunning` flag set, and the `loadHistorySession` guard then bails — nothing retried, because `sse_replay_done` (the other retry path) only fires when the SSE replay buffer is non-empty, and the buffer is empty after a server restart (so refreshing after the in-app self-update reliably showed an empty grid even though the results were in the database). The watchdog now re-attempts the restore on every 4-second poll while nothing is shown and no scan is running, clearing stale running flags first (both scan locks are confirmed free at that point). Additionally, `/api/scan/status` now reports `google_running` separately from `running` (which only ever reflected the M365 + file lock), so a refresh during a live Google scan is detected instead of being treated as idle.
---
## [1.7.7] — 2026-06-15
### Changed
- **Share modal no longer leaves a stale link in the create box** — after clicking "Create", the generated-link preview row ("Copy link:") stayed visible at the top of the modal even though the new link was already listed under Active links with its own Copy button — so it looked like the form hadn't cleared. The redundant preview row is removed; creating a link now resets the form and briefly highlights the new entry in the Active links list, where it can be copied. (The 1.7.4 fix cleared the input fields but not this preview row.)
### Added
- **Reverse-proxy / HTTPS setup guide** — new `docs/setup/ZORAXY_SETUP.md` walks through putting the scanner behind Zoraxy with a Let's Encrypt certificate on a LAN-only deployment: DNS A-record to a private IP, ACME via DNS-01 challenge (HTTP-01 cannot reach a LAN-only host), proxy rule to `127.0.0.1:5100`, binding the app to loopback with `--host 127.0.0.1`, and scanner-specific verification (SSE streaming, HTTPS share links, self-update). Linked from the README (new "HTTPS / reverse proxy" section) and SECURITY.md.
### Fixed
- **SECURITY.md corrections** — the web UI binds to `0.0.0.0` by default, not `127.0.0.1` as claimed; the MSAL token cache path was still the pre-1.x `~/.gdpr_scanner_config.json` (actual: `~/.gdprscanner/token.json`).
---
## [1.7.6] — 2026-06-11
### Fixed
- **Update restart leaked the listening socket and hopped to port 5101** — Werkzeug marks its server socket inheritable (`srv.socket.set_inheritable(True)`, unconditionally, for its debug reloader), so the in-app update's `os.execv` restart carried the old listening socket into the new process as a zombie listener: same PID listening on both 5100 (never accepted — clients hang) and 5101 (the actual server). The 1.7.3 `SO_REUSEADDR`/grace-period fix couldn't help because the port genuinely was occupied — by the restarting process itself. `_restart_self()` now marks every fd above stderr close-on-exec before the exec (`_mark_fds_cloexec()`, enumerating `/proc/self/fd` on Linux), so the old socket dies with the exec and the new server rebinds 5100 immediately.
---
## [1.7.5] — 2026-06-11
### Fixed
- **Stale UI after updating the server** — Flask served `/static/` files with no `Cache-Control` header, so browsers cached JS/CSS heuristically (often for days). After a server update — including the new in-app self-update, whose post-install reload hit the cache — the backend was new but the frontend stayed old, and fixes appeared "not to work" until a hard refresh. `SEND_FILE_MAX_AGE_DEFAULT = 0` now makes every static file revalidate via ETag: unchanged files answer with a cheap 304, changed files are re-fetched immediately on the next normal page load.
---
## [1.7.4] — 2026-06-10
### Fixed
+14 -6
View File
@@ -28,13 +28,15 @@ python -m pytest tests/ -q
**`_scan_bytes` injection pattern** — `scan_engine.py` defines no-op stubs at module level (avoids circular import). `gdpr_scanner.py` overwrites them at startup. `routes/google_scan.py` resolves them lazily via `gdpr_scanner.__getattr__`. Do not import them directly in those modules.
**Blueprints** in `routes/` — see `routes/CLAUDE.md` for SSE constraints, export, preview, scheduler, NER, audit log, viewer, and other route-specific rules.
**Blueprints** in `routes/` — see `routes/CLAUDE.md` for SSE constraints, export, preview, scheduler, NER, audit log, viewer, software update, and other route-specific rules.
**Self-update (server only)** — `routes/updates.py` powers **Settings → General → Software update**: git fetch → ff-only merge → conditional `pip install` → `os.execv` restart (same PID; marks fds close-on-exec first so Werkzeug's inheritable listening socket doesn't leak and squat the port). Only enabled for git checkouts (`_supported()` is false for frozen desktop builds). `update_gdpr.sh` is the CLI/cron equivalent. Refused while a scan runs; optional daily auto-update thread (`config.json["auto_update"]`). Restart keeps port 5100 (the port probe uses `SO_REUSEADDR` + a 10s grace). See `routes/CLAUDE.md` → "Software update".
**Frontend:** `templates/index.html` (SPA), `static/style.css` (all styles), `static/js/*.js` (11 ES modules + `state.js`). `static/app.js` is an archived monolith — no longer loaded.
**Checkpoint / resume** — all three scan engines save progress to `~/.gdprscanner/checkpoint_{prefix}.json` every 25 items. Prefixes: `m365`, `google`, `file_{source_id}`. Use `_cp_path(prefix)` — do not hard-code filenames. The Scan button calls `checkCheckpoint(() => startScan(false))` so a resume banner is offered before any grid clearing. `POST /api/scan/clear_checkpoint` globs and deletes all `checkpoint_*.json` files.
**Data dir** `~/.gdprscanner/`: `scanner.db`, `config.json`, `settings.json`, `schedule.json`, `token.json`, `delta.json`, `checkpoint_m365.json`, `checkpoint_google.json`, `checkpoint_file_*.json`, `smtp.json`, `machine_id` (**never delete** — Fernet key), `role_overrides.json`, `google_sa.json`, `google.json`, `src_toggles.json`, `app.lock`, `viewer_tokens.json`
**Data dir** `~/.gdprscanner/`: `scanner.db`, `config.json` (also holds `claude_api_key`/`claude_ner` and the `auto_update` flag), `settings.json`, `schedule.json`, `token.json`, `delta.json`, `checkpoint_m365.json`, `checkpoint_google.json`, `checkpoint_file_*.json`, `smtp.json`, `machine_id` (**never delete** — Fernet key), `role_overrides.json`, `google_sa.json`, `google.json`, `src_toggles.json`, `app.lock`, `viewer_tokens.json`. Static files are served with `SEND_FILE_MAX_AGE_DEFAULT=0` (ETag revalidation) so the UI is fresh after a self-update — do not re-add long static caching.
## Non-obvious files
@@ -44,15 +46,18 @@ python -m pytest tests/ -q
| `routes/state.py` | Shared mutable state + scan locks (not a typical Flask state file) |
| `routes/google_scan.py` | Google scan execution lives here, not in `google_connector.py` |
| `routes/viewer.py` | Viewer token + PIN API; also owns brute-force rate-limit state |
| `static/js/viewer.js` | Share modal, token CRUD, viewer PIN settings UI |
| `static/js/viewer.js` | Share modal, token CRUD, viewer PIN settings UI. Also defines `window._copyText` (HTTP-safe clipboard helper reused by `log.js`) |
| `lang/da.json` | Primary language — source of truth is `en.json` |
| `build_gdpr.py` | Desktop app builder; contains embedded `LAUNCHER_CODE` for PyInstaller |
| `routes/updates.py` | Self-update routes + `os.execv` restart with fd-cleanup; git-checkout only |
| `update_gdpr.sh` | CLI/cron self-update (fetch, ff-merge, deps, service restart) |
| `docs/setup/ZORAXY_SETUP.md` | HTTPS via Zoraxy reverse proxy (LAN-only, Let's Encrypt DNS-01) |
## Tests
212 tests in `tests/`. No integration tests for live M365/Google connections.
215 tests in `tests/`. No integration tests for live M365/Google connections.
**`tests/test_updates.py`** — 11 tests for the software-update routes (`routes/updates.py`). All git interaction goes through a mocked `_git()`; `_schedule_restart` is patched so no test re-execs the process, and `gdpr_db.log_audit_event` is patched so no test writes the real database.
**`tests/test_updates.py`** — 12 tests for the software-update routes (`routes/updates.py`). All git interaction goes through a mocked `_git()`; `_schedule_restart` is patched so no test re-execs the process, and `gdpr_db.log_audit_event` is patched so no test writes the real database. Includes `_mark_fds_cloexec` (the socket-leak guard for the restart).
**`tests/test_google_scan.py`** — 19 tests for the Google Workspace scan module. Route tests for `GET /api/google/scan/users`, `POST /api/google/scan/start`, `POST /api/google/scan/cancel`. Engine tests for `_run_google_scan` using synchronous invocation with mocked `broadcast`, `_scan_bytes`, `checkpoint.*`, `scan_engine._with_disposition`, and `gdpr_db.get_db`. The `clean_google_state` autouse fixture releases `_google_scan_lock` and clears `_google_scan_abort` after each test.
@@ -88,7 +93,10 @@ All options live in the profile `options` dict and apply to **all three scan eng
- **`get_sessions(limit=50, window_seconds=300)`** — groups `scans` rows by 300 s window. Groups built ascending, returned descending. `ref_scan_id` is the highest `scan_id` in each group. Do not change window size independently of `get_session_items`.
- **`get_session_items(ref_scan_id=N)`** — anchors 300 s window to that scan's `started_at`. Window is **symmetric**: `started_at BETWEEN ref.started_at - 300 AND ref.started_at + 300`. Do not revert to a one-sided lower bound.
- **`get_related_items(item_id, ref_scan_id, window_seconds=300)`** — self-joins `cpr_index` to find items sharing ≥1 CPR hash. Uses same 300 s symmetric window — do not change independently.
- **`GET /api/db/flagged?ref=N`** — passes `ref_scan_id` to `get_session_items`; viewer scope enforcement still applies.
- **`account_name` (display name) is persisted** (migration 11) so DB-loaded cards show the user badge. Legacy rows predating it have `account_name=''` — the frontend `_accountPill` resolves a fallback and still shows the group badge from `user_role`. `save_item` must keep writing `card["account_name"]` (both M365 and Google cards carry it).
- **Scans must be finalised or their items are invisible** — `get_session_items`, `get_open_items`, and `latest_scan_id` all filter on `finished_at IS NOT NULL`. The file scan finalises in a `finally`; M365 (`run_scan`) and Google (`_run_google_scan`) `return` early on abort, so each now calls `finish_scan` before that abort-return. A process kill (deploy/OOM/crash) mid-scan still strands a scan → **`finalize_orphan_scans()`** runs once at server startup (`gdpr_scanner.py` `__main__`, before the scheduler) and finalises every `finished_at IS NULL` scan (safe because nothing is scanning at boot). Do not add a scan-results query that ignores `finished_at` instead of fixing finalisation.
- **`get_open_items()`** — returns every flagged item with **no action taken**, across **all** scans (not just the latest session window). "Open" = no `dispositions` row, or one whose `status='unreviewed'`. Because `flagged_items` PK is `(id, scan_id)`, the same item recurs per scan; the query dedupes by `id`, keeping the row from the highest finished `scan_id`. This powers the **default landing view** so items don't drop out of sight once a newer scan opens a fresh session.
- **`GET /api/db/flagged`** — **with `?ref=N`** → `get_session_items(ref_scan_id=N)` (history mode); **without ref** → `get_open_items()` (default + viewer). Viewer scope enforcement applies to both. Do not change the no-ref `get_session_items()` default elsewhere (`export.py`, `scan_scheduler.py` still rely on latest-session for the current scan's report/email).
- See `static/js/CLAUDE.md` for the frontend history browser behaviour and `sse_replay_done` retry fix.
## Global gotchas
+2 -2
View File
@@ -21,8 +21,8 @@ very welcome.
```bash
# Clone and set up a virtual environment
git clone https://github.com/your-org/gdpr-scanner.git
cd gdpr-scanner
git clone https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git
cd GDPRScanner
python3 -m venv venv
source venv/bin/activate # macOS / Linux
venv\Scripts\activate # Windows
+11 -2
View File
@@ -534,6 +534,14 @@ API endpoints: `GET /api/update/check`, `POST /api/update/apply`, `GET/POST /api
---
### HTTPS / reverse proxy
The scanner itself serves plain HTTP. For encrypted transport on a LAN — recommended, since scan results contain CPR numbers — put it behind a TLS-terminating reverse proxy and bind the app to loopback (`--host 127.0.0.1`) so the proxy is the only way in. Share links automatically follow the HTTPS hostname, and the browser Clipboard API (Copy buttons) works natively in a secure context.
See [ZORAXY_SETUP.md](docs/setup/ZORAXY_SETUP.md) for a complete walkthrough: Zoraxy, Let's Encrypt via DNS-01 challenge (required when the hostname resolves to a private IP), proxy rule, and the scanner-specific verification steps.
---
### Article 30 report
The **Art.30** button in the filter bar generates a GDPR **Article 30 Register of Processing Activities** as a Word document (`.docx`).
@@ -627,8 +635,8 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting a pull request.
```bash
# Quick start for contributors
git clone https://github.com/your-org/gdpr-scanner.git
cd gdpr-scanner
git clone https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git
cd GDPRScanner
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python gdpr_scanner.py # GDPRScanner on port 5100 (auto-increments if in use)
@@ -737,6 +745,7 @@ See [SUGGESTIONS.md](SUGGESTIONS.md) for the full feature roadmap with implement
| `docs/manuals/MANUAL-DA.md` | End-user manual in Danish (15 sections) — served at `/manual?lang=da` |
| `docs/setup/M365_SETUP.md` | Step-by-step Microsoft 365 setup guide |
| `docs/setup/GOOGLE_SETUP.md` | Step-by-step Google Workspace setup guide |
| `docs/setup/ZORAXY_SETUP.md` | HTTPS via Zoraxy reverse proxy — LAN-only deployment with Let's Encrypt DNS-01 |
| `build_gdpr.py` | PyInstaller build script — generates `m365_launcher.py`, packages desktop app |
| `lang/en.json` | English translations (source of truth) |
| `lang/da.json` | Danish translations (primary language) |
+11 -4
View File
@@ -8,9 +8,16 @@ git commit -m "Your message"
git push origin main
```
GitHub Actions builds Windows and Linux binaries automatically and updates the
**"Latest build (main)"** pre-release at:
`https://github.com/<owner>/GDPRScanner/releases/tag/latest`
The build workflow (`.github/workflows/build.yml`) builds Windows and Linux
binaries and updates the **"Latest build (main)"** pre-release at:
`https://git.fejl40.nu/Fejl40.nu/GDPRScanner/releases/tag/latest`
> **Note:** the repository moved from GitHub to Gitea (`git.fejl40.nu`). The
> workflow still publishes with the GitHub-only `softprops/action-gh-release`
> action, so releases are **not yet produced on Gitea**. Until the workflow is
> ported to Gitea Actions, build and attach binaries manually. Server installs
> are unaffected — they self-update from `git.fejl40.nu` via git
> (see `update_gdpr.sh`).
## Versioned release
@@ -22,7 +29,7 @@ git push origin main
git push origin v1.6.14
```
Actions creates a proper GitHub Release named `v1.6.14` with auto-generated
Actions creates a proper release named `v1.6.14` with auto-generated
release notes and the built binaries attached.
Append `-beta` or `-rc` to the tag to mark it as a pre-release:
+4 -5
View File
@@ -12,14 +12,13 @@ We support only the latest release. Please update before reporting a bug.
## Reporting a Vulnerability
**Please do not file a public GitHub issue for security vulnerabilities.**
**Please do not file a public issue for security vulnerabilities.**
This tool processes sensitive personal data including Danish CPR numbers (national
identifiers). Security issues should be reported privately so a fix can be prepared
before public disclosure.
**Report to:** Open a [GitHub Security Advisory](https://github.com/your-org/gdpr-scanner/security/advisories/new)
(Settings → Security → Advisories → New draft advisory)
**Report to:** [support@fejl40.nu](mailto:support@fejl40.nu) — put "Security" in the subject line.
Please include:
- A description of the vulnerability and its potential impact
@@ -55,9 +54,9 @@ Out of scope:
- CPR numbers are stored in the SQLite database as **SHA-256 hashes only** — never in plaintext
- SMTP passwords are stored in `~/.gdprscanner/smtp.json` with chmod 600
- Microsoft OAuth tokens are stored in the MSAL token cache in `~/.gdpr_scanner_config.json`
- Microsoft OAuth tokens are stored in the MSAL token cache in `~/.gdprscanner/token.json`
- Scan results are stored locally in `~/.gdprscanner/scanner.db` — never transmitted externally
- The web UI binds to `127.0.0.1` by default — it is not designed to be exposed to the internet
- The web UI binds to `0.0.0.0` by default so reviewers on the LAN can reach it — it is not designed to be exposed to the internet. For encrypted transport, put it behind a TLS-terminating reverse proxy and bind the app to loopback with `--host 127.0.0.1` — see [docs/setup/ZORAXY_SETUP.md](docs/setup/ZORAXY_SETUP.md)
---
+36
View File
@@ -45,6 +45,42 @@ Six root causes fixed in `scan_engine.py` and `document_scanner.py`:
## Pending
### M365 not scanning M365+GWS accounts after the 1.7.12 update — ⏳ awaiting verification
**Reported 2026-09-28.** After updating the server to 1.7.12, a scan did not cover the Microsoft 365 sources for accounts shown with the unified **M365 + GWS** pill, even though they were ticked. After running the update again the problem no longer appeared. Status: being tested by the maintainer; reopen with the details below if it comes back.
**What was verified (against a copy of real profile data, scans intercepted, nothing actually scanned):**
- Applying `STAFF-M365-GWS-FULL` and pressing Scan sends `POST /api/scan/start` with all 33 accounts (31 M365+GWS, 2 M365-only), each with its M365 GUID and no `google:` ids; the Google scan gets the 31 Google emails.
- Server side, the only 1.7.12 change that filters accounts is `scan_engine._resolve_scan_users()`, which drops ids starting with `google:`. M365+GWS accounts are sent with their M365 GUID, so they pass. No other change since 1.7.9 filters accounts.
**Likely cause: a half-applied update** (not proven — no logs from the first update):
- `update_gdpr.sh` only restarts the process if a systemd unit named `gdprscanner` exists; otherwise it prints "restart GDPRScanner manually" and the old Python keeps running against the new static JS.
- A browser tab left open through the update keeps running the old JS modules until reloaded.
- Both update paths auto-stash local edits (`git stash`), silently removing any hotfix made directly on the server.
**Checking a server after an update** (install dir `/opt/gdprscanner`):
```bash
git -C /opt/gdprscanner log --oneline -1 # expect the release commit on origin/main
cat /opt/gdprscanner/VERSION # expect the released version
git -C /opt/gdprscanner status --short # expect no output
git -C /opt/gdprscanner stash list # any entries = local changes the updater set aside
systemctl show gdprscanner -p ActiveEnterTimestamp # must be after the update (CLI/cron updates only)
journalctl -u gdprscanner --since "<update time>" --no-pager | grep "\[skus\] Loaded" # one line per app start
```
The **web-UI update** (Settings → Software update) restarts via `os.execv` in the same process (same PID), so `ActiveEnterTimestamp` does **not** change — use the `[skus] Loaded …` startup line instead: there must be one after the release commit was pushed. (`[ou-roles] Loaded …` lines are logged per account-list load, not per start.)
Then hard-refresh the browser (Cmd+Shift+R) before scanning. Also confirm `git -C /opt/gdprscanner remote -v` points at `https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git` (the repo moved from GitHub).
**If it recurs, collect:** the log from "Starter scanning: …" onward (look for *"Ingen Microsoft 365-konti valgt"*, *"Sprang N konto(er) over, der kun findes i Google Workspace"*, or no Outlook/OneDrive lines at all); how the scan was started (profile / hand-ticked / scheduled job) and which profile; whether the Elev/Ansat filter or search was active; and the output of the checks above.
**Rollback:** 1.7.11 (`v1.7.11`) doesn't have this symptom but still sends Google-only accounts to Graph (404s). Reverting `b547622` on `main` restores that behaviour.
**Possible hardening (not started):**
1. The page notices when the server's `VERSION` has changed since it loaded and asks the user to reload.
2. `update_gdpr.sh` exits non-zero with a clear warning when it cannot restart the service, instead of only logging it.
**Size:** Small · **Priority:** High until verified
---
### #15 — Scan profiles ✅
Named, reusable scan configurations. Full spec in SUGGESTIONS.md §15.
**Size:** Large · **Priority:** High
+1 -1
View File
@@ -1 +1 @@
1.7.4
1.7.13
+7
View File
@@ -878,6 +878,13 @@ def _load_smtp_config() -> dict:
cfg = json.loads(_SMTP_CONFIG_PATH.read_text(encoding="utf-8"))
if cfg.get("password"):
cfg["password"] = _decrypt_password(cfg["password"])
# Normalise legacy key names written by an older settings-tab UI
# (`user`/`starttls`) to the canonical keys every reader expects
# (`username`/`use_tls`), so configs saved before the fix still work.
if "username" not in cfg and "user" in cfg:
cfg["username"] = cfg["user"]
if "use_tls" not in cfg and "starttls" in cfg:
cfg["use_tls"] = cfg["starttls"]
return cfg
except Exception:
pass
File renamed without changes.
File renamed without changes.
+21 -9
View File
@@ -1,6 +1,6 @@
# GDPR Scanner — Brugermanual
Version 1.7.4
Version 1.7.13
---
@@ -185,7 +185,7 @@ En statuslinje viser:
Resultater vises i hovedområdet efterhånden som de findes — du behøver ikke vente på, at scanningen er færdig, før du begynder at gennemgå dem.
Klik på **Stop** for at afbryde. Et kontrolpunkt gemmes automatisk, så du kan fortsætte senere.
Klik på **Stop** for at afbryde. Knappen skifter til **Stopper…**, mens scanneren gør det element færdigt, den er i gang med (det kan tage et øjeblik, f.eks. for en stor PDF), hvorefter scanningen stopper. Et kontrolpunkt gemmes automatisk, så du kan fortsætte senere. Stopper du en planlagt scanning, stoppes hele jobbet — resterende kilder springes over, og der sendes ingen rapport-e-mail for de delvise resultater.
### 4.5 Genoptag en afbrudt scanning
@@ -200,6 +200,8 @@ Klik på **▶ Genoptag** for at fortsætte fra det sted, scanningen slap. Klik
## 5. Forstå resultaterne
Når du åbner appen, viser gitteret **alle åbne fund** — alle markerede elementer, der stadig kræver handling (dvs. uden disposition), på tværs af alle dine scanninger og ikke kun den seneste. Efterhånden som du mærker elementer (behold, anonymisér, slet, falsk positiv …), forsvinder de fra denne visning, så det, der står tilbage, er dit udestående arbejde. Hvert element vises én gang med sin nyeste tilstand. Vil du i stedet se en enkelt tidligere scanning, så brug sessionsvælgeren (se *Gennemse tidligere scanningssessioner* nedenfor).
Hvert fundet element vises som et kort. Her er forklaringen på mærker og labels:
### Kildemærker
@@ -256,7 +258,7 @@ Når en scanning er afsluttet, kan du gennemse resultaterne fra en tidligere sca
- Klik på **Sessioner**-knappen i historikbanneret (der vises over resultatgitteret, når en scanning er afsluttet) for at åbne sessionsvælgeren.
- Hver række viser dato og tidspunkt, hvilke kilder der blev scannet, og hvor mange elementer der blev fundet. Et **Δ**-mærkat angiver delta-scanninger; **Seneste** markerer den nyeste session.
- Klik på en række for at indlæse den pågældende sessions resultater i gitteret. Et historikbanner erstatter statuslinjen med sessionens oplysninger.
- Klik på **Seneste scanning** i banneret for at vende tilbage til den nyeste session.
- Klik på **Åbne fund** i banneret for at forlade den tidligere session og vende tilbage til standardvisningen med alle elementer, der stadig kræver handling.
- Start af en ny scanning afslutter automatisk historiktilstanden og skifter til live-resultater.
Alle filtre, eksporter og dispositionsmærkning fungerer normalt, mens du gennemser tidligere sessioner.
@@ -380,7 +382,7 @@ Konfigurer venstre panel præcis som du ønsker det — herunder hvilke M365-kil
Klik på profil-rullemenuen i topbjælken og vælg en profil. Alle indstillinger i venstre panel — kilder, konti, indstillinger og datofilter — indlæses på én gang. Venstre panel viser derefter din aktive tilstand, og du kan justere hvad som helst, inden du scanner.
En **Ryd**-knap vises ved siden af rullemenuen, når en profil er valgt. Klik på den for at rydde profil­etiketten uden at ændre indstillingerne i venstre panel. Det er nyttigt, når du vil køre en engangsscan uden at overskrive en gemt profil.
En **Ryd**-knap vises ved siden af rullemenuen, når en profil er valgt. Klik på den for at rydde profil­etiketten uden at ændre indstillingerne i venstre panel. En scanning ændrer aldrig en gemt profil — profiler ændres kun, når du gemmer dem med **Gem** eller i profileditoren.
### Administrer profiler
@@ -441,8 +443,8 @@ Klik på **🔗**-knappen øverst til højre i topbjælken for at åbne delingsp
- **Bruger** — modtageren ser kun elementer tilhørende en bestemt medarbejder. Vælg personen fra søgefeltet; scanneren matcher automatisk både deres M365- og Google Workspace-e-mailadresser. Brug denne mulighed, når du vil give en enkelt medarbejder adgang til sine egne scanningsresultater.
3. Angiv eventuelt et **Datointerval** — brug felterne "Elementer fra" og "Elementer til" for at begrænse modtagerens visning til elementer ændret inden for en bestemt periode. Lad begge felter stå tomme for ingen datobegrænsning.
4. Vælg en **Udløbsdato** — 7 dage, 30 dage, 90 dage, 1 år eller Aldrig.
5. Klik på **Opret**. Der genereres et unikt link: `http://host:5100/view?token=…`
6. Klik på **Kopiér** for at kopiere linket til udklipsholderen, og send det til gennemgangeren.
5. Klik på **Opret**. Formularen ryddes, og det nye link vises øverst i listen **Aktive links** nedenfor, kortvarigt fremhævet.
6. Klik på **Kopiér** i linkets række for at kopiere det til udklipsholderen, og send det til gennemgangeren.
Gennemgangeren åbner linket i en browser. De kan se resultatgitteret (afgrænset til det tilladte rolleomfang) og mærke dispositioner, men kan ikke starte scanninger, ændre indstillinger, se loginoplysninger eller slette elementer.
@@ -526,7 +528,17 @@ Klik på **Gem** for at gemme, og klik derefter på **Test** for at sende en tes
> Hvis din konto har MFA (to-faktor-godkendelse) aktiveret, kan du ikke bruge din almindelige adgangskode. Du skal oprette en **app-adgangskode** i din kontos sikkerhedsindstillinger:
> - **Personlig Microsoft-konto**: account.microsoft.com/security → App-adgangskoder
> - **Gmail**: myaccount.google.com → Sikkerhed → 2-trinsbekræftelse → App-adgangskoder
> - **Gmail / Google Workspace**: myaccount.google.com → Sikkerhed → 2-trinsbekræftelse → App-adgangskoder (for Google Workspace-konti skal din administrator først tillade app-adgangskoder eller opsætte et SMTP-relay)
### Send altid via SMTP (spring Microsoft Graph over)
Når scanneren er logget på Microsoft 365, sender den normalt e-mail gennem Microsoft 365 direkte, uden at bruge SMTP-indstillingerne ovenfor. Det er praktisk, men det kan ikke levere til visse adresser — især en adresse på et Google-hostet underdomæne af dit Microsoft 365-domæne, som Microsoft 365 opfatter som intern og kasserer i stilhed (ingen levering, ingen fejl).
Slå **Send altid via SMTP (spring Microsoft Graph over)** til for at tvinge al e-mail — test-e-mails, manuelle rapporter og automatisk e-mail efter scanning — gennem den SMTP-server, du har konfigureret ovenfor. Brug dette, når dine rapporter sendes til en postkasse, som Microsoft 365 ikke kan levere til (f.eks. en Google Workspace-adresse), med `smtp.gmail.com` / `smtp-relay.gmail.com` som SMTP-vært.
### Send rapport efter manuel scanning
Slå **Send rapport efter manuel scanning** til for automatisk at sende rapporten pr. e-mail til dine konfigurerede modtagere, hver gang en manuel scanning er færdig.
### Send en rapport manuelt
@@ -649,7 +661,7 @@ Brug **Art.30**-knappen til at eksportere artikel 30-rapporten. Det er et Word-d
Scanneren klassificerer brugere som ansatte eller elever ud fra deres Microsoft 365-licenstype eller Google Workspace-organisationsenhed. Du kan bruge dette filter i kontolisten til at begrænse en scanning til kun ansatte, kun elever eller en bestemt person. Det er nyttigt, fordi reglerne for behandling af elevdata — særligt for børn under 15 år — adskiller sig fra reglerne for medarbejderdata i henhold til databeskyttelsesloven.
**Hvordan tilføjer jeg en konto, der ikke er på listen?**
I kontoafsnittet i venstre panel er der et felt **+ Tilføj konto manuelt**. Indtast e-mailadressen eller UPN'en, og den tilføjes til den aktuelle sessions kontoliste.
I kontoafsnittet i venstre panel er der et felt **+ Tilføj konto manuelt**. Indtast e-mailadressen eller UPN'en, og den tilføjes til den aktuelle sessions kontoliste. Vil du beholde kontoen fast, så tilføj den i profileditoren med **+ Tilføj konto** — scanneren slår den op i Microsoft 365, og kontoen medtages, hver gang profilen bruges, også i planlagte scanninger. Manuelt tilføjede konti skal findes i Microsoft 365.
**Kører scanneren? Jeg kan ikke se en statuslinje.**
Tjek aktivitetsloggen nederst på skærmen. Hvis en scanning kører, vises der beskeder her. Hvis du ikke ser noget, er scanningen muligvis afsluttet eller ikke startet. Kontrollér også, at du har valgt mindst én kilde og mindst én konto.
@@ -671,4 +683,4 @@ For en typisk skole- eller kommunescanning er omkostningen ubetydelig — Claude
---
*GDPR Scanner v1.7.4 — teknisk opsætning og konfiguration: se README.md*
*GDPR Scanner v1.7.13 — teknisk opsætning og konfiguration: se README.md*
+21 -9
View File
@@ -1,6 +1,6 @@
# GDPR Scanner — User Manual
Version 1.7.4
Version 1.7.13
---
@@ -185,7 +185,7 @@ A progress bar appears showing:
Results appear in the main area as they are found — you do not need to wait for the scan to finish before reviewing them.
To stop a scan, click **Stop**. A checkpoint is saved automatically so you can resume later.
To stop a scan, click **Stop**. The button changes to **Stopping…** while the scanner finishes the item it is working on (this can take a moment, e.g. for a large PDF), then the scan stops. A checkpoint is saved automatically so you can resume later. Stopping a scheduled scan stops the whole job — any remaining sources are skipped, and no report email is sent for the partial results.
### 4.5 Resuming an Interrupted Scan
@@ -200,6 +200,8 @@ Click **▶ Genoptag** to continue from where the scan left off. Click **Start f
## 5. Understanding the Results
When you open the app, the grid shows **all open items** — every flagged item that still needs action (i.e. has no disposition), across all of your scans, not just the most recent one. As you tag items (kept, redacted, deleted, false positive, …) they drop out of this view, so what remains is your outstanding work. Each item appears once, showing its most recent state. To look at a single past scan instead, use the session picker (see *Browsing past scan sessions* below).
Each flagged item appears as a card. Here is what the badges and labels mean:
### Source badges
@@ -256,7 +258,7 @@ Once a scan has completed, you can review results from any earlier scan session
- Click the **Sessions** button in the history banner (which appears above the results grid after a scan completes) to open the session picker.
- Each row shows the date and time, which sources were scanned, and how many items were flagged. A **Δ** badge marks delta scans; **Latest** marks the most recent session.
- Click any row to load that session's results into the grid. A history banner replaces the progress bar, showing the session details.
- Click **Latest scan** in the banner to jump back to the most recent session.
- Click **Open items** in the banner to leave the past session and return to the default view of all items still needing action.
- Starting a new scan automatically exits history mode and switches back to live results.
All filters, exports, and disposition tagging work normally while browsing past sessions.
@@ -380,7 +382,7 @@ Configure the sidebar exactly as you want it — including which M365 sources, G
Click the profile dropdown in the top bar and select a profile. All sidebar settings — sources, accounts, options, and date filter — are loaded at once. The sidebar then shows your live state and you can adjust anything before scanning.
A **Clear** button appears next to the dropdown after you select a profile. Click it to clear the profile label without changing the sidebar settings. This is useful when you want to run a one-off scan without overwriting a saved profile.
A **Clear** button appears next to the dropdown after you select a profile. Click it to clear the profile label without changing the sidebar settings. Running a scan never changes a saved profile — profiles only change when you save them with **Save** or in the profile editor.
### Managing profiles
@@ -441,8 +443,8 @@ Click the **🔗** button in the top-right of the top bar to open the Share pane
- **User** — the recipient sees only the items belonging to a specific employee. Select the person from the search box; the scanner matches both their M365 and Google Workspace email addresses automatically. Use this when you want to give an individual employee access to their own scan results.
3. Optionally set a **Date range** — use the "Items from" and "Items until" date fields to limit the recipient to items modified within a specific period. This lets you, for example, create a link covering only last year's scan results. Leave both fields blank for no date restriction.
4. Choose an **Expiry** — 7 days, 30 days, 90 days, 1 year, or Never.
5. Click **Create**. A unique link is generated: `http://host:5100/view?token=…`
6. Click **Copy** to copy the link to your clipboard, then send it to the reviewer.
5. Click **Create**. The form clears and the new link appears at the top of the **Active links** list below, briefly highlighted.
6. Click **Copy** on that link's row to copy it to your clipboard, then send it to the reviewer.
The reviewer opens the link in any browser. They see the results grid (filtered to their permitted scope) and can tag dispositions but cannot start scans, change settings, view credentials, or delete items.
@@ -526,7 +528,17 @@ Click **Gem** to save, then click **Test** to send a test email and verify the c
> If your account has MFA (two-factor authentication) enabled, you cannot use your regular password. You need to create an **App Password** in your account security settings:
> - **Microsoft personal account**: account.microsoft.com/security → App passwords
> - **Gmail**: myaccount.google.com → Security → 2-Step Verification → App passwords
> - **Gmail / Google Workspace**: myaccount.google.com → Security → 2-Step Verification → App passwords (for Google Workspace accounts your administrator must first allow App Passwords, or set up an SMTP relay)
### Always send via SMTP (skip Microsoft Graph)
When the scanner is signed in to Microsoft 365, it normally sends email through Microsoft 365 directly, without using the SMTP settings above. This is convenient, but it cannot deliver to some addresses — most notably an address on a Google-hosted subdomain of your Microsoft 365 domain, which Microsoft 365 treats as internal and silently discards (no delivery, no error).
Turn on **Send altid via SMTP (spring Microsoft Graph over)** to force all email — test emails, manual reports, and the after-scan auto-email — through the SMTP server you configured above. Use this when your reports go to a mailbox Microsoft 365 won't deliver to (for example a Google Workspace address), with `smtp.gmail.com` / `smtp-relay.gmail.com` as the SMTP host.
### Email report after manual scan
Turn on **Send rapport efter manuel scanning** to automatically email the report to your configured recipients every time a manual scan finishes.
### Sending a report manually
@@ -649,7 +661,7 @@ Use the **Art.30** button to export the Article 30 report. It is a Word document
The scanner classifies users as staff (Ansat) or students (Elev) based on their Microsoft 365 licence type or Google Workspace organisational unit. You can use this filter in the accounts list to restrict a scan to only staff, only students, or a specific individual. This is useful because the rules for processing student data — especially for children under 15 — differ from staff data under Databeskyttelsesloven.
**How do I add an account that is not in the list?**
In the accounts section of the sidebar, there is an **+ Tilføj konto manuelt** (Add account manually) field. Enter the email address or UPN and it will be added to the current session's account list.
In the accounts section of the sidebar, there is an **+ Tilføj konto manuelt** (Add account manually) field. Enter the email address or UPN and it will be added to the current session's account list. To keep an account permanently, add it in the profile editor with **+ Tilføj konto** — the scanner looks it up in Microsoft 365, and the account is included whenever that profile is used, including in scheduled scans. Manually added accounts must exist in Microsoft 365.
**Is the scanner running? I cannot see a progress bar.**
Check the activity log at the bottom of the screen. If a scan is running it will show messages there. If you see nothing, the scan may have completed or not started. Also check that you have at least one source ticked and at least one account selected.
@@ -671,4 +683,4 @@ For a typical school or municipality scan the cost is negligible — Claude Haik
---
*GDPR Scanner v1.7.4 — for technical setup and configuration see README.md*
*GDPR Scanner v1.7.13 — for technical setup and configuration see README.md*
+148
View File
@@ -0,0 +1,148 @@
# HTTPS via Zoraxy Reverse Proxy
Step-by-step guide for putting GDPRScanner behind [Zoraxy](https://github.com/tobychui/zoraxy) with a Let's Encrypt certificate, on a LAN-only deployment.
Why bother on an internal network:
- **Encryption in transit** — the scanner streams CPR numbers, document previews, and share links. Serving that over plain HTTP to DPO reviewers is itself a compliance finding.
- **Secure context** — the browser Clipboard API (share-link Copy buttons) only exists on HTTPS or localhost. Over plain HTTP the app falls back to a legacy copy mechanism.
- **A real hostname** — `https://gdprscanner.example.dk` instead of `http://10.x.x.x:5100` in share links, bookmarks, and emails.
This guide assumes Zoraxy runs **on the same host** as the scanner. If it runs elsewhere, replace `127.0.0.1:5100` with the scanner host's LAN IP and firewall port 5100 to the Zoraxy host only.
---
## 1. DNS record
Create an A-record for the hostname pointing at the server's **LAN IP**:
```
gdprscanner.example.dk A 10.x.x.x
```
A public DNS record pointing at a private IP is fine — outsiders can resolve the name but cannot route to the address, which is exactly the "LAN-only" goal.
> **Consequence:** because the server is not reachable from the internet, Let's Encrypt's default HTTP-01 challenge cannot work. The certificate **must** be issued via the **DNS-01 challenge** (step 4). If you prefer not to publish the internal IP at all, use an internal/split-horizon DNS record instead — DNS-01 still works since it validates against the public DNS zone, not the server.
---
## 2. Install Zoraxy
```bash
mkdir -p /opt/zoraxy && cd /opt/zoraxy
wget -O zoraxy https://github.com/tobychui/zoraxy/releases/latest/download/zoraxy_linux_amd64
chmod +x zoraxy
```
`/etc/systemd/system/zoraxy.service`:
```ini
[Unit]
Description=Zoraxy reverse proxy
After=network.target
[Service]
WorkingDirectory=/opt/zoraxy
ExecStart=/opt/zoraxy/zoraxy
Restart=always
[Install]
WantedBy=multi-user.target
```
```bash
systemctl daemon-reload && systemctl enable --now zoraxy
```
Open the management UI at `http://<server-ip>:8000` and create the admin account.
> Menu names below may differ slightly between Zoraxy versions — the concepts to look for are: ACME certificate with DNS challenge, host-based proxy rule, TLS on the incoming port.
---
## 3. Incoming port and TLS
In Zoraxy's global settings:
- Set the incoming proxy port to **443** and enable **TLS**.
- Enable **force-redirect port 80 → 443** so plain-HTTP visits upgrade automatically.
---
## 4. Certificate via ACME (DNS-01)
In **TLS / SSL Certificates → ACME**:
1. Enter the hostname (`gdprscanner.example.dk`).
2. Enable the **DNS challenge** and select the DNS provider that hosts your zone (Cloudflare, Simply.com, etc.).
3. Paste the provider's **API token/credentials** — created in the DNS provider's control panel.
4. Request the certificate. Zoraxy renews it automatically.
If your DNS host has no API, Zoraxy can generate a **self-signed certificate** as a fallback — it works, but every client machine must trust it manually. Getting a DNS API token is the better one-time investment.
---
## 5. Proxy rule
**HTTP Proxy → New Proxy Rule**:
| Field | Value |
|---|---|
| Matching hostname | `gdprscanner.example.dk` |
| Target | `127.0.0.1:5100` |
| TLS to target | Off (the scanner speaks plain HTTP locally) |
---
## 6. Close the side doors
**Bind the scanner to loopback** so only Zoraxy can reach Flask. Wherever the scanner is started (systemd unit or `start_gdpr.sh`), add:
```bash
--host 127.0.0.1
```
After a restart, `http://<server-ip>:5100` stops responding by design. The in-app self-update restart preserves the argument.
Optional hardening:
- Add a Zoraxy **Access Rule** whitelisting your LAN CIDR (e.g. `10.0.0.0/8`) on the proxy rule.
- Firewall the Zoraxy **management port 8000** to admin machines only.
---
## 7. Firewall / perimeter checklist
The Zoraxy whitelist (step 6) is an **application-layer** control — a rejected request has still completed the TCP and TLS handshake against your box, and any proxy host you forget to tag is fully exposed. The firewall is the real perimeter. Work this checklist whenever you stand up or replace the edge firewall:
- [ ] **No inbound port-forward unless a service is intentionally public.** A LAN-only deployment needs *zero* inbound forwards — DNS-01 (step 4) is outbound-only, so certificates issue and renew with the firewall fully closed.
- [ ] **If any service is intentionally public** (e.g. a media server), forward **443 only to the Zoraxy host** — never to individual app hosts. Everything then enters through Zoraxy, where the per-host Access Rule decides public vs. private.
- [ ] **The per-host whitelist stays your public/private boundary even with the firewall in place** — it is not made redundant by the firewall. Public hosts use the `default` rule; every internal-only host gets **Local Access Only**.
- [ ] **New proxy hosts default to public.** Zoraxy applies the `default` rule to any host with no rule set, so a freshly-added internal service is reachable the moment it exists. Set its Access Rule to **Local Access Only** *at creation time*.
- [ ] **Management ports are LAN-only.** Zoraxy admin (`:8000`) and any app admin UI must never be forwarded; tag them **Local Access Only** as well.
- [ ] **Verify from off-network.** From a connection outside the LAN (e.g. a phone on mobile data), confirm private hostnames are blocked and only the intentionally-public ones respond:
```bash
curl -v https://gdprscanner.example.dk # should fail/refuse from outside
nmap -Pn -p 80,443,5100 <your-public-IP> # only intentionally-open ports listed
```
---
## 8. Verify the scanner-specific behaviour
1. `https://gdprscanner.example.dk` loads with a valid padlock; `http://` redirects.
2. **Run a scan and watch result cards stream in live** — that is the Server-Sent Events connection (`/api/scan/stream`) passing through the proxy. If progress stalls while the scan log advances, look at proxy buffering/timeout settings.
3. Create a **share link** — it must start with `https://gdprscanner.example.dk/view?token=…`. The app uses the page origin automatically on HTTPS (the LAN-IP rewrite only applies when browsing at localhost). The Copy buttons now use the native Clipboard API.
4. **Settings → General → Software update → Check for updates** still works (outbound git fetch is unaffected by the proxy).
---
## Troubleshooting
| Symptom | Cause / fix |
|---|---|
| Certificate request fails | HTTP-01 attempted against an unreachable host — make sure the **DNS challenge** is selected and the API credentials are for the zone's actual DNS host |
| Cards don't stream during scans | Proxy buffering the SSE response — check Zoraxy timeout/buffering settings for the rule |
| Share links still show the LAN IP | Page was loaded via the old `http://<ip>:5100` URL — use the HTTPS hostname; links follow the page origin |
| `http://<ip>:5100` still reachable | The `--host 127.0.0.1` flag is missing from the scanner's launch command |
+2 -2
View File
@@ -106,7 +106,7 @@ except ImportError:
MASK_AVAILABLE = False
try:
import fitz as _fitz # PyMuPDF — for secure (sanitised) PDF redaction
import pymupdf as _fitz # PyMuPDF — for secure (sanitised) PDF redaction
PYMUPDF_AVAILABLE = True
except ImportError:
PYMUPDF_AVAILABLE = False
@@ -2727,7 +2727,7 @@ def count_faces_in_file(path, poppler_path=None, neighbors: int = 4) -> int:
elif ext == ".pdf":
if PYMUPDF_AVAILABLE:
import fitz as _fitz
import pymupdf as _fitz
doc = _fitz.open(str(path))
for page_idx in range(min(5, len(doc))):
pix = doc[page_idx].get_pixmap(dpi=100)
+1 -1
View File
@@ -38,7 +38,7 @@ try:
FileAttributes, FilePipePrinterAccessMask, ShareAccess,
ImpersonationLevel,
)
from smbprotocol.query_info import FileDirectoryInformation
from smbprotocol.file_info import FileDirectoryInformation
SMB_OK = True
except ImportError:
SMB_OK = False
+68 -2
View File
@@ -29,11 +29,14 @@ Usage (from gdpr_scanner.py)
import hashlib
import json
import logging
import sqlite3
import time
from pathlib import Path
from typing import Iterator
logger = logging.getLogger(__name__)
from pathlib import Path as _P
_DATA_DIR = _P.home() / ".gdprscanner"
_DATA_DIR.mkdir(exist_ok=True)
@@ -225,6 +228,7 @@ _MIGRATIONS: list[tuple[int, str]] = [
emailed INTEGER NOT NULL DEFAULT 0,
error TEXT NOT NULL DEFAULT ''
)"""),
(11, "ALTER TABLE flagged_items ADD COLUMN account_name TEXT NOT NULL DEFAULT ''"),
]
@@ -326,8 +330,8 @@ class ScanDB:
url, drive_id, size_kb, modified, cpr_count, risk,
thumb_b64, thumb_mime, attachments, user_role, transfer_risk,
special_category, face_count, exif_json, full_path,
email_count, phone_count, body_excerpt, scanned_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
email_count, phone_count, body_excerpt, account_name, scanned_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
(
card.get("id", ""),
scan_id,
@@ -354,6 +358,7 @@ class ScanDB:
card.get("email_count", 0),
card.get("phone_count", 0),
card.get("body_excerpt", ""),
card.get("account_name", ""),
now,
),
)
@@ -432,6 +437,33 @@ class ScanDB:
c.commit()
def finalize_orphan_scans(self) -> int:
"""Finalise scans left unfinished by a crash, kill, or mid-scan restart.
After a fresh process start nothing is scanning, so any scan still
carrying finished_at IS NULL is dead — the process that owned it is gone.
Its already-saved flagged_items were stranded: both get_session_items
and get_open_items require finished_at, so those items are invisible and
effectively lost. Finalising the orphans on startup makes them show up
and prevents permanent data loss from interrupted scans (the M365 and
Google engines return early on abort and never reach finish_scan; only
the file scan finalises in a finally block).
Safe to call only when no scan is running (i.e. at startup). Returns the
number of scans finalised.
"""
rows = self._connect().execute(
"SELECT id, total_scanned FROM scans WHERE finished_at IS NULL"
).fetchall()
count = 0
for sid, total in rows:
try:
self.finish_scan(sid, total or 0)
count += 1
except Exception as e:
logger.warning("[db] finalize_orphan_scans: scan %s failed: %s", sid, e)
return count
# ── Query helpers ─────────────────────────────────────────────────────────
def latest_scan_id(self) -> int | None:
@@ -536,6 +568,40 @@ class ScanDB:
result.append(d)
return result
def get_open_items(self) -> list[dict]:
"""Return every flagged item across all scans that has no action taken.
"Open" means the item has no disposition row (or a row whose status is
still 'unreviewed'). Unlike get_session_items this is NOT limited to the
latest scan window — it surfaces all outstanding items so nothing slips
out of view once a newer scan starts a fresh session.
flagged_items has a composite PK of (id, scan_id), so the same logical
item appears once per scan that flagged it. We deduplicate by id, keeping
the row from the most recent finished scan, so each open item shows once.
"""
rows = self._connect().execute(
"""SELECT fi.*, COALESCE(d.status, 'unreviewed') AS disposition
FROM flagged_items fi
JOIN scans s ON fi.scan_id = s.id
LEFT JOIN dispositions d ON d.item_id = fi.id
WHERE s.finished_at IS NOT NULL
AND (d.item_id IS NULL OR d.status = 'unreviewed')
AND fi.scan_id = (
SELECT MAX(fi2.scan_id)
FROM flagged_items fi2
JOIN scans s2 ON fi2.scan_id = s2.id
WHERE fi2.id = fi.id AND s2.finished_at IS NOT NULL
)
ORDER BY fi.cpr_count DESC""",
).fetchall()
result = []
for r in rows:
d = dict(r)
d["attachments"] = json.loads(d.get("attachments") or "[]")
result.append(d)
return result
def get_related_items(self, item_id: str, ref_scan_id: int | None = None,
window_seconds: int = 300) -> list[dict]:
"""Return flagged items from the same session that share at least one CPR
+26 -11
View File
@@ -317,6 +317,11 @@ app = Flask(__name__,
template_folder=_os.path.join(_BASE_DIR, "templates"),
static_folder=_os.path.join(_BASE_DIR, "static"))
# Static files must revalidate on every load (cheap 304s via ETag). Without
# this there is no Cache-Control header and browsers cache JS/CSS heuristically
# for days — after a self-update the backend is new but the UI stays stale.
app.config["SEND_FILE_MAX_AGE_DEFAULT"] = 0
# Session secret — derived from machine_id so it survives restarts without a separate file.
# machine_id is also the Fernet key (base64-encoded 32 bytes); we use its raw bytes as the secret.
try:
@@ -739,9 +744,13 @@ def _build_article30_docx() -> tuple[bytes, str]:
# user_ids are stored in the options JSON column
opts_json = json.loads(row["options"] or "{}") if row else {}
for u in opts_json.get("user_ids", []):
# Scheduled scans of editor-saved profiles stored bare id
# strings; .get() on those raised and dropped the whole map.
if not isinstance(u, dict):
continue
uid = u.get("id", "")
name = u.get("displayName", "")
if uid and name:
if uid and name and name != uid:
_acct_map[uid] = name
except Exception:
pass
@@ -1522,16 +1531,9 @@ def local_ip():
def scan_stream():
q = queue.Queue(maxsize=512)
_sse_queues.append(q)
# Filter replay buffer: only include events from the current scan
# (avoids replaying stale events from a previous scan)
replay_scan_id = _sse_mod._current_scan_id
buf = []
if replay_scan_id:
for msg in list(_sse_buffer):
if f'"scan_id": "{replay_scan_id}"' in msg:
buf.append(msg)
else:
buf = list(_sse_buffer)
# Replay only the running scan's events, never those of a finished or
# stopped scan (see sse.replay_messages)
buf = _sse_mod.replay_messages()
def generate():
try:
yield ": connected\n\n"
@@ -2300,6 +2302,19 @@ Example --settings file with SMTP:
print(f"\n GDPRScanner\n ──────────────────────────────")
print(f" Open: http://{args.host}:{args.port}")
# Recover scans left unfinished by a crash / kill / mid-scan restart.
# Nothing is scanning at startup, so any scan with finished_at IS NULL is
# dead; finalising it makes its already-saved items visible again instead
# of stranding them (both get_session_items and get_open_items require a
# finished scan). Must run before the scheduler can start a new scan.
try:
if DB_OK:
_recovered = _get_db().finalize_orphan_scans()
if _recovered:
print(f" Recovered {_recovered} unfinished scan(s) from a prior restart")
except Exception as _orphan_err:
print(f" Orphan-scan recovery: failed ({_orphan_err})")
# Start in-process scheduler (#19)
try:
import scan_scheduler as _sched_mod
+10 -2
View File
@@ -31,6 +31,8 @@ import threading
from pathlib import Path
from typing import Iterator, Optional
logger = logging.getLogger(__name__)
# ── google-auth / google-api-python-client ────────────────────────────────────
try:
from google.oauth2 import service_account
@@ -93,15 +95,21 @@ _MAX_EXPORT_BYTES = 20 * 1024 * 1024 # 20 MB
_OU_ROLES_PATH = Path(__file__).parent / "classification" / "google_ou_roles.json"
def _load_ou_roles() -> tuple[list, list]:
"""Load student/staff OU prefix lists from skus/google_ou_roles.json.
"""Load student/staff OU prefix lists from classification/google_ou_roles.json.
Returns (student_prefixes, staff_prefixes) — both lowercased."""
try:
import json as _j
data = _j.loads(_OU_ROLES_PATH.read_text(encoding="utf-8"))
students = [p.lower() for p in data.get("student_ou_prefixes", [])]
staff = [p.lower() for p in data.get("staff_ou_prefixes", [])]
logger.info("[ou-roles] Loaded %d student + %d staff OU prefixes from classification/google_ou_roles.json",
len(students), len(staff))
return students, staff
except Exception:
except FileNotFoundError:
logger.warning("[ou-roles] classification/google_ou_roles.json not found — using hardcoded fallback prefixes only")
return ["/elever", "/students"], ["/personale", "/staff", "/lærere", "/ansatte"]
except Exception as e:
logger.error("[ou-roles] Failed to load classification/google_ou_roles.json: %s — using hardcoded fallback prefixes only", e)
return ["/elever", "/students"], ["/personale", "/staff", "/lærere", "/ansatte"]
def classify_ou_role(org_unit_path: str) -> str:
+11 -1
View File
@@ -106,7 +106,7 @@
"history_lbl": "Historik",
"history_items": "fund",
"history_btn_sessions": "Sessioner",
"history_btn_latest": "Seneste scanning",
"history_btn_latest": "Åbne fund",
"history_picker_empty": "Ingen tidligere scanninger",
"history_delta_badge": "Delta",
"history_latest_badge": "Seneste",
@@ -260,6 +260,11 @@
"m365_admin_note": "Viser kun din konto. For at liste alle brugere skal en administrator give samtykke til <strong>User.Read.All</strong> i Azure Portal, eller tilføj konti manuelt nedenfor.",
"m365_btn_scan": "Scan",
"m365_btn_stop": "Stop",
"m365_no_m365_accounts": "Ingen Microsoft 365-konti valgt — Microsoft 365-kilderne springes over.",
"m365_btn_stopping": "Stopper…",
"m365_scan_stopping": "Stopper scanning — færdiggør det aktuelle element…",
"m365_scan_stopped": "Scanning stoppet.",
"m365_sched_stopped": "stoppet",
"m365_pill_flagged": "markerede",
"m365_pill_scanned": "scannet",
"m365_filter_all_sources": "Alle kilder",
@@ -366,6 +371,7 @@
"m365_smtp_recipients_hint": "Adskil med komma eller semikolon",
"m365_smtp_save": "Gem",
"m365_smtp_auto_email_manual": "Send rapport efter manuel scanning",
"m365_smtp_prefer_smtp": "Send altid via SMTP (spring Microsoft Graph over)",
"m365_smtp_send": "Send nu",
"m365_smtp_saved": "Indstillinger gemt.",
"m365_smtp_sending": "Sender…",
@@ -512,6 +518,7 @@
"m365_filter_special_only": "Art. 9 følsomme data",
"m365_badge_special": "Art.9",
"m365_phase_scanning": "Scanner…",
"m365_skipped_google_accounts": "Sprang {n} konto(er) over, der kun findes i Google Workspace — ikke i Microsoft 365",
"a30_special_cat": "Særlige kategorier (artikel 9)",
"a30_special_cat_note": "Disse elementer indeholder helbredsoplysninger, straffeoplysninger, biometriske data, religiøse, etniske, fagforeningsmæssige, politiske eller seksuelle oplysninger. Et eksplicit retsgrundlag (art. 9, stk. 2) og eventuelt en konsekvensanalyse (art. 35) er påkrævet.",
"a30_col_special": "Art. 9",
@@ -531,6 +538,9 @@
"m365_profile_save_tip": "Gem aktuelle indstillinger som profil",
"m365_profile_save_prompt": "Profilnavn:",
"m365_profile_applied": "Profil indlæst",
"m365_profile_acct_manual": "Manuel",
"m365_profile_acct_unresolved": "Ikke indlæst",
"m365_profile_manual_not_found": "Manuel konto ikke fundet i M365",
"m365_profile_saved": "Profil gemt",
"m365_profile_manage_btn": "Profiler",
"m365_profile_clear_btn": "Ryd",
+11 -1
View File
@@ -167,7 +167,7 @@
"history_lbl": "Verlauf",
"history_items": "Treffer",
"history_btn_sessions": "Sessionen",
"history_btn_latest": "Letzter Scan",
"history_btn_latest": "Offene Einträge",
"history_picker_empty": "Keine früheren Scans",
"history_delta_badge": "Delta",
"history_latest_badge": "Aktuell",
@@ -260,6 +260,11 @@
"m365_admin_note": "Es wird nur Ihr Konto angezeigt. Um alle Benutzer aufzulisten, muss ein Administrator <strong>User.Read.All</strong> im Azure Portal genehmigen oder Konten manuell unten hinzufügen.",
"m365_btn_scan": "Scannen",
"m365_btn_stop": "Stop",
"m365_no_m365_accounts": "Keine Microsoft 365-Konten ausgewählt — Microsoft 365-Quellen werden übersprungen.",
"m365_btn_stopping": "Wird gestoppt…",
"m365_scan_stopping": "Scan wird gestoppt — aktuelles Element wird abgeschlossen…",
"m365_scan_stopped": "Scan gestoppt.",
"m365_sched_stopped": "gestoppt",
"m365_pill_flagged": "markiert",
"m365_pill_scanned": "gescannt",
"m365_filter_all_sources": "Alle Quellen",
@@ -366,6 +371,7 @@
"m365_smtp_recipients_hint": "Komma- oder semikolongetrennt",
"m365_smtp_save": "Speichern",
"m365_smtp_auto_email_manual": "Bericht nach manueller Suche senden",
"m365_smtp_prefer_smtp": "Immer via SMTP senden (Microsoft Graph überspringen)",
"m365_smtp_send": "Jetzt senden",
"m365_smtp_saved": "Einstellungen gespeichert.",
"m365_smtp_sending": "Senden…",
@@ -512,6 +518,7 @@
"m365_filter_special_only": "Art. 9 besondere Kategorien",
"m365_badge_special": "Art.9",
"m365_phase_scanning": "Scanne…",
"m365_skipped_google_accounts": "{n} reine Google Workspace-Konten übersprungen — nicht in Microsoft 365",
"a30_special_cat": "Besondere Kategorien (Artikel 9)",
"a30_special_cat_note": "Diese Elemente enthalten Gesundheits-, Straf-, biometrische, religiöse, ethnische, gewerkschaftliche, politische oder sexuelle Daten. Eine ausdrückliche Rechtsgrundlage (Art. 9 Abs. 2) und ggf. eine DSFA (Art. 35) sind erforderlich.",
"a30_col_special": "Art. 9",
@@ -531,6 +538,9 @@
"m365_profile_save_tip": "Aktuelle Einstellungen als Profil speichern",
"m365_profile_save_prompt": "Profilname:",
"m365_profile_applied": "Profil geladen",
"m365_profile_acct_manual": "Manuell",
"m365_profile_acct_unresolved": "Nicht geladen",
"m365_profile_manual_not_found": "Manuelles Konto in M365 nicht gefunden",
"m365_profile_saved": "Profil gespeichert",
"m365_profile_manage_btn": "Profile",
"m365_profile_clear_btn": "Löschen",
+11 -1
View File
@@ -106,7 +106,7 @@
"history_lbl": "History",
"history_items": "items",
"history_btn_sessions": "Sessions",
"history_btn_latest": "Latest scan",
"history_btn_latest": "Open items",
"history_picker_empty": "No past scans",
"history_delta_badge": "Delta",
"history_latest_badge": "Latest",
@@ -260,6 +260,11 @@
"m365_admin_note": "Only showing your account. To list all users, an admin must grant <strong>User.Read.All</strong> consent in Azure Portal, or add accounts manually below.",
"m365_btn_scan": "Scan",
"m365_btn_stop": "Stop",
"m365_no_m365_accounts": "No Microsoft 365 accounts selected — skipping the Microsoft 365 sources.",
"m365_btn_stopping": "Stopping…",
"m365_scan_stopping": "Stopping scan — finishing the current item…",
"m365_scan_stopped": "Scan stopped.",
"m365_sched_stopped": "stopped",
"m365_pill_flagged": "flagged",
"m365_pill_scanned": "scanned",
"m365_filter_all_sources": "All sources",
@@ -366,6 +371,7 @@
"m365_smtp_recipients_hint": "Comma or semicolon separated",
"m365_smtp_save": "Save",
"m365_smtp_auto_email_manual": "Email report after manual scan",
"m365_smtp_prefer_smtp": "Always send via SMTP (skip Microsoft Graph)",
"m365_smtp_send": "Send now",
"m365_smtp_saved": "Settings saved.",
"m365_smtp_sending": "Sending…",
@@ -512,6 +518,7 @@
"m365_filter_special_only": "Art. 9 special category",
"m365_badge_special": "Art.9",
"m365_phase_scanning": "Scanning…",
"m365_skipped_google_accounts": "Skipped {n} Google Workspace-only account(s) — not in Microsoft 365",
"a30_special_cat": "Art. 9 special category items",
"a30_special_cat_note": "These items contain health, criminal, biometric, religious, ethnic, trade union, political, or sexual orientation data. An explicit legal basis (Art. 9(2)) and possibly a DPIA (Art. 35) is required.",
"a30_col_special": "Art. 9",
@@ -531,6 +538,9 @@
"m365_profile_save_tip": "Save current settings as a profile",
"m365_profile_save_prompt": "Profile name:",
"m365_profile_applied": "Profile loaded",
"m365_profile_acct_manual": "Manual",
"m365_profile_acct_unresolved": "Not loaded",
"m365_profile_manual_not_found": "Manual account not found in M365",
"m365_profile_saved": "Profile saved",
"m365_profile_manage_btn": "Profiles",
"m365_profile_clear_btn": "Clear",
+30
View File
@@ -224,6 +224,11 @@ class M365Connector:
return {"displayName": "App (service account)", "id": "", "mail": ""}
return self._get("/me")
def lookup_user(self, upn: str) -> dict:
"""Look up one directory user by UPN or email (requires User.Read.All)."""
return self._get(f"/users/{upn}",
{"$select": "id,displayName,mail,userPrincipalName,assignedLicenses"})
def list_users(self, top: int = 999) -> list:
"""List licensed domain users in the tenant (requires User.Read.All).
@@ -552,6 +557,8 @@ class M365Connector:
r.raise_for_status()
return True # 204 No Content = success
raise _requests.exceptions.RetryError(f"Gave up after {self._MAX_RETRIES} attempts: {url}")
def delete_message(self, user_id: str, message_id: str) -> bool:
"""Move an email to Deleted Items (soft delete)."""
base = "/me" if (not user_id or user_id == "me") else f"/users/{user_id}"
try:
@@ -1150,6 +1157,29 @@ class M365Connector:
break
return result
def build_role_sku_map(self, users: list) -> dict:
"""The {skuId: skuPartNumber} map classify_user_role() needs.
Tenant SKUs (get_subscribed_skus) merged with a sample of per-user
licenseDetails — the tenant call may only return the admin's own SKUs.
Shared by GET /api/users and the scan engine so both classify roles
the same way. Never raises; an empty map means ID-only classification.
"""
try:
sku_map = self.get_subscribed_skus()
except Exception:
sku_map = {}
try:
per_user = self.build_sku_map_from_users(users)
if per_user:
added = len(set(per_user) - set(sku_map))
sku_map.update(per_user)
if added:
logger.info("[skus] merged %d additional SKU(s) from per-user licenseDetails", added)
except Exception:
pass
return sku_map
def classify_user_role(self, assigned_licenses: list,
sku_map: dict) -> str:
"""Return 'student', 'staff', or 'other' based on assigned O365 licenses.
+6 -1
View File
@@ -68,6 +68,9 @@ Exception hierarchy (all inherit `M365Error(Exception)`):
- **Graph preferred over SMTP** — `smtp_test` and `send_report` try `_send_email_graph()` first; fall back to SMTP only if Graph raises. If Graph fails and no SMTP host saved, the Graph exception surfaces directly.
- **Auto-email after manual scan** — `_maybe_send_auto_email()` in `routes/scan.py` called from the `_run()` thread after `run_scan()` returns. Reads `smtp_cfg.get("auto_email_manual")`; no-ops if false, no flagged items, or no recipients.
- **Gmail vs Google Workspace** — auth error handlers check if SMTP username ends in `@gmail.com`/`@googlemail.com`; custom domains are treated as Google Workspace and error message points to the Workspace admin console.
- **Canonical SMTP config keys are `username` and `use_tls`** — all backend readers (`smtp_test`, `_send_report_email`, `_send_email_graph`) use these. The Settings → E-mailrapport tab (`scheduler.js`) historically saved `user`/`starttls`, which left `username` empty so `server.login()` was skipped and the server rejected the send. Frontend now sends the canonical keys, and `_load_smtp_config()` normalises legacy `user`→`username` / `starttls`→`use_tls` for already-saved configs. The send-report modal (`scan.js`) already used the canonical keys. Keep both UIs and the backend on `username`/`use_tls`.
- **Graph 202 ≠ delivered** — `_send_email_graph` returns on Graph's HTTP 202 (queued), and `smtp_test`/`send_report` treat that as success and never fall back to SMTP. A recipient on a domain Exchange Online considers an accepted/internal domain (e.g. a Google-hosted subdomain of the O365 domain) is silently dropped after the 202. There is no in-app fix for that routing; reaching such recipients requires SMTP (e.g. Google Workspace `smtp.gmail.com`/`smtp-relay.gmail.com`) or fixing Exchange Accepted Domains.
- **`prefer_smtp` config flag** — when truthy, `smtp_test`, `send_report`, and `_maybe_send_auto_email` (routes/scan.py) skip the Graph path entirely and send via SMTP. This is the in-app escape hatch for the Graph-202 routing trap above. The gate is `... and not smtp_cfg.get("prefer_smtp")` on each Graph branch — keep all three in sync. UI: `#st-smtpPreferSmtp` toggle (key `m365_smtp_prefer_smtp`), saved/loaded by `scheduler.js`.
## Scheduler — scan_scheduler.py + routes/scheduler.py
@@ -96,6 +99,7 @@ Optional AI-powered NER replacing spaCy. Activated via `config.json` keys `claud
- **`apply_update()` never restarts itself** — callers decide. Tests patch `_schedule_restart`; the auto-update thread calls `_restart_self()` directly.
- **Auto-update thread** — `start_auto_update_thread()` called from `gdpr_scanner.py` `__main__`. Hourly tick, applies at most once per 24 h when `config.json["auto_update"]` is true; skips (and retries next tick) while a scan runs.
- **`update_gdpr.sh`** — standalone CLI/cron equivalent of the same logic; keep stash/ff-only/requirements behaviour in sync.
- **Half-applied updates look like code bugs** — `update_gdpr.sh` restarts only if a `gdprscanner` systemd unit exists (else the old Python keeps running under the new static JS), open browser tabs keep old JS modules until reloaded, and both paths auto-stash local edits. Before debugging a post-update regression, verify the server state (HEAD, `VERSION`, clean `git status`, `git stash list`, service start time after the update) — see the "M365 not scanning M365+GWS accounts after the 1.7.12 update" entry in `TODO.md`.
## Viewer mode — routes/viewer.py
@@ -112,5 +116,6 @@ Optional AI-powered NER replacing spaCy. Activated via `config.json` keys `claud
## Gotchas
- **`_load_settings()` return** — does NOT include `file_sources`. Returns only: sources, user_ids, options, retention_years, fiscal_year_end, email_to.
- **`_save_settings()` clobbers profile fields** — called on every M365 scan start with only M365 sources/user_ids/options. The fix in `app_config.py` preserves `google_sources` and `file_sources` and rebuilds `sources` as `m365_src + google_src + file_src`. Do not simplify away this merge logic.
- **`POST /api/scan/start` must not write profiles** — it used to call `_save_settings()` with the sidebar state, which overwrote the active profile's `user_ids` (dropping Google-only users and role-filtered-out users) and, with no active profile, clobbered `profiles[0]`. Profiles change only via `/api/profiles/save`. `profile_id` in the start payload is audit-only. Do not re-add the call.
- **`_save_settings()` merge logic** — still used by `/api/settings/save`. It preserves `google_sources` and `file_sources` and rebuilds `sources` as `m365_src + google_src + file_src`. Do not simplify away this merge logic.
- **`loadLastScanSummary()` timing** — must only be called after the first `/api/scan/status` poll resolves (inside `_sseWatchdog` in `results.js`, guarded by `_initialStatusChecked`). Calling it on `DOMContentLoaded` shows a stale "no results" card during a live scan after a hard refresh.
+12 -2
View File
@@ -180,7 +180,11 @@ def db_get_disposition(item_id):
@bp.route("/api/db/flagged")
def db_flagged_items():
"""Return flagged items from the most recent completed scan session.
"""Return flagged items for the results grid.
With ?ref=N, returns the items from that specific past scan session (history
mode). Without ref, returns every item still awaiting action across all
scans (the default landing view) — not just the latest session window.
Used by the read-only viewer to load results without an active SSE connection.
Respects viewer_scope.role stored in the session for scoped tokens.
"""
@@ -197,7 +201,13 @@ def db_flagged_items():
else:
user_filt = {raw_user.lower()} if raw_user else set()
ref_scan_id = request.args.get("ref", type=int)
items = _get_db().get_session_items(ref_scan_id=ref_scan_id)
if ref_scan_id:
# History mode — a specific past session was requested.
items = _get_db().get_session_items(ref_scan_id=ref_scan_id)
else:
# Default landing / viewer — show every item still awaiting action,
# across all scans, not just the latest session window.
items = _get_db().get_open_items()
# Normalise JSON-encoded columns the same way scan_engine does for SSE cards
import json as _json
out = []
+8 -4
View File
@@ -148,8 +148,12 @@ def smtp_test():
"</body></html>"
)
# Try Graph API first
if state.connector and state.connector.is_authenticated():
# Try Graph API first — unless the user opted to always use SMTP. Graph
# returns 202 (queued) even for recipients Exchange later silently drops
# (e.g. a Google-hosted subdomain of the O365 domain), so SMTP is the only
# reliable path for those; prefer_smtp forces it.
prefer_smtp = bool(saved.get("prefer_smtp"))
if state.connector and state.connector.is_authenticated() and not prefer_smtp:
try:
_send_email_graph(subject, body_html, recipients)
return jsonify({"ok": True, "method": "graph", "recipients": recipients})
@@ -285,8 +289,8 @@ def send_report():
"</body></html>"
)
# Try Graph API first
if state.connector and state.connector.is_authenticated():
# Try Graph API first — unless prefer_smtp is set (see smtp_test for why).
if state.connector and state.connector.is_authenticated() and not smtp_cfg.get("prefer_smtp"):
try:
_send_email_graph(subject, body_html, recipients,
attachment_bytes=xl_bytes, attachment_name=fname)
+13 -8
View File
@@ -54,7 +54,7 @@ def _maybe_send_auto_email():
"</body></html>"
)
if state.connector and state.connector.is_authenticated():
if state.connector and state.connector.is_authenticated() and not smtp_cfg.get("prefer_smtp"):
try:
_send_email_graph(subject, body_html, recipients,
attachment_bytes=xl_bytes, attachment_name=fname)
@@ -76,9 +76,13 @@ def scan_status():
acquired = state._scan_lock.acquire(blocking=False)
if acquired:
state._scan_lock.release()
g_acquired = state._google_scan_lock.acquire(blocking=False)
if g_acquired:
state._google_scan_lock.release()
return jsonify({
"running": not acquired,
"scan_id": _sse_mod._current_scan_id or None,
"running": not acquired, # M365 + file scan lock
"google_running": not g_acquired, # Google scan lock (separate)
"scan_id": _sse_mod._current_scan_id or None,
})
@@ -99,17 +103,18 @@ def scan_start():
return jsonify({"error": "scan already running"}), 409
options = request.get_json() or {}
state._scan_abort.clear()
# Profiles are loaders: a scan must never write the sidebar state back into
# a profile (it overwrote user_ids, dropped Google-only users, and hit
# profiles[0] when no profile was active). profile_id is audit-only here.
profile_id = options.pop("profile_id", None)
_save_settings({
"sources": options.get("sources", []),
"user_ids": options.get("user_ids", []),
"options": options.get("options", {}),
}, profile_id=profile_id)
def _run():
from scan_engine import run_scan
try:
run_scan(options)
_maybe_send_auto_email()
except Exception:
_log.exception("[scan] background scan thread crashed")
_audit("scan_error", "background scan thread crashed", ip="")
finally:
state._scan_lock.release()
threading.Thread(target=_run, daemon=True).start()
+23 -2
View File
@@ -118,13 +118,34 @@ def apply_update() -> dict:
"from": chk["current"], "to": chk["latest"]}
def _mark_fds_cloexec() -> None:
"""Mark every fd above stderr close-on-exec.
Werkzeug calls ``srv.socket.set_inheritable(True)`` unconditionally
(for its debug reloader), so without this the listening socket leaks
into the exec'd process: it sits on the port as a zombie listener no
one accepts from, the port probe sees the port as busy, and the new
server hops to port+1 while clients hang against the dead socket.
"""
try:
fds = [int(f) for f in os.listdir("/proc/self/fd")] # Linux
except (OSError, ValueError):
fds = list(range(3, 4096))
for fd in fds:
if fd > 2:
try:
os.set_inheritable(fd, False)
except OSError:
pass
def _restart_self() -> None:
"""Re-exec the current process so the updated code is loaded.
Keeps the same PID, so it works both under systemd and when launched
manually via start_gdpr.sh. Listening sockets are close-on-exec, so
the new process can rebind the port.
manually via start_gdpr.sh.
"""
_mark_fds_cloexec()
try:
os.execv(sys.executable, [sys.executable] + sys.argv)
except OSError:
+4 -20
View File
@@ -24,25 +24,9 @@ def get_users():
out = []
seen = set()
# Build SKU map for role classification.
# get_subscribed_skus() tries /subscribedSkus → /me/licenseDetails.
# Then always merge per-user licenseDetails on top — this ensures we
# have skuPartNumbers for every distinct SKU in the tenant, not just
# the admin's own license (which is all /me/licenseDetails returns).
try:
sku_map = state.connector.get_subscribed_skus()
except Exception:
sku_map = {}
try:
per_user = state.connector.build_sku_map_from_users(users)
if per_user:
added = len(set(per_user) - set(sku_map))
sku_map.update(per_user)
if added:
logger.info("[skus] merged %d additional SKU(s) from per-user licenseDetails", added)
except Exception:
pass
# SKU map for role classification — shared with the scan engine so the
# account list and scanned cards classify Elev/Ansat identically.
sku_map = state.connector.build_role_sku_map(users)
# Load any manual role overrides set by the admin
_role_overrides = _load_role_overrides()
@@ -176,7 +160,7 @@ def lookup_user():
if not upn:
return jsonify({"error": "upn required"}), 400
try:
data = state.connector._get(f"/users/{upn}", {"$select": "id,displayName,mail,userPrincipalName"})
data = state.connector.lookup_user(upn)
_email = data.get("mail") or data.get("userPrincipalName", upn)
return jsonify({
"id": data["id"],
+113 -9
View File
@@ -41,6 +41,95 @@ except ImportError:
from routes import state as _state
_MANUAL_PREFIX = "manual:"
_GOOGLE_PREFIX = "google:"
def _resolve_scan_users(conn, user_ids: list[dict]) -> list[dict]:
"""Turn the requested accounts into Graph user ids for the M365 scan.
- "manual:<email>" (older profile-editor saves, or saves made while M365
was not connected) is looked up to get the directory id.
- "google:<email>" is a Google-Workspace-only account: it has no M365
mailbox/drive and Graph 404s on it. Profiles and older UI selections can
contain them, so they are dropped here (the Google scan covers them).
Accounts that can't be used are reported and skipped — never widened to
"all users".
"""
resolved, seen = [], set()
skipped_google = []
for u in user_ids:
uid = u.get("id", "")
if uid.startswith(_GOOGLE_PREFIX):
skipped_google.append(uid[len(_GOOGLE_PREFIX):])
continue
if uid.startswith(_MANUAL_PREFIX):
upn = uid[len(_MANUAL_PREFIX):]
try:
data = conn.lookup_user(upn)
_email = data.get("mail") or data.get("userPrincipalName", upn)
u = {**u, "id": data["id"],
"displayName": _resolve_display_name(data.get("displayName", ""), _email, upn)}
except Exception as e:
logger.warning("[run_scan] manual account %r not found: %s", upn, e)
broadcast("scan_error", {"file": upn, "error": f"Manual account not found in M365: {e}"})
continue
if u["id"] in seen:
continue
seen.add(u["id"])
resolved.append(u)
if skipped_google:
logger.info("[run_scan] skipped %d Google-only account(s) in M365 scan", len(skipped_google))
broadcast("scan_phase", {"phase": LANG.get(
"m365_skipped_google_accounts",
"Skipped {n} Google Workspace-only account(s) — not in Microsoft 365").replace(
"{n}", str(len(skipped_google)))})
return resolved
def _enrich_bare_users(conn, user_ids: list[dict]) -> list[dict]:
"""Fill in displayName and userRole for accounts given only by id.
Profiles saved by the profile editor (and scheduled scans of them) pass
plain id strings, which run_scan normalises to displayName=<id>,
userRole="other" — so their cards showed the GUID as the name and no
Elev/Ansat badge. Look those accounts up in the directory once, using the
same role classification as GET /api/users. Accounts that can't be looked
up keep what they had.
"""
bare = {id(u) for u in user_ids
if not u.get("displayName") or u.get("displayName") == u.get("id")}
if not bare:
return user_ids
directory: dict[str, dict] = {}
sku_map: dict = {}
try:
users = conn.list_users()
directory = {u["id"]: u for u in users if u.get("id")}
sku_map = conn.build_role_sku_map(users)
except Exception as e: # e.g. delegated mode without User.Read.All
logger.info("[run_scan] directory list unavailable for name lookup: %s", e)
enriched = []
for u in user_ids:
if id(u) in bare:
uid = u.get("id", "")
d = directory.get(uid)
if d is None:
try:
d = conn.lookup_user(uid) # not in the licensed-member list
except Exception:
d = None
if d:
_email = d.get("mail") or d.get("userPrincipalName", "")
u = {**u, "displayName": _resolve_display_name(d.get("displayName", ""), _email, uid)}
if d.get("assignedLicenses") is not None and u.get("userRole", "other") == "other":
u["userRole"] = conn.classify_user_role(d.get("assignedLicenses", []), sku_map)
enriched.append(u)
return enriched
def _get_scan_abort():
return _state._scan_abort
@@ -522,12 +611,6 @@ def run_scan(options: dict):
if rem < 3600: return f"{int(rem/60)}m"
return f"{int(rem/3600)}h"
def _check_abort():
if _state._scan_abort.is_set():
broadcast("scan_cancelled", {"completed": completed})
return True
return False
def _broadcast_card(item_meta: dict, cprs: list, pii_counts: dict | None = None):
card = {
"id": item_meta.get("id", ""),
@@ -645,6 +728,8 @@ def run_scan(options: dict):
me_info.get("displayName", ""),
me_info.get("mail") or me_info.get("userPrincipalName", "me"))}]
else:
user_ids = _resolve_scan_users(conn, user_ids)
user_ids = _enrich_bare_users(conn, user_ids)
sample = user_ids[0] if user_ids else None
logger.info("[run_scan] user_ids: %d entries, type=%s, sample=%s",
len(user_ids), type(user_ids).__name__, sample)
@@ -1057,6 +1142,26 @@ def run_scan(options: dict):
if scanned_ids:
work_items = [(k, m, f) for k, m, f in work_items if m.get("id", "") not in scanned_ids]
def _finish_aborted(done: int) -> None:
"""Stop requested: checkpoint, finalise the DB scan, announce it."""
# Save checkpoint so scan can be resumed later
_save_checkpoint(ck_key, scanned_ids, _state.flagged_items, _state.scan_meta)
# Finalise the DB scan record so items found before the stop stay
# visible — the early return otherwise skips finish_scan below,
# stranding them (invisible to get_session_items / get_open_items).
if _db and _db_scan_id:
try:
_db.finish_scan(_db_scan_id, done)
except Exception as _e:
logger.error("[db] finish_scan (aborted) failed: %s", _e)
broadcast("scan_cancelled", {"completed": done - resumed_count})
# Stop pressed during collection: the per-user loops just break out, so
# stop here — announcing scan_start/"Scanning…" made Stop look like a restart.
if _state._scan_abort.is_set():
_finish_aborted(resumed_count)
return
total = len(work_items)
broadcast("scan_start", {
"total": total + resumed_count,
@@ -1075,9 +1180,8 @@ def run_scan(options: dict):
_items_since_save = 0
idx = -1
while _work_q:
if _check_abort():
# Save checkpoint so scan can be resumed later
_save_checkpoint(ck_key, scanned_ids, _state.flagged_items, _state.scan_meta)
if _state._scan_abort.is_set():
_finish_aborted(resumed_count + idx + 1)
return
idx += 1
kind, meta, _ = _work_q.popleft() # releases this item from the deque immediately
+27 -12
View File
@@ -351,6 +351,9 @@ class ScanScheduler:
)
_fs_extra = {k: _profile_opts[k] for k in _FS_OPT_KEYS if k in _profile_opts}
for fs in options.get("file_sources", []):
# Stop pressed during an earlier stage — don't start more stages
if _m._scan_abort.is_set():
break
# Resolve string IDs to full source dicts
if isinstance(fs, str):
fs = _all_file_sources.get(fs, {"path": fs, "label": fs})
@@ -374,7 +377,12 @@ class ScanScheduler:
# Legacy profiles store everything in sources[]
google_sources = [s for s in options.get("sources", [])
if s in ("gmail", "gdrive")]
if google_sources and state.google_connector:
# POST /api/scan/stop sets both abort events. _scan_abort was cleared
# when this job started, so if it is set now the user pressed Stop
# during an earlier stage — the Google stage must not start, and its
# abort event must not be cleared (that turned Stop into a restart).
stopped = _m._scan_abort.is_set()
if google_sources and state.google_connector and not stopped:
from routes.google_scan import (
_run_google_scan as _rgs,
_scan_lock as _gsl,
@@ -383,12 +391,15 @@ class ScanScheduler:
if _gsl.acquire(blocking=False):
try:
_gsa.clear()
logger.info("[scheduler] Starting Google scan — sources=%s", google_sources)
_rgs({
"sources": google_sources,
"user_emails": [], # empty → scan all workspace users
"options": options.get("options", {}),
})
# Stop sets _scan_abort first, so re-check after clearing:
# a Stop landing in between must still win.
if not _m._scan_abort.is_set():
logger.info("[scheduler] Starting Google scan — sources=%s", google_sources)
_rgs({
"sources": google_sources,
"user_emails": [], # empty → scan all workspace users
"options": options.get("options", {}),
})
except Exception as _ge:
import traceback as _tb3
logger.error("[scheduler] Google scan error: %s\n%s", _ge, _tb3.format_exc())
@@ -399,18 +410,21 @@ class ScanScheduler:
run["flagged"] = len(_m.flagged_items)
run["scanned"] = _m.scan_meta.get("total_scanned", 0)
run["status"] = "completed"
logger.info("[scheduler] Job '%s' completed — %d flagged, %d scanned",
job_cfg.get("name", ""), run["flagged"], run["scanned"])
stopped = _m._scan_abort.is_set()
run["status"] = "cancelled" if stopped else "completed"
logger.info("[scheduler] Job '%s' %s — %d flagged, %d scanned",
job_cfg.get("name", ""), run["status"], run["flagged"], run["scanned"])
if job_cfg.get("auto_email") and state.flagged_items:
# A stopped job's results are partial — don't email them or apply
# retention based on them.
if not stopped and job_cfg.get("auto_email") and state.flagged_items:
try:
self._send_email_report(job_cfg)
run["emailed"] = 1
except Exception as e:
run["error"] = f"Scan OK, email failed: {e}"
if job_cfg.get("auto_retention") and job_cfg.get("retention_years"):
if not stopped and job_cfg.get("auto_retention") and job_cfg.get("retention_years"):
try:
self._run_retention(job_cfg)
except Exception as e:
@@ -420,6 +434,7 @@ class ScanScheduler:
_m.broadcast("scheduler_done", {
"flagged": run["flagged"], "scanned": run["scanned"],
"emailed": run["emailed"], "job_name": job_cfg.get("name", ""),
"cancelled": stopped,
})
except Exception as e:
+26 -2
View File
@@ -19,6 +19,29 @@ logger = logging.getLogger(__name__)
_sse_queues: list = []
_sse_buffer: deque = deque(maxlen=500)
_current_scan_id: str = ""
# Scans that have ended (done / stopped). Their events stay in the buffer but
# are never replayed: replaying a stopped scan re-showed the Stop button and
# progress bar on every reconnect, so Stop looked like a restart.
_finished_scan_ids: deque = deque(maxlen=20)
# Events that end the scan identified by _current_scan_id.
# scan_done: M365 finished; scan_cancelled: M365 stopped; file_scan_done:
# file scan finished or stopped. (Google scans don't set _current_scan_id.)
_TERMINAL_EVENTS = frozenset({"scan_done", "scan_cancelled", "file_scan_done"})
def replay_messages() -> list[str]:
"""Buffered messages to replay to a newly connected browser.
While a scan runs, only that scan's events. Otherwise everything except
events of scans that have already ended (done or stopped).
"""
buf = list(_sse_buffer)
if _current_scan_id:
tag = f'"scan_id": "{_current_scan_id}"'
return [m for m in buf if tag in m]
finished = [f'"scan_id": "{sid}"' for sid in _finished_scan_ids]
return [m for m in buf if not any(tag in m for tag in finished)]
def broadcast(event: str, data: dict):
global _current_scan_id
@@ -31,8 +54,9 @@ def broadcast(event: str, data: dict):
q.put_nowait(msg)
except queue.Full:
pass
# Clear scan_id after scan_done so replay knows the scan is finished
if event == "scan_done" and _current_scan_id:
# Clear scan_id when the scan ends so replay knows it is finished
if event in _TERMINAL_EVENTS and _current_scan_id:
_finished_scan_ids.append(_current_scan_id)
_current_scan_id = ""
# When no browser is watching (e.g. scheduled scan), log key events
if not _sse_queues:
+21 -3
View File
@@ -34,19 +34,30 @@ Never revert to `!!window._googleConnected` / `_fileSources.length > 0` — thos
- **Do not close `S.es` in `scan_done` if other scans are still running** — M365 (`scan_done`), Google (`google_scan_done`), and File (`file_scan_done`) each emit their own done event. Close `S.es` only when all concurrent scans have finished: `scan_done` checks `!S._googleScanRunning && !S._fileScanRunning`; `google_scan_done` checks `!S._m365ScanRunning && !S._fileScanRunning`; `file_scan_done` checks `!S._m365ScanRunning && !S._googleScanRunning`.
- **Scheduled scans** — `S._userStartedScan` is false for scheduler-triggered runs, so SSE is never closed and future scheduler events continue to arrive.
- **Two separate abort events** — `state._scan_abort` (M365 + file) and `state._google_scan_abort` (Google). `POST /api/scan/stop` sets **both**. `_check_abort()` inside `_run_google_scan` must use the module-level `_scan_abort` alias (`= state._google_scan_abort`), not `gdpr_scanner._scan_abort`.
- **Stop must never look like a restart** — four rules, each fixing a way Stop used to appear to restart the scan:
- `sse.replay_messages()` never replays a scan that has ended. `scan_done`, `scan_cancelled` and `file_scan_done` move `_current_scan_id` into `_finished_scan_ids`. Do not go back to clearing the id only on `scan_done`: stopped M365 scans and all file scans stayed "current", and every reconnect (page load, watchdog) replayed them, bringing back the Stop button and progress bar.
- The `scan_cancelled` handler clears `S._m365ScanRunning` and, like `scan_done`, keeps the Stop button and SSE only while a Google or file scan is still running. `scheduler_done` and `scheduler_error` clear all three running flags.
- The Stop button goes through `_setStopBtnDisplay(display, fresh)`; do not set `stopBtn.style.display` directly. `stopScan()` switches it to a disabled "Stopping…" state. Hiding it, or passing `fresh=true` (from `startScan()` and `scheduler_started`), resets it.
- Server side: `run_scan` checks for Stop straight after collection, before `scan_start`. The scheduler checks `_scan_abort` between stages and never clears the Google abort flag after a Stop. A stopped job is recorded as `cancelled`, with no email or retention.
- **`_check_abort()` emits `google_scan_done`, not `scan_cancelled`** — `scan_cancelled` unconditionally closes the SSE; `google_scan_done` checks whether other scans are still running before closing.
- **`scan_phase` replay sets running flags — handled by `sse_replay_done`** — the `scan_phase` handler sets running flags to `true` whenever all flags are `false` and a source keyword is found in the phase text. On page refresh this fires during SSE replay of a completed scan, temporarily making the scan appear running. The `sse_replay_done` handler retries `loadHistorySession(null)` if no scan is running and `S._historyRefScanId` is still `null` after replay. Do not remove either the flag-setting logic or the retry.
- **Google Drive uses a lazy generator, not `list()`** — `iter_drive_files()` iterated directly so `_check_abort()` fires between items. Wrapping in `list()` blocks the thread for the entire enumeration.
## Scan history browser — history.js + results.js
- **`S._historyRefScanId`** — `null` = live/SSE mode; positive int = viewing a past session. Set by `loadHistorySession()`; cleared by `exitHistoryMode()`.
- **Auto-load on page load** — `results.js` calls `window.loadHistorySession?.(null)` once when the SSE watchdog confirms `!status.running`. `_initialStatusChecked` guard ensures this fires at most once per page load. The `sse_replay_done` handler in `scan.js` retries if `loadHistorySession` bailed due to stale running flags set during replay.
- **`S._historyRefScanId`** — `null` = live/SSE mode **or** the default open-items view; positive int = viewing a past session. Set by `loadHistorySession()`; cleared by `exitHistoryMode()`.
- **`loadHistorySession(null)` → `loadOpenItems()`** — passing `null` no longer resolves to the latest session. It now loads **all open (unactioned) items across every scan** via `GET /api/db/flagged` (no `ref`), leaves `_historyRefScanId` null, and shows no history banner. The "Open items" banner button (`onclick="loadHistorySession(null)"`, key `history_btn_latest`) therefore returns to this open-items view. Specific sessions are still loaded with a positive `ref`, which keeps the re-scan resolved-diff. Do not revert `null` to "resolve latest ref" — that reintroduces the "only the last scan is shown" complaint.
- **Auto-load on page load** — `_sseWatchdog()` in `results.js` calls `window.loadHistorySession?.(null)` whenever `/api/scan/status` reports neither `running` (M365 + file lock) nor `google_running` (Google lock) **and** nothing is shown yet (`!S._historyRefScanId && !S.flaggedData.length`). This is **not one-shot** — it retries on every 4s poll until a session is restored, because (a) the replay buffer is empty after a server restart so `sse_replay_done` never fires, and (b) a completed scan's replayed `scan_phase` can leave a running flag set that would otherwise block the load forever. Because both locks are confirmed free, the watchdog clears the stale `_m365/_google/_fileScanRunning` flags before calling. Do not revert to a one-shot `_initialStatusChecked` gate — that reintroduces the "blank grid after refresh/restart" bug. `/api/scan/status` **must** report `google_running` separately; `running` alone misses live Google scans. The `sse_replay_done` handler in `scan.js` still retries for the non-empty-buffer (no-restart) case.
- **History banner** (`#historyBanner`) — shown when `S._historyRefScanId` is set. Do not hide/show from outside `history.js`.
- **Session picker** (`#historyDropdown`) — rendered inside `[data-history-wrap]` so the outside-click handler works correctly. Do not move the picker outside this wrapper.
- **Cache invalidation** — `invalidateHistoryCache()` clears `_sessions` and `_latestRefScanId`. All three `*_done` SSE handlers call `window.invalidateHistoryCache?.()`.
- **Re-scan diff** — items present in the previous session but absent from the current one are tagged `_resolved: true`, rendered with `.card-resolved` and a green ✓ badge, and NOT added to `S.flaggedData` (grid-only, cannot be bulk-selected or exported).
- **Mode transitions** — `startScan()` calls `window.exitHistoryMode?.()` before clearing the grid.
- **`renderGrid(files)` hides the landing cards** — whenever `files.length > 0` it hides `#emptyState` and `#lastScanSummary` and shows `#grid`. This is centralised here because the live `scan_file_flagged` handler (`scan.js`) shows the grid but does NOT clear those panels, so results would render *underneath* a still-visible landing/last-scan card until a manual refresh. Do not move this hiding back into individual callers — every render path (live SSE, `loadOpenItems`, history, filters) must clear the landing. The empty case (`files.length === 0`) is left untouched so callers still control the empty/landing state.
## Card user/group badge — results.js
- **`_accountPill(f)`** builds the account/role pill for both card layouts (list + grid). The **group badge is driven by `f.user_role`** (`student`/`staff`) alone, so it renders even with no display name — items from scans saved before `account_name` was persisted (DB migration 11) have only `user_role` + `account_id`. The user label resolves best-effort: `f.account_name` → `S._allUsers` match (by `id` or `email`) → email-style `account_id` → omit. An `account_name` equal to `account_id` (a GUID stored by older scheduled scans of editor-saved profiles) counts as missing, and a role other than `student`/`staff` is filled from the same `S._allUsers` match. Do not re-nest the role badge inside an `account_name` check (the old bug) — that hides the group badge for legacy items. Both layouts call `_accountPill(f)`; keep them sharing the one helper.
## CPR cross-referencing — results.js
@@ -74,7 +85,14 @@ Never revert to `!!window._googleConnected` / `_fileSources.length > 0` — thos
- **`scheduler.js` strings must use `t()`** — frequency labels, "Next", "Running...", "Disabled", empty-job text, and empty-history text all have translation keys. Do not hard-code English strings in `schedLoad()` or `schedRenderJobs()`.
- **Scheduler UI — `schedToggleReportOnly()`** — dims the Profile row, shows/hides `#schedReportOnlyHint`, and forces `#schedAutoEmail` checked. Called from the checkbox `onchange` handler and at the start of `schedAddJob()` / `schedEditJob()`.
- **Profile editor accounts** — default to unchecked. Only explicitly saved `user_ids` are checked.
- **Profile editor accounts** — default to unchecked. Only explicitly saved `user_ids` are checked. Saved IDs not in `S._allUsers` (`manual:<email>`, users not loaded yet, users gone from the directory) are rendered by `_unresolvedAccountRows()` as checked rows so the next save keeps them — do not go back to building rows from `S._allUsers` alone (that silently dropped them on save). `user_ids: "all"` is kept as `"all"` when saved before users have loaded. The editor saves each account as `{id, displayName, userRole}` (plus `email`/`manual` for manual accounts), the same shape as 💾 — not a bare id string: scheduled scans pass profile accounts straight to `run_scan`, and bare ids produced GUID-named cards with no role badge. Server side, `scan_engine._enrich_bare_users()` names any bare ids that remain (older profiles) using `conn.build_role_sku_map()`, the same role classification as `GET /api/users`.
- **Manual accounts must be scannable** — they aren't in the `/api/users` directory list, so:
- `getSelectedUsers()` treats a missing `platform` as `'m365'`. Sidebar `addUserManually()` users historically had no platform and were silently excluded from every scan. Do not go back to a strict `u.platform === 'm365'` check.
- The profile editor's "+ Tilføj konto" looks the email up via `/api/users/lookup` and stores `{id: <directory id>, displayName, email, manual: true}`. It falls back to `"manual:<email>"` only when the lookup fails. Checkboxes with `data-manual` save as objects; everything else saves as plain id strings.
- `_applyProfile()` → `_addProfileManualUsers()` pushes manual accounts into `S._allUsers` (`manual: true`, `platform: 'm365'`). Legacy `manual:<email>` entries are looked up and the pending ID is rewritten to the resolved one. Without this the account can never be ticked in the sidebar, so it's never scanned.
- `_userListFinal()` keys off `S._m365UsersLoaded`, not `S._allUsers.length`, because manual users can be pushed before the directory loads.
- Server side, `scan_engine._resolve_scan_users()` resolves any remaining `manual:<email>` (e.g. from scheduled scans) via `conn.lookup_user()`, and drops `google:<email>` (Google-only accounts, which Graph 404s on). It skips and reports failures and never widens to "all users".
- **Pending profile users wait for the Google merge** — `_applyPendingProfileUsers()` keeps `window._pendingProfileUserIds` until every ID is matched or `_userListFinal()` (users loaded, and Google either not connected or `S._googleUsersMerged`). `_mergeGoogleUsers()` re-applies in its `finally`. A saved `google:<email>` also matches an M365 user merged to `platform: 'both'` via `u.googleEmail`. Do not clear the pending IDs unconditionally — Google-only users would stay unticked.
- **Date presets** — stored as `years * 365` (integer days). Do not use `* 365.25`.
- **`copyTokenLink` is async** — called from `onclick` as fire-and-forget. Do not make it synchronous.
- **Escape scan-derived strings with `esc()`** — `results.js` defines `esc()` (escapes `& < > " '`). Every value that originates from scanned content (`f.name`, `f.account_name`, `f.folder`, `f.source`, `f.modified`, `label`, image `alt`, and the same fields on `item`/related rows) must pass through `esc()` before going into `innerHTML` or a `title=`/`alt=` attribute. These are attacker-influenceable (e.g. a file named with markup), so an unescaped interpolation is stored XSS — including in shared read-only viewer sessions. Numeric counts (`cpr_count`, `size_kb`) don't need it. When embedding an object in an `onclick` payload, also `.replace(/"/g,'&quot;')` the `JSON.stringify(...)`.
+39 -9
View File
@@ -38,20 +38,50 @@ function invalidateHistoryCache() {
// ── Load a session into the results grid ──────────────────────────────────────
async function loadHistorySession(refScanId) {
// refScanId: null → latest session, positive int → specific session
let resolvedRef = refScanId;
if (resolvedRef === null) {
const sessions = _sessions !== null ? _sessions : await _fetchSessions();
// Bail if a scan started while we were fetching sessions
// Default landing view: every flagged item still awaiting action, across all
// scans (not just the latest session). Leaves S._historyRefScanId null (live
// mode) and shows no history banner — this is "now", not a past session.
async function loadOpenItems() {
// Bail if a scan is running — live SSE owns the grid then.
if (S._m365ScanRunning || S._googleScanRunning || S._fileScanRunning) return;
try {
const r = await fetch('/api/db/flagged');
const items = await r.json();
if (S._m365ScanRunning || S._googleScanRunning || S._fileScanRunning) return;
if (!sessions.length) {
// No scans in DB — nothing to show
closeHistoryPicker();
if (!Array.isArray(items) || items.length === 0) {
S._historyRefScanId = null;
_setHistoryBanner(false);
window.loadLastScanSummary?.();
return;
}
resolvedRef = sessions[0].ref_scan_id;
S._historyRefScanId = null;
S.flaggedData = items;
S.filteredData = [];
const grid = document.getElementById('grid');
const emptyState = document.getElementById('emptyState');
const lastScan = document.getElementById('lastScanSummary');
if (emptyState) emptyState.style.display = 'none';
if (lastScan) lastScan.style.display = 'none';
if (grid) { grid.innerHTML = ''; grid.style.display = 'grid'; }
window.renderGrid(items);
try { window.markOverdueCards(); } catch(_) {}
try { window.loadTrend(); } catch(_) {}
_setHistoryBanner(false);
} catch(e) {
console.error('[history] failed to load open items:', e);
}
}
async function loadHistorySession(refScanId) {
// refScanId: null → all open (unreviewed) items across every scan,
// positive int → a specific past session
if (refScanId === null) return loadOpenItems();
const resolvedRef = refScanId;
try {
const r = await fetch('/api/db/flagged?ref=' + resolvedRef);
+137 -18
View File
@@ -215,6 +215,7 @@ function _applyProfile(profile) {
}
} else if (Array.isArray(profile.user_ids) && profile.user_ids.length) {
window._pendingProfileUserIds = profile.user_ids.map(u => u.id || u);
_addProfileManualUsers(profile.user_ids);
_applyPendingProfileUsers();
} else if (Array.isArray(profile.user_ids) && profile.user_ids.length === 0) {
// Explicitly empty list — deselect everyone so previous sidebar state doesn't persist
@@ -225,13 +226,74 @@ function _applyProfile(profile) {
log(t('m365_profile_applied', 'Profile loaded') + ': ' + profile.name);
}
// Google-only users (id "google:<email>") arrive later than M365 users, via the
// un-awaited _mergeGoogleUsers(). Keep the pending IDs until every one is
// matched or the user list is final, else those users stay unticked.
function _userListFinal() {
return S._m365UsersLoaded && (!window._googleConnected || S._googleUsersMerged);
}
function _applyPendingProfileUsers() {
const ids = window._pendingProfileUserIds;
if (!ids || !ids.length || !S._allUsers.length) return;
// Select only the users listed in the profile
S._allUsers.forEach(u => { u.selected = ids.includes(u.id); });
const wanted = new Set(ids);
// A saved google:<email> may since have merged onto an M365 user (platform 'both')
// Must return a real boolean: the sidebar and getSelectedUsers() treat any
// selected value other than `false` as ticked, so an `undefined` here (users
// without googleEmail) ticked — and scanned — every M365-only/GWS-only user.
const matches = u => wanted.has(u.id) || !!(u.googleEmail && wanted.has('google:' + u.googleEmail));
S._allUsers.forEach(u => { u.selected = matches(u); });
renderAccountList();
window._pendingProfileUserIds = null;
const found = S._allUsers.filter(matches).length;
if (found >= wanted.size || _userListFinal()) window._pendingProfileUserIds = null;
}
function _isManualEntry(u) {
return typeof u === 'object' ? !!u.manual : String(u).startsWith('manual:');
}
// Manual accounts aren't in the /api/users directory list, so they must be added
// to S._allUsers (as sidebar manual users) or they can never be selected/scanned.
// Legacy "manual:<email>" entries are looked up to get their directory id.
function _addProfileManualUsers(entries) {
entries.filter(_isManualEntry).forEach(async function(u) {
if (typeof u === 'object' && !String(u.id).startsWith('manual:')) {
if (!S._allUsers.some(x => x.id === u.id)) {
S._allUsers.push({id: u.id, displayName: u.displayName || u.email || u.id,
email: u.email || '', userRole: u.userRole || 'other',
platform: 'm365', manual: true, selected: false});
}
return; // selection is handled by _applyPendingProfileUsers()
}
const legacyId = u.id || u;
try {
const email = legacyId.slice('manual:'.length);
const d = await _lookupM365User(email);
if (!d) {
// Not connected / not found — scheduled scans retry the lookup at scan time
log(t('m365_profile_manual_not_found', 'Manual account not found in M365') + ': ' + email, 'err');
return;
}
let user = S._allUsers.find(x => x.id === d.id);
if (!user) {
user = {...d, platform: 'm365', manual: true};
S._allUsers.push(user);
}
user.selected = true;
// Pending apply may re-run later — make it select the resolved id, not the legacy one
const pend = window._pendingProfileUserIds;
if (pend) window._pendingProfileUserIds = pend.map(x => x === legacyId ? d.id : x);
renderAccountList();
} catch(e) { /* leave for the scan-time fallback */ }
});
}
// Returns {id, displayName, email} or null when M365 isn't connected / the user doesn't exist.
async function _lookupM365User(upn) {
const r = await fetch('/api/users/lookup?upn=' + encodeURIComponent(upn));
if (!r.ok) return null;
const d = await r.json();
return d && d.id ? d : null;
}
async function saveCurrentAsProfile() {
@@ -365,19 +427,26 @@ function _openEditorForProfile(profile) {
const srcCheck = (id) => allSources.includes(id) ? 'checked' : '';
// Build account list from S._allUsers
const savedIds = new Set((profile.user_ids || []).map(u => u.id || u));
// If no saved IDs match current users, treat as all-selected (new profile or users changed)
const anyMatch = savedIds.size > 0 && S._allUsers.some(u => savedIds.has(u.id));
const savedAll = profile.user_ids === 'all';
const savedList = Array.isArray(profile.user_ids) ? profile.user_ids : [];
const savedIds = new Set(savedList.map(u => u.id || u));
const isSaved = u => savedAll || savedIds.has(u.id)
|| !!(u.googleEmail && savedIds.has('google:' + u.googleEmail));
// Remember "all" so saving before the user list has loaded doesn't turn it into []
window._pmgmtSavedAll = savedAll;
const accountRows = S._allUsers.map(u => {
// Only check if the user was explicitly saved — default to unchecked like the main window
const checked = anyMatch && savedIds.has(u.id) ? 'checked' : '';
const checked = isSaved(u) ? 'checked' : '';
// data-name lets the save keep the display name with the id (see _pmgmtSaveFullEdit)
const manualAttrs = ` data-name="${_esc(u.displayName || '')}"` + (u.manual
? ` data-manual="1" data-email="${_esc(u.email || '')}"` : '');
const platBadge = u.platform === 'both' ? '<span style="font-size:9px;padding:1px 5px;border-radius:10px;background:linear-gradient(90deg,#E6F1FB 50%,#EAF3DE 50%);color:#1a4a1a;font-weight:500;border:0.5px solid #b5d4b5">M365+GWS</span>'
: (u.platform || 'm365') === 'google' ? '<span style="font-size:9px;padding:1px 5px;border-radius:10px;background:#EAF3DE;color:#3B6D11;font-weight:500">GWS</span>'
: '<span style="font-size:9px;padding:1px 5px;border-radius:10px;background:#E6F1FB;color:#185FA5;font-weight:500">M365</span>';
const roleBadge = u.userRole === 'student' ? t('role_student','Elev') : u.userRole === 'staff' ? t('role_staff','Ansat') : t('role_other','Anden');
const roleOverrideStyle = u.roleOverride ? 'color:var(--color-text-info);outline:1px solid var(--color-border-info);' : '';
return `<label class="pmgmt-acct-row" data-uid="${_esc(u.id)}" data-role="${_esc(u.userRole || 'other')}"><input type="checkbox" ${checked} data-uid="${_esc(u.id)}"><span style="flex:1;color:var(--color-text-primary);overflow:hidden;text-overflow:ellipsis;white-space:nowrap">${_esc(u.displayName)}</span>${platBadge}<button type="button" class="pmgmt-role-badge" data-uid="${_esc(u.id)}" onclick="_pmgmtCycleRole(this.getAttribute('data-uid'),event)" style="font-size:9px;padding:1px 5px;border-radius:10px;background:#D3D1C7;border:none;cursor:pointer;${roleOverrideStyle}">${roleBadge}</button></label>`;
}).join('');
return `<label class="pmgmt-acct-row" data-uid="${_esc(u.id)}" data-role="${_esc(u.userRole || 'other')}"><input type="checkbox" ${checked} data-uid="${_esc(u.id)}"${manualAttrs}><span style="flex:1;color:var(--color-text-primary);overflow:hidden;text-overflow:ellipsis;white-space:nowrap">${_esc(u.displayName)}</span>${platBadge}<button type="button" class="pmgmt-role-badge" data-uid="${_esc(u.id)}" onclick="_pmgmtCycleRole(this.getAttribute('data-uid'),event)" style="font-size:9px;padding:1px 5px;border-radius:10px;background:#D3D1C7;border:none;cursor:pointer;${roleOverrideStyle}">${roleBadge}</button></label>`;
}).join('') + _unresolvedAccountRows(savedList);
body.innerHTML = `
<div>
@@ -464,6 +533,31 @@ function _openEditorForProfile(profile) {
_renderEditorSources((profile.sources || []).concat(profile.google_sources || []).concat(profile.file_sources || []));
}
// Saved accounts that aren't in S._allUsers — manually added ("manual:<email>"),
// not loaded yet, or gone from the directory. Rendered as checked rows so the
// next save keeps them instead of silently dropping them.
function _unresolvedAccountRows(savedList) {
const known = new Set();
S._allUsers.forEach(u => {
known.add(u.id);
if (u.googleEmail) known.add('google:' + u.googleEmail);
});
const seen = new Set();
return savedList.map(u => {
const id = u.id || u;
if (!id || known.has(id) || seen.has(id)) return '';
seen.add(id);
const isManual = _isManualEntry(u);
const label = (typeof u === 'object' && u.displayName) || id.replace(/^(manual|google):/, '');
const badge = isManual ? t('m365_profile_acct_manual', 'Manual')
: t('m365_profile_acct_unresolved', 'Not loaded');
const savedName = typeof u === 'object' && u.displayName && u.displayName !== id ? u.displayName : '';
const manualAttrs = (savedName ? ` data-name="${_esc(savedName)}"` : '') + (isManual && typeof u === 'object'
? ` data-manual="1" data-email="${_esc(u.email || '')}"` : '');
return `<label class="pmgmt-acct-row" data-uid="${_esc(id)}" data-role="${_esc((typeof u === 'object' && u.userRole) || 'other')}"><input type="checkbox" checked data-uid="${_esc(id)}"${manualAttrs}><span style="flex:1;color:var(--text);overflow:hidden;text-overflow:ellipsis;white-space:nowrap" title="${_esc(id)}">${_esc(label)}</span><span style="font-size:9px;padding:1px 5px;border-radius:10px;background:#D3D1C7;color:#444441">${_esc(badge)}</span></label>`;
}).join('');
}
function _peSetDate(val) {
if (!val) return;
const ms = new Date() - new Date(val);
@@ -592,16 +686,24 @@ function _pmgmtRoleFilter(role) {
_pmgmtFilterAccounts(document.getElementById('pmgmtAcctSearch')?.value || '');
}
function _pmgmtAddManual() {
const email = prompt('E-mail adresse:');
if (!email || !email.trim()) return;
async function _pmgmtAddManual() {
const email = (prompt('E-mail adresse:') || '').trim();
if (!email) return;
const list = document.getElementById('pmgmtAcctList');
if (!list) return;
const id = 'manual:' + email.trim().toLowerCase();
if (list.querySelector(`input[data-uid="${id}"]`)) return; // already exists
// Resolve to the directory id so the scan can use it. If M365 isn't reachable
// right now, fall back to "manual:<email>" — the scan engine resolves that later.
let d = null;
try { d = await _lookupM365User(email); } catch(e) { /* offline — use fallback */ }
const id = d ? d.id : 'manual:' + email.toLowerCase();
const label = d ? (d.displayName || email) : email;
const existing = [...list.querySelectorAll('input[data-uid]')].find(cb => cb.dataset.uid === id);
if (existing) { existing.checked = true; return; } // already listed — just tick it
const lbl = document.createElement('label');
lbl.className = 'pmgmt-acct-row';
lbl.innerHTML = `<input type="checkbox" checked data-uid="${_esc(id)}"><span style="flex:1;color:var(--text);overflow:hidden;text-overflow:ellipsis;white-space:nowrap">${_esc(email.trim())}</span><span style="font-size:9px;padding:1px 5px;border-radius:10px;background:#D3D1C7;color:#444441">Manuel</span>`;
lbl.dataset.uid = id;
lbl.dataset.role = 'other';
lbl.innerHTML = `<input type="checkbox" checked data-uid="${_esc(id)}" data-manual="1" data-name="${_esc(label)}" data-email="${_esc(d ? (d.email || email) : email)}"><span style="flex:1;color:var(--text);overflow:hidden;text-overflow:ellipsis;white-space:nowrap" title="${_esc(email)}">${_esc(label)}</span><span style="font-size:9px;padding:1px 5px;border-radius:10px;background:#D3D1C7;color:#444441">${_esc(t('m365_profile_acct_manual', 'Manual'))}</span>`;
list.appendChild(lbl);
}
@@ -635,9 +737,22 @@ async function _pmgmtSaveFullEdit() {
const effectiveGoogleSources = googleRendered ? googleSources : (profile.google_sources || []);
const effectiveFileSources = fileRendered ? fileSources : (profile.file_sources || []);
const allSources = m365Sources.concat(effectiveGoogleSources).concat(effectiveFileSources);
const user_ids = Array.from(document.querySelectorAll('#pmgmtAcctList input[type=checkbox]:checked'))
.map(cb => cb.dataset.uid)
.filter(Boolean);
let user_ids = Array.from(document.querySelectorAll('#pmgmtAcctList input[type=checkbox]:checked'))
.filter(cb => cb.dataset.uid)
.map(cb => {
// Save name + role with the id (same shape as 💾 / getSelectedUsers()):
// scheduled scans pass these straight to the scan, and a bare id became
// the card's "name" with no Elev/Ansat badge.
const role = cb.closest('label')?.dataset.role || 'other';
if (cb.dataset.manual) {
return {id: cb.dataset.uid, displayName: cb.dataset.name || '', email: cb.dataset.email || '',
userRole: role, manual: true};
}
return cb.dataset.name ? {id: cb.dataset.uid, displayName: cb.dataset.name, userRole: role}
: cb.dataset.uid;
});
// "all" profile opened before users loaded: there were no rows to tick, so keep "all"
if (window._pmgmtSavedAll && !S._allUsers.length && !user_ids.length) user_ids = 'all';
const updated = {
...profile,
name,
@@ -769,6 +884,10 @@ window._esc = _esc;
window._pmgmtUse = _pmgmtUse;
window._pmgmtOpenEditor = _pmgmtOpenEditor;
window._openEditorForProfile = _openEditorForProfile;
window._unresolvedAccountRows = _unresolvedAccountRows;
window._addProfileManualUsers = _addProfileManualUsers;
window._lookupM365User = _lookupM365User;
window._userListFinal = _userListFinal;
window._peSetDate = _peSetDate;
window._peSetYear = _peSetYear;
window._renderEditorSources = _renderEditorSources;
+69 -13
View File
@@ -25,6 +25,41 @@ const SOURCE_BADGES = {
smb: ['🌐', 'badge-smb', 'Network'],
};
// Build the user/group pill for a card. The group (role) badge is driven by
// user_role alone so it shows even when no display name is available — e.g.
// items from earlier scans saved before account_name was persisted. For those
// the user label is resolved best-effort from the loaded user list (by id or
// email), falling back to an email-style account_id. Returns '' when there is
// neither a label nor a role to show.
function _accountPill(f) {
let role = f.user_role;
let label = f.account_name || '';
// Scheduled scans of editor-saved profiles stored the account id (a GUID) as
// the name, with role "other" — treat that like a missing name.
if (label && f.account_id && label === String(f.account_id)) label = '';
const needName = !label && f.account_id;
const needRole = role !== 'student' && role !== 'staff';
if (f.account_id && (needName || needRole)) {
const aid = String(f.account_id);
const u = (S._allUsers || []).find(function(u) {
return u.id === f.account_id ||
(u.email && u.email.toLowerCase() === aid.toLowerCase());
});
if (u) {
if (needName) label = u.displayName || '';
if (needRole) role = u.userRole;
} else if (needName && aid.includes('@')) {
label = aid; // an email is already human-readable
}
}
const roleBadge =
role === 'student' ? '<span class="role-badge">' + t('role_student', 'Elev') + '</span>' :
role === 'staff' ? '<span class="role-badge">' + t('role_staff', 'Ansat') + '</span>' : '';
if (!label && !roleBadge) return '';
const title = label || role || '';
return '<span class="account-pill" title="' + esc(title) + '">' + roleBadge + (label ? esc(label) : '') + '</span>';
}
function appendCard(f) {
const search = document.getElementById('filterSearch').value.trim().toLowerCase();
const srcVal = document.getElementById('filterSource').value;
@@ -61,6 +96,7 @@ function appendCard(f) {
(f.source_type === 'smb' || f.source_type === 'sftp') ? _redactExts.has(_fileExt) : false
);
const redactBtn = _redactable ? `<button class="card-redact-btn" title="${t('redact_btn','Redact CPR')}" onclick="event.stopPropagation();redactItem(${JSON.stringify(f).replace(/"/g,'&quot;')},this.closest('.card'))">✏</button>` : '';
const acctPill = _accountPill(f);
if (S.isListView) {
card.innerHTML = `
@@ -68,7 +104,7 @@ function appendCard(f) {
<div class="card-info list-info">
<div class="card-name" title="${esc(f.name)}">${esc(f.name)}</div>
<div class="card-meta">${f.size_kb} KB · ${esc(f.modified || '')}${f.folder ? ' · 📂 ' + esc(f.folder) : ''}</div>
<div class="card-source"><span class="source-badge ${badgeCls}">${esc(label)}</span> ${esc(f.source || '')}${f.account_name ? ' · <span class="account-pill" title="' + esc(f.account_name) + '">' + (f.user_role === 'student' ? '<span class="role-badge">' + t('role_student','Elev') + '</span>' : f.user_role === 'staff' ? '<span class="role-badge">' + t('role_staff','Ansat') + '</span>' : '') + esc(f.account_name) + '</span>' : ''}${f.transfer_risk === 'external-recipient' ? ' <span class="role-pill" style="background:#7B2D00;color:#FFD0B0">⚠ Ext.</span>' : f.transfer_risk ? ' <span class="role-pill" style="background:#003D7B;color:#B0D4FF">🔗</span>' : ''}</div>
<div class="card-source"><span class="source-badge ${badgeCls}">${esc(label)}</span> ${esc(f.source || '')}${acctPill ? ' · ' + acctPill : ''}${f.transfer_risk === 'external-recipient' ? ' <span class="role-pill" style="background:#7B2D00;color:#FFD0B0">⚠ Ext.</span>' : f.transfer_risk ? ' <span class="role-pill" style="background:#003D7B;color:#B0D4FF">🔗</span>' : ''}</div>
</div>
<span class="cpr-badge">${f.cpr_count} CPR</span>
${f.email_count > 0 ? '<span class="email-badge">' + f.email_count + ' ' + t('m365_badge_emails', 'e-mail') + '</span> ' : ''}
@@ -84,7 +120,7 @@ function appendCard(f) {
<div class="card-name" title="${esc(f.name)}">${esc(f.name)}</div>
<div class="card-meta">${f.size_kb} KB · ${esc(f.modified || '')}</div>
${f.folder ? `<div class="card-meta" style="font-size:10px" title="${esc(f.folder)}">📂 ${esc(f.folder)}</div>` : ''}
<div class="card-source"><span class="source-badge ${badgeCls}">${esc(label)}</span>${f.account_name ? ' <span class="account-pill" title="' + esc(f.account_name) + '">' + (f.user_role === "student" ? '<span class="role-badge">' + t("role_student","Elev") + "</span>" : f.user_role === "staff" ? '<span class="role-badge">' + t("role_staff","Ansat") + "</span>" : "") + esc(f.account_name) + '</span>' : ''}${f.transfer_risk === "external-recipient" ? ' <span class="role-pill" style="background:#7B2D00;color:#FFD0B0">⚠ Ext.</span>' : f.transfer_risk ? ' <span class="role-pill" style="background:#003D7B;color:#B0D4FF">🔗</span>' : ''}</div>
<div class="card-source"><span class="source-badge ${badgeCls}">${esc(label)}</span>${acctPill ? ' ' + acctPill : ''}${f.transfer_risk === "external-recipient" ? ' <span class="role-pill" style="background:#7B2D00;color:#FFD0B0">⚠ Ext.</span>' : f.transfer_risk ? ' <span class="role-pill" style="background:#003D7B;color:#B0D4FF">🔗</span>' : ''}</div>
<span class="cpr-badge">${f.cpr_count} CPR</span>${f.email_count > 0 ? ' <span class="email-badge">' + f.email_count + ' ' + t('m365_badge_emails', 'e-mail') + '</span>' : ''}${f.phone_count > 0 ? ' <span class="phone-badge">' + f.phone_count + ' ' + t('m365_badge_phones', 'tlf.') + '</span>' : ''}${f.face_count > 0 ? ' <span class="photo-face-badge">' + f.face_count + ' ' + t('m365_badge_faces', f.face_count === 1 ? 'face' : 'faces') + '</span>' : ''}${f.exif && f.exif.gps ? ' <span class="photo-face-badge" style="background:#0a3a5a;color:#7ec8d0">🌍 GPS</span>' : ''}${f._deleted ? ' <span class="resolved-badge" style="background:#3a1a1a;color:#ff9b9b">🗑 ' + t('delete_badge', 'Deleted') + '</span>' : ''}${f._redacted ? ' <span class="resolved-badge">✏ ' + t('redact_badge', 'Redacted') + '</span>' : ''}${f._resolved ? ' <span class="resolved-badge">✓ ' + t('history_resolved_badge', 'Resolved') + '</span>' : ''}${f.overdue ? ' <span class="overdue-badge">🗓 Overdue</span>' : ''}
</div>
${delBtn}${redactBtn}`;
@@ -96,6 +132,17 @@ function renderGrid(files) {
const grid = document.getElementById('grid');
grid.innerHTML = '';
files.forEach(f => appendCard(f));
// Whenever results are rendered, the landing/last-scan cards must be hidden —
// the live scan_file_flagged path shows the grid but does not clear them, so
// results would otherwise appear underneath the still-visible landing page
// until a manual refresh. Centralised here so every render path is covered.
if (files && files.length) {
const es = document.getElementById('emptyState');
if (es) es.style.display = 'none';
const ls = document.getElementById('lastScanSummary');
if (ls) ls.style.display = 'none';
if (grid) grid.style.display = S.isListView ? 'block' : 'grid';
}
_updateBulkBar();
updateDispositionStats();
}
@@ -736,25 +783,34 @@ function _ensureSSE() {
function _sseWatchdog() {
fetch('/api/scan/status').then(function(r) { return r.json(); }).then(function(status) {
if (status.running) {
var anyRunning = status.running || status.google_running;
if (anyRunning) {
// A scan is in progress — make sure SSE is connected and progress UI is visible
_ensureSSE();
if (!S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) {
if (status.running && !S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) {
document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block';
// /api/scan/status checks the M365 lock — if running=true it's an M365 scan
_setStopBtnDisplay('inline-block');
// status.running reflects the M365 + file lock; treat as an M365 reconnect
S._m365ScanRunning = true; _renderProgressSegments();
document.getElementById('progressFile').textContent = t('m365_sse_reconnecting', 'Reconnecting to running scan…');
log(t('m365_sse_reconnecting', 'Reconnecting to running scan…'));
}
} else if (!S._historyRefScanId && !(S.flaggedData && S.flaggedData.length)) {
// No scan of any kind is running (authoritative, both locks free) and
// nothing is shown yet — restore the last saved session from the DB.
// Retried on every poll, not one-shot: the initial attempt can be blocked
// by running flags that SSE replay of a *completed* scan set but never
// cleared, and sse_replay_done only fires for a non-empty buffer (so it
// never retries after a server restart clears the replay buffer).
// Both locks are confirmed free, so clear any stale flags first.
S._m365ScanRunning = false;
S._googleScanRunning = false;
S._fileScanRunning = false;
window.loadHistorySession?.(null);
}
if (!_initialStatusChecked) {
_initialStatusChecked = true;
if (!status.running) window.loadHistorySession?.(null);
}
// When no scan is running, we still keep polling — the SSE connection
// may have died and we need to detect the *next* scheduled scan.
// The SSE itself is only opened/reopened when a scan is detected.
_initialStatusChecked = true;
// Keep polling even when idle — the SSE connection may have died and we
// need to detect the next scheduled scan (SSE is only opened on demand).
}).catch(function(err) {
// Status endpoint unavailable — server might be restarting
console.warn('[SSE] status poll failed:', err);
+65 -24
View File
@@ -317,7 +317,7 @@ function _attachScanListeners(source) {
else if (phaseSrc === 'file') { S._fileScanRunning = true; }
else { S._m365ScanRunning = true; }
document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block';
_setStopBtnDisplay('inline-block');
_renderProgressSegments();
}
_setProgressPhase(d.phase);
@@ -330,9 +330,9 @@ function _attachScanListeners(source) {
: (d.total > 0 ? Math.round((d.index || d.completed || 0) / d.total * 100) : 0);
S._srcPct[src] = pct;
// If reconnecting mid-scan the running flag may not be set yet — ensure segment exists
if (src === 'm365' && !S._m365ScanRunning) { S._m365ScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); }
if (src === 'google' && !S._googleScanRunning) { S._googleScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); }
if (src === 'file' && !S._fileScanRunning) { S._fileScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); }
if (src === 'm365' && !S._m365ScanRunning) { S._m365ScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
if (src === 'google' && !S._googleScanRunning) { S._googleScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
if (src === 'file' && !S._fileScanRunning) { S._fileScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
var fill = document.getElementById('progressFill_' + src);
if (fill) fill.style.width = pct + '%';
document.getElementById('progressFile').textContent = d.file || '';
@@ -368,15 +368,20 @@ function _attachScanListeners(source) {
log((d.file ? d.file + ': ' : '') + d.error, 'err');
});
source.addEventListener('scan_cancelled', function() {
if (S._userStartedScan) {
// scan_cancelled is the M365 engine's "stopped" event. Clear its running
// flag (leaving it set made later replays/status polls treat the stopped
// scan as live) and, like scan_done, only tear down the UI once no Google
// or file scan is still running.
S._m365ScanRunning = false;
var _anyRunning = S._googleScanRunning || S._fileScanRunning;
if (S._userStartedScan && !_anyRunning) {
S._userStartedScan = false;
if (S.es) { S.es.close(); S.es = null; }
}
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_clearProgressBar();
setLogLive('');
log('Scan stopped.', 'warn');
document.getElementById('scanBtn').disabled = _anyRunning;
_setStopBtnDisplay(_anyRunning ? 'inline-block' : 'none');
if (_anyRunning) { _renderProgressSegments(); } else { _clearProgressBar(); setLogLive(''); }
log(t('m365_scan_stopped', 'Scan stopped.'), 'warn');
});
source.addEventListener('scan_done', function(e) {
var d = JSON.parse(e.data);
@@ -401,7 +406,7 @@ function _attachScanListeners(source) {
}
if (!_anyRunning) setLogLive('');
document.getElementById('scanBtn').disabled = _anyRunning;
document.getElementById('stopBtn').style.display = _anyRunning ? 'inline-block' : 'none';
_setStopBtnDisplay(_anyRunning ? 'inline-block' : 'none');
if (!_anyRunning) _clearProgressBar();
document.getElementById('statsSection').style.display = 'block';
document.getElementById('statScanned').textContent = d.total_scanned;
@@ -437,7 +442,7 @@ function _attachScanListeners(source) {
}
setLogLive('');
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_setStopBtnDisplay('none');
_clearProgressBar();
document.getElementById('statsSection').style.display = 'block';
document.getElementById('statsPill').style.display = 'block';
@@ -466,7 +471,7 @@ function _attachScanListeners(source) {
}
setLogLive('');
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_setStopBtnDisplay('none');
_clearProgressBar();
document.getElementById('statsSection').style.display = 'block';
document.getElementById('statsPill').style.display = 'block';
@@ -502,7 +507,7 @@ function _attachSchedulerListeners(source) {
log('\uD83D\uDD50 ' + t('m365_sched_title','Scheduled scan') + ': ' + (d.job_name||'') + '\u2026');
// Show progress UI so scan_phase / scan_progress events are visible
document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block';
_setStopBtnDisplay('inline-block', true);
S._srcPct = { m365: 0, google: 0, file: 0 }; S._m365ScanRunning = true; _renderProgressSegments();
_setProgressPhase((d.job_name||'') + '\u2026');
document.getElementById('progressFile').textContent = '';
@@ -512,25 +517,32 @@ function _attachSchedulerListeners(source) {
// scheduler_started was missed (e.g. browser reconnected mid-scan)
console.log('[SSE] scan_start received');
document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block';
_setStopBtnDisplay('inline-block');
// Ensure at least the M365 segment is rendered (scan_start is M365-only)
if (!S._m365ScanRunning) { S._m365ScanRunning = true; _renderProgressSegments(); }
});
source.addEventListener('scheduler_done', function(e) {
var d = JSON.parse(e.data);
console.log('[SSE] scheduler_done received:', d);
// The job's stages are all over — scheduler_started set the M365 flag
S._m365ScanRunning = S._googleScanRunning = S._fileScanRunning = false;
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_setStopBtnDisplay('none');
_clearProgressBar();
log('\u2713 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + (d.flagged||0) + ' flagged', 'ok');
if (d.cancelled) {
log('\u23f9 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + t('m365_sched_stopped','stopped') + ' (' + (d.flagged||0) + ' flagged)', 'warn');
} else {
log('\u2713 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + (d.flagged||0) + ' flagged', 'ok');
}
markOverdueCards();
loadTrend();
});
source.addEventListener('scheduler_error', function(e) {
var d = JSON.parse(e.data);
console.log('[SSE] scheduler_error received:', d);
S._m365ScanRunning = S._googleScanRunning = S._fileScanRunning = false;
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_setStopBtnDisplay('none');
_clearProgressBar();
log('\u26A0 ' + t('m365_sched_title','Scheduled scan') + ' failed: ' + (d.error||''), 'err');
});
@@ -557,14 +569,20 @@ function startScan(resume) {
document.getElementById('resumeBanner').style.display = 'none';
document.getElementById('logPanel').innerHTML = '<div class="log-line log-live" id="logLive" style="display:none"></div>';
try { sessionStorage.removeItem(_LOG_SESSION_KEY); } catch(e) {}
S._m365ScanRunning = sources.length > 0;
// The M365 scan needs explicit accounts — the server treats an empty list as
// "every user in the tenant". With only Google accounts ticked, skip it.
const runM365 = sources.length > 0 && user_ids.length > 0;
if (sources.length && !runM365) {
log(t('m365_no_m365_accounts', 'No Microsoft 365 accounts selected — skipping the Microsoft 365 sources.'), 'warn');
}
S._m365ScanRunning = runM365;
S._googleScanRunning = googleSources.length > 0;
S._fileScanRunning = fileSources.length > 0;
S._srcPct = { m365: 0, google: 0, file: 0 };
S._progressCurrentUser = '';
_renderProgressSegments();
document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block';
_setStopBtnDisplay('inline-block', true);
// progress segments rendered by _renderProgressSegments() called above
document.getElementById('progressFile').textContent = '';
_setProgressPhase(t('scan_preparing', 'Preparing…'));
@@ -594,15 +612,15 @@ function startScan(resume) {
if (label === 'google') S._googleScanRunning = false;
if (!S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) {
document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none';
_setStopBtnDisplay('none');
if (S.es) { S.es.close(); S.es = null; }
S._userStartedScan = false;
}
}
setTimeout(() => {
// Fire M365 scan if any M365 sources are selected
if (sources.length > 0) {
// Fire M365 scan if any M365 sources and accounts are selected
if (runM365) {
fetch('/api/scan/start', {
method: 'POST', headers: {'Content-Type':'application/json'},
body: JSON.stringify({sources, user_ids, options, resume: !!resume,
@@ -664,8 +682,30 @@ function startScan(resume) {
}
// Show/hide the Stop button. Hiding — or passing fresh=true when a new scan
// starts — resets it from its "Stopping…" state, so a new scan always gets a
// normal, clickable Stop button even if a previous scan never reported its end.
function _setStopBtnDisplay(display, fresh) {
const btn = document.getElementById('stopBtn');
if (!btn) return;
btn.style.display = display;
if (display === 'none' || fresh) {
btn.disabled = false;
btn.textContent = t('m365_btn_stop', 'Stop');
}
}
function stopScan() {
fetch('/api/scan/stop', {method:'POST'});
// Immediate feedback: the scan only notices the stop between items, which
// can take a while (e.g. OCR on a large PDF). Stays "Stopping…" until the
// scan's done/stopped event hides the button.
const btn = document.getElementById('stopBtn');
if (btn) { btn.disabled = true; btn.textContent = t('m365_btn_stopping', 'Stopping…'); }
log(t('m365_scan_stopping', 'Stopping scan — finishing the current item…'), 'warn');
fetch('/api/scan/stop', {method:'POST'}).catch(function(e) {
if (btn) { btn.disabled = false; btn.textContent = t('m365_btn_stop', 'Stop'); }
log('Stop failed: ' + e, 'err');
});
}
// ── Trend sparkline (#7) ──────────────────────────────────────────────────────
@@ -795,6 +835,7 @@ window._attachScanListeners = _attachScanListeners;
window._attachSchedulerListeners = _attachSchedulerListeners;
window.startScan = startScan;
window.stopScan = stopScan;
window._setStopBtnDisplay = _setStopBtnDisplay;
window.drawSparkline = drawSparkline;
window.loadTrend = loadTrend;
window.updateStats = updateStats;
+12 -5
View File
@@ -242,7 +242,8 @@ function schedLoadHistory() {
var html = '';
runs.forEach(function(r) {
var ts = r.started_at ? new Date(r.started_at * 1000).toLocaleString() : '-';
var icon = r.status === 'completed' ? '\u2713' : r.status === 'failed' ? '\u2716' : '\u23f3';
var icon = r.status === 'completed' ? '\u2713' : r.status === 'failed' ? '\u2716'
: r.status === 'cancelled' ? '\u23f9' : '\u23f3';
var jname = r.job_name ? '<strong>' + _esc(r.job_name) + '</strong> - ' : '';
html += icon + ' ' + jname + ts + ' - ' + (r.flagged||0) + ' flagged';
if (r.emailed) html += ' \u2709';
@@ -314,15 +315,17 @@ function stLoadSmtp() {
const set = function(id, val) { const el=document.getElementById(id); if(el) el.value=val||''; };
set('st-smtpHost', d.host);
set('st-smtpPort', d.port || 587);
set('st-smtpUser', d.user);
set('st-smtpUser', d.username);
set('st-smtpFrom', d.from_addr);
set('st-smtpTo', Array.isArray(d.recipients) ? d.recipients.join(', ') : (d.recipients||''));
const tls = document.getElementById('st-smtpTls');
if (tls) tls.checked = d.starttls !== false;
if (tls) tls.checked = d.use_tls !== false;
const pw = document.getElementById('st-smtpPw');
if (pw) pw.value = d.has_password ? '\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022' : '';
const ae = document.getElementById('st-smtpAutoEmail');
if (ae) ae.checked = !!d.auto_email_manual;
const ps = document.getElementById('st-smtpPreferSmtp');
if (ps) ps.checked = !!d.prefer_smtp;
}).catch(function(){});
}
@@ -333,11 +336,15 @@ async function stSmtpSave() {
const body = {
host: document.getElementById('st-smtpHost').value.trim(),
port: parseInt(document.getElementById('st-smtpPort').value) || 587,
user: document.getElementById('st-smtpUser').value.trim(),
// Backend (routes/email.py) reads these exact keys — `username`/`use_tls`,
// not `user`/`starttls`. Sending the wrong keys leaves username empty so
// server.login() is skipped and the SMTP server rejects the send.
username: document.getElementById('st-smtpUser').value.trim(),
from_addr: document.getElementById('st-smtpFrom').value.trim(),
recipients: document.getElementById('st-smtpTo').value.split(/[,;]/).map(function(s){return s.trim();}).filter(Boolean),
starttls: document.getElementById('st-smtpTls').checked,
use_tls: document.getElementById('st-smtpTls').checked,
auto_email_manual: !!(document.getElementById('st-smtpAutoEmail') || {}).checked,
prefer_smtp: !!(document.getElementById('st-smtpPreferSmtp') || {}).checked,
};
if (pw !== null) body.password = pw;
st.style.color = 'var(--muted)'; st.textContent = t('m365_smtp_saving','Saving...');
+2
View File
@@ -19,6 +19,8 @@ export const S = {
_progressCurrentUser: '',
// Users
_allUsers: [],
_m365UsersLoaded: false, // true once loadUsers() has fetched the directory
_googleUsersMerged: false, // true once _mergeGoogleUsers() has added Google users
// Auth
_currentAppMode: null,
// Profiles
+25 -9
View File
@@ -28,6 +28,7 @@ async function loadUsers() {
u.selected = prevSelected.has(u.id) ? prevSelected.get(u.id) : false;
});
S._allUsers = [...fetched, ...toAdd];
S._m365UsersLoaded = true;
// Apply deferred "select all" from a profile chosen before users loaded
if (window._pendingProfileAllUsers) {
S._allUsers.forEach(u => { u.selected = true; });
@@ -107,6 +108,12 @@ async function _mergeGoogleUsers() {
S._allUsers = S._allUsers.concat(googleUsers);
renderAccountList();
} catch(e) { /* Google users unavailable */ }
finally {
// Google users are now as complete as they'll get (even on error/early
// return) — finish a profile apply that was waiting on google:<email> IDs.
S._googleUsersMerged = true;
_applyPendingProfileUsers();
}
}
let _activeRoleFilter = ''; // '' = all, 'staff', 'student'
@@ -415,7 +422,7 @@ async function addUserManually() {
const d = await r.json();
if (d.error) { alert('User not found: ' + d.error); return; }
if (S._allUsers.find(u => u.id === d.id)) { alert('User already in list.'); return; }
S._allUsers.push({...d, manual: true});
S._allUsers.push({...d, platform: 'm365', manual: true});
input.value = '';
renderAccountList(S._allUsers.filter(u => !u.manual).length <= 1);
} catch(e) {
@@ -441,21 +448,30 @@ function selectAllAccounts(checked) {
function getSelectedUsers() {
// Only return M365 users — Google users are handled separately via selectedGoogleEmails
let selected = S._allUsers.filter(u => u.selected !== false && (u.platform === 'm365' || u.platform === 'both'));
// Manually added users may have no platform — they are M365 lookups, so treat missing as 'm365'
let selected = S._allUsers.filter(u => u.selected !== false && ['m365', 'both'].includes(u.platform || 'm365'));
// Respect the active role filter — hidden users must not sneak into the scan
// even if they were checked before the filter was applied.
if (_activeRoleFilter) {
selected = selected.filter(u => (u.userRole || 'other') === _activeRoleFilter);
}
if (selected.length) {
return selected.map(u => ({
id: u.id, displayName: u.displayName, userRole: u.userRole || 'other'
}));
// manual/email let a profile saved from this selection re-add manual users on apply
return selected.map(u => Object.assign(
{ id: u.id, displayName: u.displayName, userRole: u.userRole || 'other' },
u.manual ? { manual: true, email: u.email || '' } : {}));
}
// Fallback to DOM if S._allUsers not yet populated
return Array.from(document.querySelectorAll('.account-check:checked')).map(cb => ({
id: cb.dataset.id, displayName: cb.dataset.name, userRole: cb.dataset.role || 'other'
}));
// Once the user list is loaded it is authoritative: no ticked M365 users means
// none. (Falling back to the DOM here collected every ticked checkbox —
// including Google-only "google:<email>" rows — and sent them to the M365
// scan, where Graph 404s on them.)
if (S._allUsers.length) return [];
// Fallback to DOM only while S._allUsers is not yet populated
return Array.from(document.querySelectorAll('.account-check:checked'))
.filter(cb => !String(cb.dataset.id || '').startsWith('google:'))
.map(cb => ({
id: cb.dataset.id, displayName: cb.dataset.name, userRole: cb.dataset.role || 'other'
}));
}
// ── Window exports (HTML handlers + cross-module calls) ─────────────────────
+16 -20
View File
@@ -154,7 +154,6 @@ function _resetShareForm() {
function openShareModal() {
document.getElementById('shareBackdrop').classList.add('open');
document.getElementById('shareNewLinkRow').style.display = 'none';
_resetShareForm();
_renderTokenList();
fetch('/api/viewer/pin').then(function(r){ return r.json(); }).then(function(d) {
@@ -167,7 +166,7 @@ function closeShareModal() {
document.getElementById('shareBackdrop').classList.remove('open');
}
async function _renderTokenList() {
async function _renderTokenList(highlightToken) {
const list = document.getElementById('shareTokenList');
list.innerHTML = '<div style="font-size:12px;color:var(--muted);padding:4px 0">' + t('lbl_loading', 'Loading…') + '</div>';
try {
@@ -222,6 +221,17 @@ async function _renderTokenList() {
'<button title="' + t('share_revoke', 'Revoke') + '" onclick="revokeToken(\'' + tok.token + '\',this.closest(\'div[style]\'))" ' +
'style="height:24px;padding:0 8px;background:none;border:1px solid var(--danger);color:var(--danger);border-radius:4px;font-size:11px;cursor:pointer;flex-shrink:0">' + t('share_revoke', 'Revoke') + '</button>';
list.appendChild(row);
// Briefly highlight a freshly created link so it is easy to find and copy.
if (highlightToken && tok.token === highlightToken) {
row.style.transition = 'border-color .3s, background .3s';
row.style.borderColor = 'var(--accent)';
row.style.background = 'rgba(80,160,80,.18)';
setTimeout(function() { row.scrollIntoView({block: 'nearest'}); }, 0);
setTimeout(function() {
row.style.borderColor = 'var(--border)';
row.style.background = 'var(--bg)';
}, 2500);
}
});
} catch(e) {
list.innerHTML = '<div style="font-size:12px;color:var(--danger);padding:4px 0">' + t('share_load_error', 'Failed to load links.') + '</div>';
@@ -264,23 +274,16 @@ async function createShareLink() {
});
if (!r.ok) throw new Error('Server error ' + r.status);
const entry = await r.json();
const url = (await _getShareBaseUrl()) + '/view?token=' + encodeURIComponent(entry.token);
const urlInput = document.getElementById('shareNewLinkUrl');
urlInput.value = url;
document.getElementById('shareNewLinkRow').style.display = 'block';
document.getElementById('shareCopyBtn').textContent = t('log_copy', 'Copy');
// The new link appears in the active-links list below (each row has its
// own Copy button) — reset the form and highlight the just-created row
// rather than leaving a stale link preview in the create box.
_resetShareForm();
_renderTokenList();
_renderTokenList(entry.token);
} catch(e) {
alert(t('share_create_error', 'Failed to create link:') + ' ' + e.message);
}
}
function copyShareLink() {
const url = document.getElementById('shareNewLinkUrl').value;
_copyText(url, document.getElementById('shareCopyBtn'));
}
async function copyTokenLink(token, btn) {
const url = (await _getShareBaseUrl()) + '/view?token=' + encodeURIComponent(token);
_copyText(url, btn);
@@ -328,12 +331,6 @@ async function revokeToken(token, rowEl) {
if (!list.children.length) {
list.innerHTML = '<div style="font-size:12px;color:var(--muted);padding:4px 0">' + t('share_no_links', 'No active links.') + '</div>';
}
// Hide the copy row if the just-revoked token was the last created
const newRow = document.getElementById('shareNewLinkRow');
if (newRow) {
const shownUrl = document.getElementById('shareNewLinkUrl')?.value || '';
if (shownUrl.includes(token)) newRow.style.display = 'none';
}
} catch(e) {
alert(t('share_revoke_error', 'Failed to revoke:') + ' ' + e.message);
}
@@ -502,7 +499,6 @@ window._shareScopeTypeChanged = _shareScopeTypeChanged;
window.openShareModal = openShareModal;
window.closeShareModal = closeShareModal;
window.createShareLink = createShareLink;
window.copyShareLink = copyShareLink;
window._copyText = _copyText;
window.copyTokenLink = copyTokenLink;
window.revokeToken = revokeToken;
+5 -8
View File
@@ -375,7 +375,7 @@ document.addEventListener('DOMContentLoaded', applyI18n);
<button id="historyPickerBtn" type="button" onclick="openHistoryPicker()" style="height:24px;padding:0 10px;background:none;border:1px solid var(--border);color:var(--muted);border-radius:4px;font-size:11px;cursor:pointer" data-i18n="history_btn_sessions">Sessions</button>
<div id="historyDropdown" style="display:none;position:absolute;right:0;top:calc(100% + 4px);background:var(--surface);border:1px solid var(--border);border-radius:6px;z-index:9999;width:300px;max-height:260px;overflow-y:auto;box-shadow:0 4px 12px rgba(0,0,0,.25)"></div>
</div>
<button id="historyLatestBtn" type="button" onclick="loadHistorySession(null)" style="display:none;height:24px;padding:0 10px;background:none;border:1px solid var(--accent);color:var(--accent);border-radius:4px;font-size:11px;cursor:pointer;flex-shrink:0" data-i18n="history_btn_latest">Latest scan</button>
<button id="historyLatestBtn" type="button" onclick="loadHistorySession(null)" style="display:none;height:24px;padding:0 10px;background:none;border:1px solid var(--accent);color:var(--accent);border-radius:4px;font-size:11px;cursor:pointer;flex-shrink:0" data-i18n="history_btn_latest">Open items</button>
</div>
<!-- Filter bar — full width, above grid + preview -->
@@ -845,6 +845,10 @@ document.addEventListener('DOMContentLoaded', applyI18n);
<label data-i18n="m365_smtp_auto_email_manual">Email report after manual scan</label>
<label class="toggle" style="flex:unset"><input type="checkbox" id="st-smtpAutoEmail"><span class="toggle-slider"></span></label>
</div>
<div class="settings-row">
<label data-i18n="m365_smtp_prefer_smtp">Always send via SMTP (skip Microsoft Graph)</label>
<label class="toggle" style="flex:unset"><input type="checkbox" id="st-smtpPreferSmtp"><span class="toggle-slider"></span></label>
</div>
<div style="display:flex;justify-content:flex-end;gap:8px;margin-top:4px">
<div id="st-smtpStatus" style="flex:1;font-size:11px;color:var(--muted);align-self:center"></div>
<button onclick="stSmtpSave()" style="background:none;border:1px solid var(--border);color:var(--muted);height:26px;padding:0 12px;border-radius:6px;font-size:12px;cursor:pointer;box-sizing:border-box" data-i18n="btn_save">Save</button>
@@ -1094,13 +1098,6 @@ document.addEventListener('DOMContentLoaded', applyI18n);
</div>
<button onclick="createShareLink()" style="height:30px;padding:0 14px;background:var(--accent);color:#fff;border:none;border-radius:5px;font-size:12px;cursor:pointer;flex-shrink:0" data-i18n="share_create">Create</button>
</div>
<div id="shareNewLinkRow" style="display:none;margin-top:10px">
<div style="font-size:11px;color:var(--muted);margin-bottom:4px" data-i18n="share_copy_link_prompt">Copy link:</div>
<div style="display:flex;gap:6px;align-items:center">
<input id="shareNewLinkUrl" type="text" readonly style="flex:1;font-size:11px;padding:5px 8px;background:var(--bg2,var(--bg));border:1px solid var(--border);border-radius:5px;color:var(--text);min-width:0">
<button onclick="copyShareLink()" id="shareCopyBtn" style="height:26px;padding:0 10px;background:none;border:1px solid var(--border);color:var(--muted);border-radius:5px;font-size:11px;cursor:pointer;flex-shrink:0" data-i18n="log_copy">Copy</button>
</div>
</div>
</div>
<!-- Existing tokens -->
+169
View File
@@ -0,0 +1,169 @@
"""
Result cards must show the account's name and Elev/Ansat role.
Regression: profiles saved by the profile editor stored bare id strings, and
scheduled scans pass those straight to run_scan, which normalised them to
displayName=<id>, userRole="other" — so cards showed the GUID as the name and
no role badge. run_scan now looks bare accounts up in the directory.
"""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
import scan_engine
class _Conn:
"""Directory stub: list_users + role classification + single lookups."""
def __init__(self, directory=None, extra=None, list_fails=False):
self.directory = directory or []
self.extra = extra or {} # users only reachable via lookup_user
self.list_fails = list_fails
self.list_calls = 0
self.lookups: list[str] = []
def list_users(self):
self.list_calls += 1
if self.list_fails:
raise RuntimeError("Authorization_RequestDenied")
return list(self.directory)
def build_role_sku_map(self, users):
return {"sku-staff": "STANDARDWOFFPACK_FACULTY", "sku-stud": "STANDARDWOFFPACK_STUDENT"}
def classify_user_role(self, assigned_licenses, sku_map):
names = {sku_map.get(l.get("skuId", ""), "") for l in assigned_licenses}
if any("FACULTY" in n for n in names):
return "staff"
if any("STUDENT" in n for n in names):
return "student"
return "other"
def lookup_user(self, uid):
self.lookups.append(uid)
if uid in self.extra:
return self.extra[uid]
raise RuntimeError("Request_ResourceNotFound")
DIRECTORY = [
{"id": "guid-henrik", "displayName": "Henrik Højmark", "mail": "henrik@school.dk",
"assignedLicenses": [{"skuId": "sku-staff"}]},
{"id": "guid-elna", "displayName": "Elna Sofie", "mail": "elna@school.dk",
"assignedLicenses": [{"skuId": "sku-stud"}]},
]
def _bare(uid): # what run_scan makes of a bare id string
return {"id": uid, "displayName": uid, "userRole": "other"}
class TestEnrichBareUsers:
def test_bare_ids_get_name_and_role_from_directory(self):
out = scan_engine._enrich_bare_users(_Conn(DIRECTORY), [_bare("guid-henrik"), _bare("guid-elna")])
assert out == [
{"id": "guid-henrik", "displayName": "Henrik Højmark", "userRole": "staff"},
{"id": "guid-elna", "displayName": "Elna Sofie", "userRole": "student"},
]
def test_named_accounts_untouched_and_directory_not_listed(self):
conn = _Conn(DIRECTORY)
users = [{"id": "guid-henrik", "displayName": "Henrik Højmark", "userRole": "staff"}]
assert scan_engine._enrich_bare_users(conn, users) == users
assert conn.list_calls == 0 # no directory call when nothing is bare
def test_existing_role_is_kept(self):
# e.g. a role the UI already knew — only a missing ("other") role is filled
out = scan_engine._enrich_bare_users(
_Conn(DIRECTORY), [{"id": "guid-elna", "displayName": "guid-elna", "userRole": "staff"}])
assert out[0]["displayName"] == "Elna Sofie" and out[0]["userRole"] == "staff"
def test_account_outside_member_list_is_looked_up(self):
extra = {"guid-guest": {"id": "guid-guest", "displayName": "Gæst Lærer", "mail": "g@school.dk",
"assignedLicenses": [{"skuId": "sku-staff"}]}}
conn = _Conn(DIRECTORY, extra=extra)
out = scan_engine._enrich_bare_users(conn, [_bare("guid-guest")])
assert out == [{"id": "guid-guest", "displayName": "Gæst Lærer", "userRole": "staff"}]
assert conn.lookups == ["guid-guest"]
def test_directory_list_unavailable_falls_back_to_lookup(self):
extra = {"guid-henrik": DIRECTORY[0]}
out = scan_engine._enrich_bare_users(_Conn(list_fails=True, extra=extra), [_bare("guid-henrik")])
assert out[0]["displayName"] == "Henrik Højmark"
def test_unknown_account_keeps_what_it_had(self):
out = scan_engine._enrich_bare_users(_Conn(DIRECTORY), [_bare("guid-gone")])
assert out == [_bare("guid-gone")]
class TestBuildRoleSkuMap:
def _conn(self):
from m365_connector import M365Connector
return M365Connector.__new__(M365Connector) # no auth needed for these calls
def test_merges_tenant_and_per_user_skus(self, monkeypatch):
conn = self._conn()
monkeypatch.setattr(conn, "get_subscribed_skus", lambda: {"a": "ADMIN_SKU"})
monkeypatch.setattr(conn, "build_sku_map_from_users", lambda users: {"b": "FACULTY_SKU"})
assert conn.build_role_sku_map([{"id": "x"}]) == {"a": "ADMIN_SKU", "b": "FACULTY_SKU"}
def test_never_raises(self, monkeypatch):
conn = self._conn()
def _boom(*a, **kw): raise RuntimeError("403")
monkeypatch.setattr(conn, "get_subscribed_skus", _boom)
monkeypatch.setattr(conn, "build_sku_map_from_users", _boom)
assert conn.build_role_sku_map([]) == {}
class TestUsersRouteUsesSharedMap:
def test_api_users_classifies_with_build_role_sku_map(self, monkeypatch):
import gdpr_scanner
from routes import state
import routes.users as users_mod
monkeypatch.setattr(users_mod, "_load_role_overrides", lambda: {})
conn = MagicMock()
conn.is_app_mode = True
conn.list_users.return_value = DIRECTORY
conn.build_role_sku_map.return_value = {"sku-staff": "X_FACULTY"}
conn.classify_user_role.side_effect = lambda lic, m: "staff" if m else "other"
monkeypatch.setattr(state, "connector", conn)
gdpr_scanner.app.config["TESTING"] = True
with gdpr_scanner.app.test_client() as c:
data = c.get("/api/users").get_json()
conn.build_role_sku_map.assert_called_once_with(DIRECTORY)
assert {u["userRole"] for u in data["users"]} == {"staff"}
class TestRunScanNamesBareIds:
def test_scheduled_style_bare_ids_are_named_in_the_scan(self, monkeypatch):
"""End to end: bare id strings (as a scheduled scan passes them) reach the
per-user collection with their real name."""
from routes import state
import sse
sent = []
monkeypatch.setattr(scan_engine, "broadcast", lambda ev, data: sent.append((ev, data)))
monkeypatch.setattr(scan_engine, "DB_OK", False)
monkeypatch.setattr(scan_engine, "_load_checkpoint", lambda key, **kw: None)
monkeypatch.setattr(scan_engine, "_save_checkpoint", lambda *a, **kw: None)
monkeypatch.setattr(scan_engine, "_clear_checkpoint", lambda **kw: None)
monkeypatch.setattr(scan_engine, "_load_role_overrides", lambda: {})
monkeypatch.setattr(state, "flagged_items", [])
stub = _Conn(DIRECTORY)
conn = MagicMock()
conn.is_app_mode = True
conn.list_users.side_effect = stub.list_users
conn.build_role_sku_map.side_effect = stub.build_role_sku_map
conn.classify_user_role.side_effect = stub.classify_user_role
conn.list_all_mail_folders_for.return_value = []
monkeypatch.setattr(state, "connector", conn)
saved = sse._current_scan_id
try:
scan_engine.run_scan({"sources": ["email"], "options": {},
"user_ids": ["guid-henrik", "guid-elna"]})
finally:
sse._current_scan_id = saved
phases = " | ".join(d.get("phase", "") for e, d in sent if e == "scan_phase")
assert "Henrik Højmark" in phases and "Elna Sofie" in phases
assert "guid-henrik" not in phases
+33
View File
@@ -252,3 +252,36 @@ class TestFernet:
def test_decrypt_empty_returns_empty(self):
result = app_config._decrypt_password("")
assert result == ""
class TestSmtpConfigLegacyKeys:
"""SMTP config saved by the older settings tab used `user`/`starttls`;
readers expect `username`/`use_tls`. _load_smtp_config must normalise them."""
def test_legacy_keys_normalised_on_load(self, tmp_path, monkeypatch):
import json
p = tmp_path / "smtp.json"
p.write_text(json.dumps({
"host": "smtp.gmail.com", "port": 587,
"user": "netadmin@adm.example.dk", # legacy key
"starttls": True, # legacy key
"from_addr": "netadmin@adm.example.dk",
"recipients": ["a@example.dk"],
}), encoding="utf-8")
monkeypatch.setattr(app_config, "_SMTP_CONFIG_PATH", p)
cfg = app_config._load_smtp_config()
assert cfg["username"] == "netadmin@adm.example.dk"
assert cfg["use_tls"] is True
def test_canonical_keys_take_precedence(self, tmp_path, monkeypatch):
import json
p = tmp_path / "smtp.json"
p.write_text(json.dumps({
"username": "canonical@example.dk",
"user": "legacy@example.dk",
}), encoding="utf-8")
monkeypatch.setattr(app_config, "_SMTP_CONFIG_PATH", p)
cfg = app_config._load_smtp_config()
assert cfg["username"] == "canonical@example.dk"
+68
View File
@@ -265,3 +265,71 @@ class TestExportImport:
tgt.import_db(str(export_path), mode="replace")
results = tgt.lookup_data_subject("290472-1234")
assert len(results) >= 1
# ─────────────────────────────────────────────────────────────────────────────
# Orphan-scan recovery (crash / kill / mid-scan restart)
# ─────────────────────────────────────────────────────────────────────────────
class TestOrphanScanRecovery:
def _start_unfinished_scan(self, db, item_id):
"""Begin a scan and save an item but never call finish_scan."""
sid = db.begin_scan({"sources": ["email"], "user_ids": []})
db.save_item(sid, _make_card(item_id=item_id))
return sid
def test_unfinished_scan_items_hidden_until_recovery(self, tmp_db):
self._start_unfinished_scan(tmp_db, "orphan-1")
# Not finalised → invisible to the open-items view
assert tmp_db.get_open_items() == []
def test_recovery_finalises_and_reveals_items(self, tmp_db):
self._start_unfinished_scan(tmp_db, "orphan-1")
self._start_unfinished_scan(tmp_db, "orphan-2")
recovered = tmp_db.finalize_orphan_scans()
assert recovered == 2
ids = {row["id"] for row in tmp_db.get_open_items()}
assert ids == {"orphan-1", "orphan-2"}
def test_recovery_leaves_finished_scans_untouched(self, tmp_db):
sid = tmp_db.begin_scan({"sources": ["email"], "user_ids": []})
tmp_db.save_item(sid, _make_card(item_id="done-1"))
tmp_db.finish_scan(sid, total_scanned=1)
before = tmp_db._connect().execute(
"SELECT finished_at FROM scans WHERE id=?", (sid,)
).fetchone()[0]
assert tmp_db.finalize_orphan_scans() == 0 # nothing to recover
after = tmp_db._connect().execute(
"SELECT finished_at FROM scans WHERE id=?", (sid,)
).fetchone()[0]
assert after == before # finished_at not rewritten
def test_recovery_is_idempotent(self, tmp_db):
self._start_unfinished_scan(tmp_db, "orphan-1")
assert tmp_db.finalize_orphan_scans() == 1
assert tmp_db.finalize_orphan_scans() == 0
# ─────────────────────────────────────────────────────────────────────────────
# account_name persistence (user/group badge data)
# ─────────────────────────────────────────────────────────────────────────────
class TestAccountNamePersistence:
def test_account_name_round_trips(self, tmp_db):
sid = tmp_db.begin_scan({"sources": ["email"], "user_ids": []})
tmp_db.save_item(sid, _make_card(item_id="an-1")) # account_name="Test User"
tmp_db.finish_scan(sid, total_scanned=1)
row = [r for r in tmp_db.get_open_items() if r["id"] == "an-1"][0]
assert row.get("account_name") == "Test User"
def test_account_name_column_exists(self, tmp_db):
cols = [r[1] for r in tmp_db._connect().execute(
"PRAGMA table_info(flagged_items)").fetchall()]
assert "account_name" in cols
+150
View File
@@ -0,0 +1,150 @@
"""
Manual accounts in scan profiles.
The profile editor stores manually added accounts either with their directory
id (looked up via GET /api/users/lookup) or, when M365 wasn't reachable / for
profiles saved by older versions, as "manual:<email>". The scan engine must
resolve the latter to a Graph user id or they are never scanned.
"""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
import scan_engine
class _Conn:
def __init__(self, directory: dict[str, dict]):
self.directory = directory
self.lookups: list[str] = []
def lookup_user(self, upn: str) -> dict:
self.lookups.append(upn)
if upn not in self.directory:
raise RuntimeError("Request_ResourceNotFound")
return self.directory[upn]
@pytest.fixture()
def broadcasts(monkeypatch):
sent = []
monkeypatch.setattr(scan_engine, "broadcast", lambda ev, data: sent.append((ev, data)))
return sent
class TestResolveManualUsers:
def test_manual_email_resolved_to_directory_id(self, broadcasts):
conn = _Conn({"ann@school.dk": {"id": "guid-ann", "displayName": "Ann Andersen",
"mail": "ann@school.dk"}})
out = scan_engine._resolve_scan_users(conn, [
{"id": "manual:ann@school.dk", "displayName": "manual:ann@school.dk", "userRole": "other"},
])
assert out == [{"id": "guid-ann", "displayName": "Ann Andersen", "userRole": "other"}]
assert conn.lookups == ["ann@school.dk"]
assert broadcasts == []
def test_regular_and_resolved_manual_ids_pass_through_without_lookup(self, broadcasts):
conn = _Conn({})
users = [{"id": "guid-1", "displayName": "One"},
{"id": "guid-2", "displayName": "Two", "manual": True, "email": "two@school.dk"}]
assert scan_engine._resolve_scan_users(conn, users) == users
assert conn.lookups == []
def test_unknown_manual_account_skipped_and_reported(self, broadcasts):
conn = _Conn({})
out = scan_engine._resolve_scan_users(conn, [
{"id": "guid-1", "displayName": "One"},
{"id": "manual:ghost@school.dk", "displayName": "ghost"},
])
assert out == [{"id": "guid-1", "displayName": "One"}]
assert broadcasts and broadcasts[0][0] == "scan_error"
assert "ghost@school.dk" in broadcasts[0][1]["file"]
def test_manual_account_already_selected_is_deduplicated(self, broadcasts):
conn = _Conn({"ann@school.dk": {"id": "guid-ann", "displayName": "Ann"}})
out = scan_engine._resolve_scan_users(conn, [
{"id": "guid-ann", "displayName": "Ann"},
{"id": "manual:ann@school.dk", "displayName": "ann"},
])
assert [u["id"] for u in out] == ["guid-ann"]
def test_all_manual_unresolvable_gives_empty_list_not_all_users(self, broadcasts):
# run_scan resolves inside the non-empty branch, so an empty result must
# stay empty — it must never widen to "scan every tenant user".
conn = _Conn({})
assert scan_engine._resolve_scan_users(
conn, [{"id": "manual:x@school.dk", "displayName": "x"}]) == []
class TestGoogleOnlyAccountsDropped:
"""Regression: google:<email> ids reached the M365 scan and Graph 404'd
("/users/google:anne@school.dk/mailFolders")."""
def test_google_only_accounts_are_dropped_and_reported(self, broadcasts):
conn = _Conn({})
out = scan_engine._resolve_scan_users(conn, [
{"id": "guid-1", "displayName": "One"},
{"id": "google:anne@school.dk", "displayName": "Anne"},
{"id": "google:appleadmin@adm.school.dk", "displayName": "Apple Admin"},
])
assert out == [{"id": "guid-1", "displayName": "One"}]
assert conn.lookups == []
phases = [d["phase"] for e, d in broadcasts if e == "scan_phase"]
assert phases and "2" in phases[0]
def test_only_google_accounts_gives_empty_list_not_all_users(self, broadcasts):
assert scan_engine._resolve_scan_users(_Conn({}), [
{"id": "google:anne@school.dk", "displayName": "Anne"}]) == []
def test_run_scan_never_calls_graph_for_google_ids(self, broadcasts, monkeypatch):
from routes import state
monkeypatch.setattr(scan_engine, "DB_OK", False)
monkeypatch.setattr(scan_engine, "_load_checkpoint", lambda key, **kw: None)
monkeypatch.setattr(scan_engine, "_save_checkpoint", lambda *a, **kw: None)
monkeypatch.setattr(scan_engine, "_clear_checkpoint", lambda **kw: None)
monkeypatch.setattr(state, "flagged_items", [])
conn = MagicMock()
conn.is_app_mode = True
conn.list_all_mail_folders_for.return_value = []
monkeypatch.setattr(state, "connector", conn)
import sse
saved = sse._current_scan_id
try:
scan_engine.run_scan({"sources": ["email"], "options": {}, "user_ids": [
{"id": "google:anne@school.dk", "displayName": "Anne"},
{"id": "guid-1", "displayName": "One"},
]})
finally:
sse._current_scan_id = saved
called_ids = {c.args[0] for c in conn.list_all_mail_folders_for.call_args_list}
assert "guid-1" in called_ids
assert not any(str(i).startswith("google:") for i in called_ids)
conn.list_users.assert_not_called() # never widened to the whole tenant
class TestLookupRoute:
@pytest.fixture()
def client(self):
import gdpr_scanner
gdpr_scanner.app.config["TESTING"] = True
with gdpr_scanner.app.test_client() as c:
yield c
def test_lookup_uses_connector(self, client, monkeypatch):
from routes import state
conn = MagicMock()
conn.lookup_user.return_value = {"id": "guid-ann", "displayName": "Ann",
"mail": "ann@school.dk"}
monkeypatch.setattr(state, "connector", conn)
r = client.get("/api/users/lookup?upn=ann@school.dk")
assert r.status_code == 200
assert r.get_json()["id"] == "guid-ann"
conn.lookup_user.assert_called_once_with("ann@school.dk")
def test_lookup_not_found_is_404(self, client, monkeypatch):
from routes import state
conn = MagicMock()
conn.lookup_user.side_effect = RuntimeError("Request_ResourceNotFound")
monkeypatch.setattr(state, "connector", conn)
assert client.get("/api/users/lookup?upn=ghost@school.dk").status_code == 404
+117
View File
@@ -270,6 +270,49 @@ class TestFlaggedScopeEnforcement:
ids = {row["id"] for row in r.get_json()}
assert "ci1" in ids
def test_no_ref_returns_open_items_across_all_sessions(self, client, db_patch):
# Two scans in separate session windows. The default (no-ref) view must
# surface unactioned items from BOTH, not just the latest session.
old_id = _seed_scan(db_patch, [_item("o1")])
db_patch._connect().execute(
"UPDATE scans SET started_at = started_at - 400 WHERE id = ?", (old_id,)
)
db_patch._connect().commit()
_seed_scan(db_patch, [_item("o2")])
r = client.get("/api/db/flagged")
ids = {row["id"] for row in r.get_json()}
assert ids == {"o1", "o2"}
def test_no_ref_excludes_items_with_a_disposition(self, client, db_patch):
_seed_scan(db_patch, [_item("d1"), _item("d2")])
db_patch.set_disposition("d1", "kept")
r = client.get("/api/db/flagged")
ids = {row["id"] for row in r.get_json()}
assert "d2" in ids # untouched → still open
assert "d1" not in ids # action taken → hidden
def test_no_ref_unreviewed_disposition_stays_open(self, client, db_patch):
_seed_scan(db_patch, [_item("u1")])
db_patch.set_disposition("u1", "unreviewed")
r = client.get("/api/db/flagged")
ids = {row["id"] for row in r.get_json()}
assert "u1" in ids # 'unreviewed' status is not an action
def test_no_ref_dedupes_rescanned_item_to_latest(self, client, db_patch):
# Same item flagged by two scans → appears once.
old_id = _seed_scan(db_patch, [_item("k1")])
db_patch._connect().execute(
"UPDATE scans SET started_at = started_at - 400 WHERE id = ?", (old_id,)
)
db_patch._connect().commit()
_seed_scan(db_patch, [_item("k1")])
rows = [row for row in client.get("/api/db/flagged").get_json() if row["id"] == "k1"]
assert len(rows) == 1
def test_ref_param_loads_historical_session(self, client, db_patch):
# Push first scan >300 s into the past so it occupies its own session window.
old_id = _seed_scan(db_patch, [_item("h1")])
@@ -618,3 +661,77 @@ class TestProfileRoutes:
def test_delete_missing_key_returns_400(self, client):
r = client.post("/api/profiles/delete", json={})
assert r.status_code == 400
# ---------------------------------------------------------------------------
# Scan start must not write profiles
# ---------------------------------------------------------------------------
class TestScanStartLeavesProfilesAlone:
"""
Regression: POST /api/scan/start used to call _save_settings() with the
sidebar state, overwriting the active profile's user_ids (dropping
Google-only users) — or profiles[0] when no profile was active.
"""
@pytest.fixture(autouse=True)
def _isolate(self, tmp_path, monkeypatch):
import app_config, scan_engine
monkeypatch.setattr(app_config, "_SETTINGS_PATH", tmp_path / "settings.json")
monkeypatch.setattr(scan_engine, "run_scan", lambda opts: None)
@staticmethod
def _wait_for_lock():
from routes import state
deadline = time.time() + 2.0
while not state._scan_lock.acquire(blocking=False):
assert time.time() < deadline, "scan lock was never released"
time.sleep(0.05)
state._scan_lock.release()
def _save(self, client, name, user_ids):
r = client.post("/api/profiles/save", json={
"name": name, "sources": ["email", "gmail"],
"google_sources": ["gmail"], "user_ids": user_ids, "options": {},
})
return r.get_json()["profile"]
def _get(self, client, pid):
return client.get(f"/api/profiles/get?id={pid}").get_json()["profile"]
def test_active_profile_users_unchanged(self, client, mock_connector):
users = ["m365-a", "google:b@school.dk", "manual:c@school.dk"]
p = self._save(client, "Staff", users)
r = client.post("/api/scan/start", json={
"sources": ["email"],
"user_ids": [{"id": "m365-z", "displayName": "Z"}],
"options": {"older_than_days": 0},
"profile_id": p["id"],
})
assert r.status_code == 200
self._wait_for_lock()
after = self._get(client, p["id"])
assert after["user_ids"] == users
assert after["sources"] == ["email", "gmail"]
def test_no_active_profile_leaves_first_profile_unchanged(self, client, mock_connector):
first = self._save(client, "First", ["m365-a", "m365-b"])
self._save(client, "Second", ["m365-c"])
r = client.post("/api/scan/start", json={
"sources": ["email"],
"user_ids": [{"id": "m365-z", "displayName": "Z"}],
"options": {},
"profile_id": None,
})
assert r.status_code == 200
self._wait_for_lock()
assert self._get(client, first["id"])["user_ids"] == ["m365-a", "m365-b"]
assert len(client.get("/api/profiles").get_json()["profiles"]) == 2
def test_scan_start_creates_no_profile(self, client, mock_connector):
r = client.post("/api/scan/start", json={
"sources": ["email"], "user_ids": [{"id": "m365-z"}], "options": {},
})
assert r.status_code == 200
self._wait_for_lock()
assert client.get("/api/profiles").get_json()["profiles"] == []
+16
View File
@@ -97,6 +97,22 @@ class TestScanStatus:
assert "scan_id" in data
assert data["scan_id"] is None
def test_idle_reports_google_not_running(self, client):
# The refresh/restore path relies on google_running being reported
# separately — running alone misses live Google scans.
data = client.get("/api/scan/status").get_json()
assert data["google_running"] is False
def test_google_lock_held_reports_google_running(self, client):
from routes import state
assert state._google_scan_lock.acquire(blocking=False)
try:
data = client.get("/api/scan/status").get_json()
assert data["google_running"] is True
assert data["running"] is False # M365/file lock still free
finally:
state._google_scan_lock.release()
# ---------------------------------------------------------------------------
# /api/scan/start
+218
View File
@@ -0,0 +1,218 @@
"""
Stop must stop — regressions for "pressing Stop seems to restart the scan".
- SSE replay: a stopped/finished scan's buffered events are never replayed to
a reconnecting browser (that re-showed the Stop button and progress bar).
- run_scan: Stop during collection ends the scan (checkpoint + scan_cancelled)
instead of announcing scan_start / "Scanning…".
- Scheduler: Stop during one stage prevents later stages — in particular the
Google stage no longer clears the Google abort flag and scans everyone.
"""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
import sse
@pytest.fixture(autouse=True)
def clean_scan_state():
"""Restore SSE globals and abort flags so tests don't leak into each other."""
from routes import state
saved_buf = list(sse._sse_buffer)
saved_id = sse._current_scan_id
saved_finished = list(sse._finished_scan_ids)
sse._sse_buffer.clear()
sse._current_scan_id = ""
sse._finished_scan_ids.clear()
yield
state._scan_abort.clear()
state._google_scan_abort.clear()
sse._sse_buffer.clear()
sse._sse_buffer.extend(saved_buf)
sse._current_scan_id = saved_id
sse._finished_scan_ids.clear()
sse._finished_scan_ids.extend(saved_finished)
def _events(msgs: list[str]) -> list[str]:
return [m.split("\n", 1)[0].removeprefix("event: ") for m in msgs]
# ---------------------------------------------------------------------------
# SSE replay
# ---------------------------------------------------------------------------
class TestReplay:
def test_running_scan_is_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
sse.broadcast("scan_progress", {"pct": 10})
assert _events(sse.replay_messages()) == ["scan_phase", "scan_progress"]
def test_stopped_scan_is_not_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
sse.broadcast("scan_start", {"total": 5})
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
assert sse._current_scan_id == ""
assert sse.replay_messages() == []
def test_completed_scan_is_not_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 100})
sse.broadcast("scan_done", {"total_scanned": 1, "flagged_count": 0})
assert sse.replay_messages() == []
def test_file_scan_done_ends_the_file_scan(self):
# Previously only scan_done cleared the id, so every finished file scan
# stayed "current" and was replayed on each page load.
sse._current_scan_id = "filescan_1"
sse.broadcast("scan_phase", {"phase": "Files — Share"})
sse.broadcast("file_scan_done", {"total_scanned": 3, "flagged_count": 0})
assert sse._current_scan_id == ""
assert sse.replay_messages() == []
def test_untagged_events_still_replayed_after_scan_ends(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
sse.broadcast("google_scan_done", {"total_scanned": 7}) # no scan running → untagged
assert _events(sse.replay_messages()) == ["google_scan_done"]
def test_new_scan_replays_only_its_own_events(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
sse._current_scan_id = "scan_2"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
msgs = sse.replay_messages()
assert _events(msgs) == ["scan_phase"]
assert '"scan_id": "scan_2"' in msgs[0]
# ---------------------------------------------------------------------------
# run_scan: Stop during collection
# ---------------------------------------------------------------------------
class TestRunScanStopDuringCollection:
@pytest.fixture()
def engine(self, monkeypatch):
import scan_engine
from routes import state
sent: list[tuple[str, dict]] = []
checkpoints: list = []
monkeypatch.setattr(scan_engine, "broadcast", lambda ev, data: sent.append((ev, data)))
monkeypatch.setattr(scan_engine, "DB_OK", False)
monkeypatch.setattr(scan_engine, "_load_checkpoint", lambda key, **kw: None)
monkeypatch.setattr(scan_engine, "_save_checkpoint", lambda *a, **kw: checkpoints.append(a))
monkeypatch.setattr(scan_engine, "_clear_checkpoint", lambda **kw: None)
conn = MagicMock()
conn.is_app_mode = True
monkeypatch.setattr(state, "connector", conn)
monkeypatch.setattr(state, "flagged_items", [])
opts = {"sources": [], "user_ids": [{"id": "u1", "displayName": "User"}], "options": {}}
return scan_engine, sent, checkpoints, opts
def test_stop_before_processing_does_not_announce_scan_start(self, engine):
scan_engine, sent, checkpoints, opts = engine
from routes import state
state._scan_abort.set()
scan_engine.run_scan(opts)
names = [e for e, _ in sent]
assert "scan_cancelled" in names
assert "scan_start" not in names
assert not any(e == "scan_phase" and "Scanning" in d.get("phase", "") for e, d in sent)
assert "scan_done" not in names
assert len(checkpoints) == 1 # resumable
def test_without_stop_scan_runs_to_done(self, engine):
scan_engine, sent, _, opts = engine
scan_engine.run_scan(opts)
names = [e for e, _ in sent]
assert "scan_start" in names and "scan_done" in names
assert "scan_cancelled" not in names
# ---------------------------------------------------------------------------
# Scheduler: Stop ends the whole job
# ---------------------------------------------------------------------------
class TestSchedulerStop:
JOB = {"id": "j1", "name": "Nightly", "profile_id": "", "enabled": True,
"auto_email": True, "auto_retention": True, "retention_years": 5}
OPTIONS = {"sources": ["email", "gmail"], "user_ids": [], "options": {},
"file_sources": [{"id": "f1", "path": "/srv/share", "label": "Share"}],
"google_sources": ["gmail"]}
@pytest.fixture()
def job(self, monkeypatch):
import gdpr_scanner
import scan_scheduler
import routes.google_scan as google_scan
from routes import state
sent: list[tuple[str, dict]] = []
monkeypatch.setattr(scan_scheduler, "load_jobs", lambda: [dict(self.JOB)])
monkeypatch.setattr(gdpr_scanner, "_get_db", lambda *a, **kw: None)
monkeypatch.setattr(gdpr_scanner, "broadcast", lambda ev, data: sent.append((ev, data)))
conn = MagicMock()
conn.is_authenticated.return_value = True
monkeypatch.setattr(state, "connector", conn)
monkeypatch.setattr(state, "google_connector", MagicMock())
monkeypatch.setattr(state, "flagged_items", [{"id": "found"}])
mocks = {
"run_scan": MagicMock(),
"run_file_scan": MagicMock(),
"google": MagicMock(),
}
monkeypatch.setattr(gdpr_scanner, "run_scan", mocks["run_scan"])
monkeypatch.setattr(gdpr_scanner, "run_file_scan", mocks["run_file_scan"])
monkeypatch.setattr(google_scan, "_run_google_scan", mocks["google"])
sched = scan_scheduler.ScanScheduler()
monkeypatch.setattr(sched, "_build_options", lambda cfg: {
k: (list(v) if isinstance(v, list) else dict(v) if isinstance(v, dict) else v)
for k, v in self.OPTIONS.items()})
mocks["email"] = MagicMock()
mocks["retention"] = MagicMock()
monkeypatch.setattr(sched, "_send_email_report", mocks["email"])
monkeypatch.setattr(sched, "_run_retention", mocks["retention"])
return sched, mocks, sent
def test_stop_during_m365_stage_skips_file_and_google(self, job):
from routes import state
sched, mocks, sent = job
def _user_presses_stop(opts): # what POST /api/scan/stop does mid-scan
state._scan_abort.set()
state._google_scan_abort.set()
mocks["run_scan"].side_effect = _user_presses_stop
sched._execute_scan("j1")
mocks["run_scan"].assert_called_once()
mocks["run_file_scan"].assert_not_called()
mocks["google"].assert_not_called()
assert state._google_scan_abort.is_set(), "Google abort must not be cleared after Stop"
run = sched._last_runs["j1"]
assert run["status"] == "cancelled"
mocks["email"].assert_not_called()
mocks["retention"].assert_not_called()
done = [d for e, d in sent if e == "scheduler_done"]
assert done and done[0]["cancelled"] is True
def test_uninterrupted_job_runs_every_stage(self, job):
sched, mocks, sent = job
sched._execute_scan("j1")
mocks["run_scan"].assert_called_once()
mocks["run_file_scan"].assert_called_once()
mocks["google"].assert_called_once()
assert sched._last_runs["j1"]["status"] == "completed"
mocks["email"].assert_called_once()
mocks["retention"].assert_called_once()
done = [d for e, d in sent if e == "scheduler_done"]
assert done and done[0]["cancelled"] is False
+17
View File
@@ -188,6 +188,23 @@ def test_apply_installs_requirements_when_changed(client, monkeypatch):
assert "pip" in pip_calls[0] and "-r" in pip_calls[0]
# ── Restart fd hygiene ────────────────────────────────────────────────────────
def test_mark_fds_cloexec_unmarks_inheritable_socket():
"""Werkzeug sets the listening socket inheritable; the restart must undo
that or the socket leaks through execv and squats on the port."""
import socket
import routes.updates as upd
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.set_inheritable(True)
assert s.get_inheritable() is True
upd._mark_fds_cloexec()
assert s.get_inheritable() is False
finally:
s.close()
# ── /api/update/settings ──────────────────────────────────────────────────────
def test_settings_roundtrip(client, monkeypatch):