3 Commits
Author SHA1 Message Date
StyxX65andClaude Opus 5.5 5df151724a Release 1.7.10: changelog + manual updates
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
- VERSION 1.7.9 -> 1.7.10 (patch: fixes and small behaviour changes only).
- CHANGELOG: 1.7.10 entry covering everything since 1.7.9 — Stop fixes,
  profile/manual-account fixes, delete_message, landing card,
  classification data path, scan-thread crash logging, repo move.
- Manuals (EN/DA): version, Stop -> "Stopping…" and scheduled-job stop,
  scans no longer change profiles, manual accounts in the profile editor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:07:01 +02:00
StyxX65andClaude Opus 5.5 606d8e944b Make Stop stop: no replay, no restart, immediate feedback
Build — Windows, Linux & macOS / GDPRScanner / linux (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / macos (push) Has been cancelled
Build — Windows, Linux & macOS / GDPRScanner / windows (push) Has been cancelled
Build — Windows, Linux & macOS / Create GitHub Release (push) Has been cancelled
Pressing Stop could look like the scan restarting, and for scheduled jobs
it really did restart the Google stage.

- Scheduler: check the abort flag between stages and never clear the
  Google abort after a Stop (that started a fresh all-users Google scan).
  A stopped job is recorded as "cancelled" and skips email/retention;
  scheduler_done carries cancelled=true.
- SSE: scan_cancelled and file_scan_done now end the current scan like
  scan_done, and sse.replay_messages() never replays a finished scan.
  Previously a stopped M365 scan (and every file scan) stayed "current",
  so each reconnect replayed it and re-showed Stop + progress.
- Frontend: scan_cancelled clears the M365 running flag and only tears
  down the UI when no Google/file scan is still running; scheduler_done /
  scheduler_error clear all running flags.
- run_scan: Stop during collection now ends the scan (checkpoint, finish,
  scan_cancelled) instead of announcing scan_start / "Scanning…".
- Stop button: switches to a disabled "Stopping…" state on click; all
  show/hide goes through _setStopBtnDisplay(), which resets it when hidden
  or when a new scan starts.

Adds tests/test_stop_scan.py (10 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:03:12 +02:00
StyxX65andClaude Opus 5.5 9cf23da907 Point docs at git.fejl40.nu after the move from GitHub
- README / CONTRIBUTING: clone from the Gitea repo.
- SECURITY: Gitea has no private security advisories; report to
  support@fejl40.nu instead.
- RELEASING: release URL on Gitea, plus a note that build.yml still uses
  the GitHub-only action-gh-release step, so no releases are produced on
  Gitea yet (server self-update via git is unaffected).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 09:49:06 +02:00
20 changed files with 425 additions and 86 deletions

No files matched your search

+25
View File
@@ -11,6 +11,31 @@ Version numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
--- ---
## [1.7.10] — 2026-09-28
### Changed
- **The Stop button responds immediately** — clicking **Stop** now switches the button to a disabled **Stopping…** state and logs that the scanner is finishing the current item. Previously nothing changed until that item was done (up to a minute for a large PDF being OCR'd), so the scan appeared to ignore the click.
- **Stopping a scheduled scan stops the whole job** — the remaining stages (file sources, Google Workspace) are skipped, the run is recorded as **cancelled** (⏹ in the job history), and no report email or retention run happens for the partial results.
- **Refreshing the page no longer replays the log of a finished or stopped scan** — only a scan that is still running is replayed. Results still load from the database as before.
- **Repository moved to `git.fejl40.nu`** — clone and security-contact details in the documentation now point to the Gitea repository; security reports go to support@fejl40.nu. Server installs that self-update must have their git `origin` pointed at `https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git`.
### Fixed
- **Pressing Stop looked like the scan restarting** — a stopped scan was never marked as finished, so every page load (and the 4-second reconnect check) replayed its events and brought back the Stop button and progress bar. Stopping while accounts were still being collected also announced a fresh "Scanning…" phase before stopping. For scheduled jobs, the Google stage cleared the stop request and genuinely started a new scan of all Workspace users. All three are fixed.
- **Profiles lost their selected accounts** — starting a scan overwrote the selected profile's accounts with whatever was ticked in the sidebar (dropping Google-only users and anyone hidden by the Elev/Ansat filter), and a scan with no profile selected overwrote the *first* profile. Scans no longer change profiles. The profile editor also dropped manually added accounts, and accounts that weren't loaded yet, on the next save, and Google-only accounts weren't ticked when a profile was applied — both fixed.
- **Manually added accounts were never scanned** — accounts added in the sidebar were silently left out of every scan, and accounts added in the profile editor were stored in a form Microsoft 365 couldn't use. The profile editor now looks the account up in Microsoft 365, applying a profile adds its manual accounts to the sidebar, and older profiles are resolved automatically at scan time (unknown accounts are reported and skipped, never widened to all users).
- **Deleting an Outlook message failed** with "'M365Connector' object has no attribute 'delete_message'" — the method had been lost; restored (moves the message to Deleted Items).
- **Results appeared underneath the landing card** — during a scan, results rendered below a still-visible landing/last-scan card until a manual refresh. The card is now hidden as soon as results appear.
- **Licence/OU classification data wasn't found** — the SKU and Google OU role files were read from `classification/` but lived elsewhere, so role classification silently used its fallback. The files are now where the code expects them.
- **Crashes in the background scan thread are logged** — an unexpected error in a scan used to vanish silently; it is now written to the log and the audit log.
### Internal
- Replaced deprecated imports (`fitz` → `pymupdf`, `smbprotocol.query_info` → `smbprotocol.file_info`) ahead of their removal upstream. No behaviour change.
---
## [1.7.9] — 2026-06-22 ## [1.7.9] — 2026-06-22
### Added ### Added
+2 -2
View File
@@ -21,8 +21,8 @@ very welcome.
```bash ```bash
# Clone and set up a virtual environment # Clone and set up a virtual environment
git clone https://github.com/your-org/gdpr-scanner.git git clone https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git
cd gdpr-scanner cd GDPRScanner
python3 -m venv venv python3 -m venv venv
source venv/bin/activate # macOS / Linux source venv/bin/activate # macOS / Linux
venv\Scripts\activate # Windows venv\Scripts\activate # Windows
+2 -2
View File
@@ -635,8 +635,8 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting a pull request.
```bash ```bash
# Quick start for contributors # Quick start for contributors
git clone https://github.com/your-org/gdpr-scanner.git git clone https://git.fejl40.nu/Fejl40.nu/GDPRScanner.git
cd gdpr-scanner cd GDPRScanner
python3 -m venv venv && source venv/bin/activate python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt pip install -r requirements.txt
python gdpr_scanner.py # GDPRScanner on port 5100 (auto-increments if in use) python gdpr_scanner.py # GDPRScanner on port 5100 (auto-increments if in use)
+11 -4
View File
@@ -8,9 +8,16 @@ git commit -m "Your message"
git push origin main git push origin main
``` ```
GitHub Actions builds Windows and Linux binaries automatically and updates the The build workflow (`.github/workflows/build.yml`) builds Windows and Linux
**"Latest build (main)"** pre-release at: binaries and updates the **"Latest build (main)"** pre-release at:
`https://github.com/<owner>/GDPRScanner/releases/tag/latest` `https://git.fejl40.nu/Fejl40.nu/GDPRScanner/releases/tag/latest`
> **Note:** the repository moved from GitHub to Gitea (`git.fejl40.nu`). The
> workflow still publishes with the GitHub-only `softprops/action-gh-release`
> action, so releases are **not yet produced on Gitea**. Until the workflow is
> ported to Gitea Actions, build and attach binaries manually. Server installs
> are unaffected — they self-update from `git.fejl40.nu` via git
> (see `update_gdpr.sh`).
## Versioned release ## Versioned release
@@ -22,7 +29,7 @@ git push origin main
git push origin v1.6.14 git push origin v1.6.14
``` ```
Actions creates a proper GitHub Release named `v1.6.14` with auto-generated Actions creates a proper release named `v1.6.14` with auto-generated
release notes and the built binaries attached. release notes and the built binaries attached.
Append `-beta` or `-rc` to the tag to mark it as a pre-release: Append `-beta` or `-rc` to the tag to mark it as a pre-release:
+2 -3
View File
@@ -12,14 +12,13 @@ We support only the latest release. Please update before reporting a bug.
## Reporting a Vulnerability ## Reporting a Vulnerability
**Please do not file a public GitHub issue for security vulnerabilities.** **Please do not file a public issue for security vulnerabilities.**
This tool processes sensitive personal data including Danish CPR numbers (national This tool processes sensitive personal data including Danish CPR numbers (national
identifiers). Security issues should be reported privately so a fix can be prepared identifiers). Security issues should be reported privately so a fix can be prepared
before public disclosure. before public disclosure.
**Report to:** Open a [GitHub Security Advisory](https://github.com/your-org/gdpr-scanner/security/advisories/new) **Report to:** [support@fejl40.nu](mailto:support@fejl40.nu) — put "Security" in the subject line.
(Settings → Security → Advisories → New draft advisory)
Please include: Please include:
- A description of the vulnerability and its potential impact - A description of the vulnerability and its potential impact
+1 -1
View File
@@ -1 +1 @@
1.7.9 1.7.10
+5 -5
View File
@@ -1,6 +1,6 @@
# GDPR Scanner — Brugermanual # GDPR Scanner — Brugermanual
Version 1.7.9 Version 1.7.10
--- ---
@@ -185,7 +185,7 @@ En statuslinje viser:
Resultater vises i hovedområdet efterhånden som de findes — du behøver ikke vente på, at scanningen er færdig, før du begynder at gennemgå dem. Resultater vises i hovedområdet efterhånden som de findes — du behøver ikke vente på, at scanningen er færdig, før du begynder at gennemgå dem.
Klik på **Stop** for at afbryde. Et kontrolpunkt gemmes automatisk, så du kan fortsætte senere. Klik på **Stop** for at afbryde. Knappen skifter til **Stopper…**, mens scanneren gør det element færdigt, den er i gang med (det kan tage et øjeblik, f.eks. for en stor PDF), hvorefter scanningen stopper. Et kontrolpunkt gemmes automatisk, så du kan fortsætte senere. Stopper du en planlagt scanning, stoppes hele jobbet — resterende kilder springes over, og der sendes ingen rapport-e-mail for de delvise resultater.
### 4.5 Genoptag en afbrudt scanning ### 4.5 Genoptag en afbrudt scanning
@@ -382,7 +382,7 @@ Konfigurer venstre panel præcis som du ønsker det — herunder hvilke M365-kil
Klik på profil-rullemenuen i topbjælken og vælg en profil. Alle indstillinger i venstre panel — kilder, konti, indstillinger og datofilter — indlæses på én gang. Venstre panel viser derefter din aktive tilstand, og du kan justere hvad som helst, inden du scanner. Klik på profil-rullemenuen i topbjælken og vælg en profil. Alle indstillinger i venstre panel — kilder, konti, indstillinger og datofilter — indlæses på én gang. Venstre panel viser derefter din aktive tilstand, og du kan justere hvad som helst, inden du scanner.
En **Ryd**-knap vises ved siden af rullemenuen, når en profil er valgt. Klik på den for at rydde profil­etiketten uden at ændre indstillingerne i venstre panel. Det er nyttigt, når du vil køre en engangsscan uden at overskrive en gemt profil. En **Ryd**-knap vises ved siden af rullemenuen, når en profil er valgt. Klik på den for at rydde profil­etiketten uden at ændre indstillingerne i venstre panel. En scanning ændrer aldrig en gemt profil — profiler ændres kun, når du gemmer dem med **Gem** eller i profileditoren.
### Administrer profiler ### Administrer profiler
@@ -661,7 +661,7 @@ Brug **Art.30**-knappen til at eksportere artikel 30-rapporten. Det er et Word-d
Scanneren klassificerer brugere som ansatte eller elever ud fra deres Microsoft 365-licenstype eller Google Workspace-organisationsenhed. Du kan bruge dette filter i kontolisten til at begrænse en scanning til kun ansatte, kun elever eller en bestemt person. Det er nyttigt, fordi reglerne for behandling af elevdata — særligt for børn under 15 år — adskiller sig fra reglerne for medarbejderdata i henhold til databeskyttelsesloven. Scanneren klassificerer brugere som ansatte eller elever ud fra deres Microsoft 365-licenstype eller Google Workspace-organisationsenhed. Du kan bruge dette filter i kontolisten til at begrænse en scanning til kun ansatte, kun elever eller en bestemt person. Det er nyttigt, fordi reglerne for behandling af elevdata — særligt for børn under 15 år — adskiller sig fra reglerne for medarbejderdata i henhold til databeskyttelsesloven.
**Hvordan tilføjer jeg en konto, der ikke er på listen?** **Hvordan tilføjer jeg en konto, der ikke er på listen?**
I kontoafsnittet i venstre panel er der et felt **+ Tilføj konto manuelt**. Indtast e-mailadressen eller UPN'en, og den tilføjes til den aktuelle sessions kontoliste. I kontoafsnittet i venstre panel er der et felt **+ Tilføj konto manuelt**. Indtast e-mailadressen eller UPN'en, og den tilføjes til den aktuelle sessions kontoliste. Vil du beholde kontoen fast, så tilføj den i profileditoren med **+ Tilføj konto** — scanneren slår den op i Microsoft 365, og kontoen medtages, hver gang profilen bruges, også i planlagte scanninger. Manuelt tilføjede konti skal findes i Microsoft 365.
**Kører scanneren? Jeg kan ikke se en statuslinje.** **Kører scanneren? Jeg kan ikke se en statuslinje.**
Tjek aktivitetsloggen nederst på skærmen. Hvis en scanning kører, vises der beskeder her. Hvis du ikke ser noget, er scanningen muligvis afsluttet eller ikke startet. Kontrollér også, at du har valgt mindst én kilde og mindst én konto. Tjek aktivitetsloggen nederst på skærmen. Hvis en scanning kører, vises der beskeder her. Hvis du ikke ser noget, er scanningen muligvis afsluttet eller ikke startet. Kontrollér også, at du har valgt mindst én kilde og mindst én konto.
@@ -683,4 +683,4 @@ For en typisk skole- eller kommunescanning er omkostningen ubetydelig — Claude
--- ---
*GDPR Scanner v1.7.9 — teknisk opsætning og konfiguration: se README.md* *GDPR Scanner v1.7.10 — teknisk opsætning og konfiguration: se README.md*
+5 -5
View File
@@ -1,6 +1,6 @@
# GDPR Scanner — User Manual # GDPR Scanner — User Manual
Version 1.7.9 Version 1.7.10
--- ---
@@ -185,7 +185,7 @@ A progress bar appears showing:
Results appear in the main area as they are found — you do not need to wait for the scan to finish before reviewing them. Results appear in the main area as they are found — you do not need to wait for the scan to finish before reviewing them.
To stop a scan, click **Stop**. A checkpoint is saved automatically so you can resume later. To stop a scan, click **Stop**. The button changes to **Stopping…** while the scanner finishes the item it is working on (this can take a moment, e.g. for a large PDF), then the scan stops. A checkpoint is saved automatically so you can resume later. Stopping a scheduled scan stops the whole job — any remaining sources are skipped, and no report email is sent for the partial results.
### 4.5 Resuming an Interrupted Scan ### 4.5 Resuming an Interrupted Scan
@@ -382,7 +382,7 @@ Configure the sidebar exactly as you want it — including which M365 sources, G
Click the profile dropdown in the top bar and select a profile. All sidebar settings — sources, accounts, options, and date filter — are loaded at once. The sidebar then shows your live state and you can adjust anything before scanning. Click the profile dropdown in the top bar and select a profile. All sidebar settings — sources, accounts, options, and date filter — are loaded at once. The sidebar then shows your live state and you can adjust anything before scanning.
A **Clear** button appears next to the dropdown after you select a profile. Click it to clear the profile label without changing the sidebar settings. This is useful when you want to run a one-off scan without overwriting a saved profile. A **Clear** button appears next to the dropdown after you select a profile. Click it to clear the profile label without changing the sidebar settings. Running a scan never changes a saved profile — profiles only change when you save them with **Save** or in the profile editor.
### Managing profiles ### Managing profiles
@@ -661,7 +661,7 @@ Use the **Art.30** button to export the Article 30 report. It is a Word document
The scanner classifies users as staff (Ansat) or students (Elev) based on their Microsoft 365 licence type or Google Workspace organisational unit. You can use this filter in the accounts list to restrict a scan to only staff, only students, or a specific individual. This is useful because the rules for processing student data — especially for children under 15 — differ from staff data under Databeskyttelsesloven. The scanner classifies users as staff (Ansat) or students (Elev) based on their Microsoft 365 licence type or Google Workspace organisational unit. You can use this filter in the accounts list to restrict a scan to only staff, only students, or a specific individual. This is useful because the rules for processing student data — especially for children under 15 — differ from staff data under Databeskyttelsesloven.
**How do I add an account that is not in the list?** **How do I add an account that is not in the list?**
In the accounts section of the sidebar, there is an **+ Tilføj konto manuelt** (Add account manually) field. Enter the email address or UPN and it will be added to the current session's account list. In the accounts section of the sidebar, there is an **+ Tilføj konto manuelt** (Add account manually) field. Enter the email address or UPN and it will be added to the current session's account list. To keep an account permanently, add it in the profile editor with **+ Tilføj konto** — the scanner looks it up in Microsoft 365, and the account is included whenever that profile is used, including in scheduled scans. Manually added accounts must exist in Microsoft 365.
**Is the scanner running? I cannot see a progress bar.** **Is the scanner running? I cannot see a progress bar.**
Check the activity log at the bottom of the screen. If a scan is running it will show messages there. If you see nothing, the scan may have completed or not started. Also check that you have at least one source ticked and at least one account selected. Check the activity log at the bottom of the screen. If a scan is running it will show messages there. If you see nothing, the scan may have completed or not started. Also check that you have at least one source ticked and at least one account selected.
@@ -683,4 +683,4 @@ For a typical school or municipality scan the cost is negligible — Claude Haik
--- ---
*GDPR Scanner v1.7.9 — for technical setup and configuration see README.md* *GDPR Scanner v1.7.10 — for technical setup and configuration see README.md*
+3 -10
View File
@@ -1527,16 +1527,9 @@ def local_ip():
def scan_stream(): def scan_stream():
q = queue.Queue(maxsize=512) q = queue.Queue(maxsize=512)
_sse_queues.append(q) _sse_queues.append(q)
# Filter replay buffer: only include events from the current scan # Replay only the running scan's events, never those of a finished or
# (avoids replaying stale events from a previous scan) # stopped scan (see sse.replay_messages)
replay_scan_id = _sse_mod._current_scan_id buf = _sse_mod.replay_messages()
buf = []
if replay_scan_id:
for msg in list(_sse_buffer):
if f'"scan_id": "{replay_scan_id}"' in msg:
buf.append(msg)
else:
buf = list(_sse_buffer)
def generate(): def generate():
try: try:
yield ": connected\n\n" yield ": connected\n\n"
+4
View File
@@ -260,6 +260,10 @@
"m365_admin_note": "Viser kun din konto. For at liste alle brugere skal en administrator give samtykke til <strong>User.Read.All</strong> i Azure Portal, eller tilføj konti manuelt nedenfor.", "m365_admin_note": "Viser kun din konto. For at liste alle brugere skal en administrator give samtykke til <strong>User.Read.All</strong> i Azure Portal, eller tilføj konti manuelt nedenfor.",
"m365_btn_scan": "Scan", "m365_btn_scan": "Scan",
"m365_btn_stop": "Stop", "m365_btn_stop": "Stop",
"m365_btn_stopping": "Stopper…",
"m365_scan_stopping": "Stopper scanning — færdiggør det aktuelle element…",
"m365_scan_stopped": "Scanning stoppet.",
"m365_sched_stopped": "stoppet",
"m365_pill_flagged": "markerede", "m365_pill_flagged": "markerede",
"m365_pill_scanned": "scannet", "m365_pill_scanned": "scannet",
"m365_filter_all_sources": "Alle kilder", "m365_filter_all_sources": "Alle kilder",
+4
View File
@@ -260,6 +260,10 @@
"m365_admin_note": "Es wird nur Ihr Konto angezeigt. Um alle Benutzer aufzulisten, muss ein Administrator <strong>User.Read.All</strong> im Azure Portal genehmigen oder Konten manuell unten hinzufügen.", "m365_admin_note": "Es wird nur Ihr Konto angezeigt. Um alle Benutzer aufzulisten, muss ein Administrator <strong>User.Read.All</strong> im Azure Portal genehmigen oder Konten manuell unten hinzufügen.",
"m365_btn_scan": "Scannen", "m365_btn_scan": "Scannen",
"m365_btn_stop": "Stop", "m365_btn_stop": "Stop",
"m365_btn_stopping": "Wird gestoppt…",
"m365_scan_stopping": "Scan wird gestoppt — aktuelles Element wird abgeschlossen…",
"m365_scan_stopped": "Scan gestoppt.",
"m365_sched_stopped": "gestoppt",
"m365_pill_flagged": "markiert", "m365_pill_flagged": "markiert",
"m365_pill_scanned": "gescannt", "m365_pill_scanned": "gescannt",
"m365_filter_all_sources": "Alle Quellen", "m365_filter_all_sources": "Alle Quellen",
+4
View File
@@ -260,6 +260,10 @@
"m365_admin_note": "Only showing your account. To list all users, an admin must grant <strong>User.Read.All</strong> consent in Azure Portal, or add accounts manually below.", "m365_admin_note": "Only showing your account. To list all users, an admin must grant <strong>User.Read.All</strong> consent in Azure Portal, or add accounts manually below.",
"m365_btn_scan": "Scan", "m365_btn_scan": "Scan",
"m365_btn_stop": "Stop", "m365_btn_stop": "Stop",
"m365_btn_stopping": "Stopping…",
"m365_scan_stopping": "Stopping scan — finishing the current item…",
"m365_scan_stopped": "Scan stopped.",
"m365_sched_stopped": "stopped",
"m365_pill_flagged": "flagged", "m365_pill_flagged": "flagged",
"m365_pill_scanned": "scanned", "m365_pill_scanned": "scanned",
"m365_filter_all_sources": "All sources", "m365_filter_all_sources": "All sources",
+22 -17
View File
@@ -553,12 +553,6 @@ def run_scan(options: dict):
if rem < 3600: return f"{int(rem/60)}m" if rem < 3600: return f"{int(rem/60)}m"
return f"{int(rem/3600)}h" return f"{int(rem/3600)}h"
def _check_abort():
if _state._scan_abort.is_set():
broadcast("scan_cancelled", {"completed": completed})
return True
return False
def _broadcast_card(item_meta: dict, cprs: list, pii_counts: dict | None = None): def _broadcast_card(item_meta: dict, cprs: list, pii_counts: dict | None = None):
card = { card = {
"id": item_meta.get("id", ""), "id": item_meta.get("id", ""),
@@ -1089,6 +1083,26 @@ def run_scan(options: dict):
if scanned_ids: if scanned_ids:
work_items = [(k, m, f) for k, m, f in work_items if m.get("id", "") not in scanned_ids] work_items = [(k, m, f) for k, m, f in work_items if m.get("id", "") not in scanned_ids]
def _finish_aborted(done: int) -> None:
"""Stop requested: checkpoint, finalise the DB scan, announce it."""
# Save checkpoint so scan can be resumed later
_save_checkpoint(ck_key, scanned_ids, _state.flagged_items, _state.scan_meta)
# Finalise the DB scan record so items found before the stop stay
# visible — the early return otherwise skips finish_scan below,
# stranding them (invisible to get_session_items / get_open_items).
if _db and _db_scan_id:
try:
_db.finish_scan(_db_scan_id, done)
except Exception as _e:
logger.error("[db] finish_scan (aborted) failed: %s", _e)
broadcast("scan_cancelled", {"completed": done - resumed_count})
# Stop pressed during collection: the per-user loops just break out, so
# stop here — announcing scan_start/"Scanning…" made Stop look like a restart.
if _state._scan_abort.is_set():
_finish_aborted(resumed_count)
return
total = len(work_items) total = len(work_items)
broadcast("scan_start", { broadcast("scan_start", {
"total": total + resumed_count, "total": total + resumed_count,
@@ -1107,17 +1121,8 @@ def run_scan(options: dict):
_items_since_save = 0 _items_since_save = 0
idx = -1 idx = -1
while _work_q: while _work_q:
if _check_abort(): if _state._scan_abort.is_set():
# Save checkpoint so scan can be resumed later _finish_aborted(resumed_count + idx + 1)
_save_checkpoint(ck_key, scanned_ids, _state.flagged_items, _state.scan_meta)
# Finalise the DB scan record so items found before the stop stay
# visible — this early return otherwise skips finish_scan below,
# stranding them (invisible to get_session_items / get_open_items).
if _db and _db_scan_id:
try:
_db.finish_scan(_db_scan_id, resumed_count + idx + 1)
except Exception as _e:
logger.error("[db] finish_scan (aborted) failed: %s", _e)
return return
idx += 1 idx += 1
kind, meta, _ = _work_q.popleft() # releases this item from the deque immediately kind, meta, _ = _work_q.popleft() # releases this item from the deque immediately
+27 -12
View File
@@ -351,6 +351,9 @@ class ScanScheduler:
) )
_fs_extra = {k: _profile_opts[k] for k in _FS_OPT_KEYS if k in _profile_opts} _fs_extra = {k: _profile_opts[k] for k in _FS_OPT_KEYS if k in _profile_opts}
for fs in options.get("file_sources", []): for fs in options.get("file_sources", []):
# Stop pressed during an earlier stage — don't start more stages
if _m._scan_abort.is_set():
break
# Resolve string IDs to full source dicts # Resolve string IDs to full source dicts
if isinstance(fs, str): if isinstance(fs, str):
fs = _all_file_sources.get(fs, {"path": fs, "label": fs}) fs = _all_file_sources.get(fs, {"path": fs, "label": fs})
@@ -374,7 +377,12 @@ class ScanScheduler:
# Legacy profiles store everything in sources[] # Legacy profiles store everything in sources[]
google_sources = [s for s in options.get("sources", []) google_sources = [s for s in options.get("sources", [])
if s in ("gmail", "gdrive")] if s in ("gmail", "gdrive")]
if google_sources and state.google_connector: # POST /api/scan/stop sets both abort events. _scan_abort was cleared
# when this job started, so if it is set now the user pressed Stop
# during an earlier stage — the Google stage must not start, and its
# abort event must not be cleared (that turned Stop into a restart).
stopped = _m._scan_abort.is_set()
if google_sources and state.google_connector and not stopped:
from routes.google_scan import ( from routes.google_scan import (
_run_google_scan as _rgs, _run_google_scan as _rgs,
_scan_lock as _gsl, _scan_lock as _gsl,
@@ -383,12 +391,15 @@ class ScanScheduler:
if _gsl.acquire(blocking=False): if _gsl.acquire(blocking=False):
try: try:
_gsa.clear() _gsa.clear()
logger.info("[scheduler] Starting Google scan — sources=%s", google_sources) # Stop sets _scan_abort first, so re-check after clearing:
_rgs({ # a Stop landing in between must still win.
"sources": google_sources, if not _m._scan_abort.is_set():
"user_emails": [], # empty → scan all workspace users logger.info("[scheduler] Starting Google scan — sources=%s", google_sources)
"options": options.get("options", {}), _rgs({
}) "sources": google_sources,
"user_emails": [], # empty → scan all workspace users
"options": options.get("options", {}),
})
except Exception as _ge: except Exception as _ge:
import traceback as _tb3 import traceback as _tb3
logger.error("[scheduler] Google scan error: %s\n%s", _ge, _tb3.format_exc()) logger.error("[scheduler] Google scan error: %s\n%s", _ge, _tb3.format_exc())
@@ -399,18 +410,21 @@ class ScanScheduler:
run["flagged"] = len(_m.flagged_items) run["flagged"] = len(_m.flagged_items)
run["scanned"] = _m.scan_meta.get("total_scanned", 0) run["scanned"] = _m.scan_meta.get("total_scanned", 0)
run["status"] = "completed" stopped = _m._scan_abort.is_set()
logger.info("[scheduler] Job '%s' completed — %d flagged, %d scanned", run["status"] = "cancelled" if stopped else "completed"
job_cfg.get("name", ""), run["flagged"], run["scanned"]) logger.info("[scheduler] Job '%s' %s — %d flagged, %d scanned",
job_cfg.get("name", ""), run["status"], run["flagged"], run["scanned"])
if job_cfg.get("auto_email") and state.flagged_items: # A stopped job's results are partial — don't email them or apply
# retention based on them.
if not stopped and job_cfg.get("auto_email") and state.flagged_items:
try: try:
self._send_email_report(job_cfg) self._send_email_report(job_cfg)
run["emailed"] = 1 run["emailed"] = 1
except Exception as e: except Exception as e:
run["error"] = f"Scan OK, email failed: {e}" run["error"] = f"Scan OK, email failed: {e}"
if job_cfg.get("auto_retention") and job_cfg.get("retention_years"): if not stopped and job_cfg.get("auto_retention") and job_cfg.get("retention_years"):
try: try:
self._run_retention(job_cfg) self._run_retention(job_cfg)
except Exception as e: except Exception as e:
@@ -420,6 +434,7 @@ class ScanScheduler:
_m.broadcast("scheduler_done", { _m.broadcast("scheduler_done", {
"flagged": run["flagged"], "scanned": run["scanned"], "flagged": run["flagged"], "scanned": run["scanned"],
"emailed": run["emailed"], "job_name": job_cfg.get("name", ""), "emailed": run["emailed"], "job_name": job_cfg.get("name", ""),
"cancelled": stopped,
}) })
except Exception as e: except Exception as e:
+26 -2
View File
@@ -19,6 +19,29 @@ logger = logging.getLogger(__name__)
_sse_queues: list = [] _sse_queues: list = []
_sse_buffer: deque = deque(maxlen=500) _sse_buffer: deque = deque(maxlen=500)
_current_scan_id: str = "" _current_scan_id: str = ""
# Scans that have ended (done / stopped). Their events stay in the buffer but
# are never replayed: replaying a stopped scan re-showed the Stop button and
# progress bar on every reconnect, so Stop looked like a restart.
_finished_scan_ids: deque = deque(maxlen=20)
# Events that end the scan identified by _current_scan_id.
# scan_done: M365 finished; scan_cancelled: M365 stopped; file_scan_done:
# file scan finished or stopped. (Google scans don't set _current_scan_id.)
_TERMINAL_EVENTS = frozenset({"scan_done", "scan_cancelled", "file_scan_done"})
def replay_messages() -> list[str]:
"""Buffered messages to replay to a newly connected browser.
While a scan runs, only that scan's events. Otherwise everything except
events of scans that have already ended (done or stopped).
"""
buf = list(_sse_buffer)
if _current_scan_id:
tag = f'"scan_id": "{_current_scan_id}"'
return [m for m in buf if tag in m]
finished = [f'"scan_id": "{sid}"' for sid in _finished_scan_ids]
return [m for m in buf if not any(tag in m for tag in finished)]
def broadcast(event: str, data: dict): def broadcast(event: str, data: dict):
global _current_scan_id global _current_scan_id
@@ -31,8 +54,9 @@ def broadcast(event: str, data: dict):
q.put_nowait(msg) q.put_nowait(msg)
except queue.Full: except queue.Full:
pass pass
# Clear scan_id after scan_done so replay knows the scan is finished # Clear scan_id when the scan ends so replay knows it is finished
if event == "scan_done" and _current_scan_id: if event in _TERMINAL_EVENTS and _current_scan_id:
_finished_scan_ids.append(_current_scan_id)
_current_scan_id = "" _current_scan_id = ""
# When no browser is watching (e.g. scheduled scan), log key events # When no browser is watching (e.g. scheduled scan), log key events
if not _sse_queues: if not _sse_queues:
+5
View File
@@ -34,6 +34,11 @@ Never revert to `!!window._googleConnected` / `_fileSources.length > 0` — thos
- **Do not close `S.es` in `scan_done` if other scans are still running** — M365 (`scan_done`), Google (`google_scan_done`), and File (`file_scan_done`) each emit their own done event. Close `S.es` only when all concurrent scans have finished: `scan_done` checks `!S._googleScanRunning && !S._fileScanRunning`; `google_scan_done` checks `!S._m365ScanRunning && !S._fileScanRunning`; `file_scan_done` checks `!S._m365ScanRunning && !S._googleScanRunning`. - **Do not close `S.es` in `scan_done` if other scans are still running** — M365 (`scan_done`), Google (`google_scan_done`), and File (`file_scan_done`) each emit their own done event. Close `S.es` only when all concurrent scans have finished: `scan_done` checks `!S._googleScanRunning && !S._fileScanRunning`; `google_scan_done` checks `!S._m365ScanRunning && !S._fileScanRunning`; `file_scan_done` checks `!S._m365ScanRunning && !S._googleScanRunning`.
- **Scheduled scans** — `S._userStartedScan` is false for scheduler-triggered runs, so SSE is never closed and future scheduler events continue to arrive. - **Scheduled scans** — `S._userStartedScan` is false for scheduler-triggered runs, so SSE is never closed and future scheduler events continue to arrive.
- **Two separate abort events** — `state._scan_abort` (M365 + file) and `state._google_scan_abort` (Google). `POST /api/scan/stop` sets **both**. `_check_abort()` inside `_run_google_scan` must use the module-level `_scan_abort` alias (`= state._google_scan_abort`), not `gdpr_scanner._scan_abort`. - **Two separate abort events** — `state._scan_abort` (M365 + file) and `state._google_scan_abort` (Google). `POST /api/scan/stop` sets **both**. `_check_abort()` inside `_run_google_scan` must use the module-level `_scan_abort` alias (`= state._google_scan_abort`), not `gdpr_scanner._scan_abort`.
- **Stop must never look like a restart** — four rules, each fixing a way Stop used to appear to restart the scan:
- `sse.replay_messages()` never replays a scan that has ended. `scan_done`, `scan_cancelled` and `file_scan_done` move `_current_scan_id` into `_finished_scan_ids`. Do not go back to clearing the id only on `scan_done`: stopped M365 scans and all file scans stayed "current", and every reconnect (page load, watchdog) replayed them, bringing back the Stop button and progress bar.
- The `scan_cancelled` handler clears `S._m365ScanRunning` and, like `scan_done`, keeps the Stop button and SSE only while a Google or file scan is still running. `scheduler_done` and `scheduler_error` clear all three running flags.
- The Stop button goes through `_setStopBtnDisplay(display, fresh)`; do not set `stopBtn.style.display` directly. `stopScan()` switches it to a disabled "Stopping…" state. Hiding it, or passing `fresh=true` (from `startScan()` and `scheduler_started`), resets it.
- Server side: `run_scan` checks for Stop straight after collection, before `scan_start`. The scheduler checks `_scan_abort` between stages and never clears the Google abort flag after a Stop. A stopped job is recorded as `cancelled`, with no email or retention.
- **`_check_abort()` emits `google_scan_done`, not `scan_cancelled`** — `scan_cancelled` unconditionally closes the SSE; `google_scan_done` checks whether other scans are still running before closing. - **`_check_abort()` emits `google_scan_done`, not `scan_cancelled`** — `scan_cancelled` unconditionally closes the SSE; `google_scan_done` checks whether other scans are still running before closing.
- **`scan_phase` replay sets running flags — handled by `sse_replay_done`** — the `scan_phase` handler sets running flags to `true` whenever all flags are `false` and a source keyword is found in the phase text. On page refresh this fires during SSE replay of a completed scan, temporarily making the scan appear running. The `sse_replay_done` handler retries `loadHistorySession(null)` if no scan is running and `S._historyRefScanId` is still `null` after replay. Do not remove either the flag-setting logic or the retry. - **`scan_phase` replay sets running flags — handled by `sse_replay_done`** — the `scan_phase` handler sets running flags to `true` whenever all flags are `false` and a source keyword is found in the phase text. On page refresh this fires during SSE replay of a completed scan, temporarily making the scan appear running. The `sse_replay_done` handler retries `loadHistorySession(null)` if no scan is running and `S._historyRefScanId` is still `null` after replay. Do not remove either the flag-setting logic or the retry.
- **Google Drive uses a lazy generator, not `list()`** — `iter_drive_files()` iterated directly so `_check_abort()` fires between items. Wrapping in `list()` blocks the thread for the entire enumeration. - **Google Drive uses a lazy generator, not `list()`** — `iter_drive_files()` iterated directly so `_check_abort()` fires between items. Wrapping in `list()` blocks the thread for the entire enumeration.
+1 -1
View File
@@ -779,7 +779,7 @@ function _sseWatchdog() {
_ensureSSE(); _ensureSSE();
if (status.running && !S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) { if (status.running && !S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) {
document.getElementById('scanBtn').disabled = true; document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block'; _setStopBtnDisplay('inline-block');
// status.running reflects the M365 + file lock; treat as an M365 reconnect // status.running reflects the M365 + file lock; treat as an M365 reconnect
S._m365ScanRunning = true; _renderProgressSegments(); S._m365ScanRunning = true; _renderProgressSegments();
document.getElementById('progressFile').textContent = t('m365_sse_reconnecting', 'Reconnecting to running scan…'); document.getElementById('progressFile').textContent = t('m365_sse_reconnecting', 'Reconnecting to running scan…');
+56 -21
View File
@@ -317,7 +317,7 @@ function _attachScanListeners(source) {
else if (phaseSrc === 'file') { S._fileScanRunning = true; } else if (phaseSrc === 'file') { S._fileScanRunning = true; }
else { S._m365ScanRunning = true; } else { S._m365ScanRunning = true; }
document.getElementById('scanBtn').disabled = true; document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block'; _setStopBtnDisplay('inline-block');
_renderProgressSegments(); _renderProgressSegments();
} }
_setProgressPhase(d.phase); _setProgressPhase(d.phase);
@@ -330,9 +330,9 @@ function _attachScanListeners(source) {
: (d.total > 0 ? Math.round((d.index || d.completed || 0) / d.total * 100) : 0); : (d.total > 0 ? Math.round((d.index || d.completed || 0) / d.total * 100) : 0);
S._srcPct[src] = pct; S._srcPct[src] = pct;
// If reconnecting mid-scan the running flag may not be set yet — ensure segment exists // If reconnecting mid-scan the running flag may not be set yet — ensure segment exists
if (src === 'm365' && !S._m365ScanRunning) { S._m365ScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); } if (src === 'm365' && !S._m365ScanRunning) { S._m365ScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
if (src === 'google' && !S._googleScanRunning) { S._googleScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); } if (src === 'google' && !S._googleScanRunning) { S._googleScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
if (src === 'file' && !S._fileScanRunning) { S._fileScanRunning = true; document.getElementById('scanBtn').disabled = true; document.getElementById('stopBtn').style.display = 'inline-block'; _renderProgressSegments(); } if (src === 'file' && !S._fileScanRunning) { S._fileScanRunning = true; document.getElementById('scanBtn').disabled = true; _setStopBtnDisplay('inline-block'); _renderProgressSegments(); }
var fill = document.getElementById('progressFill_' + src); var fill = document.getElementById('progressFill_' + src);
if (fill) fill.style.width = pct + '%'; if (fill) fill.style.width = pct + '%';
document.getElementById('progressFile').textContent = d.file || ''; document.getElementById('progressFile').textContent = d.file || '';
@@ -368,15 +368,20 @@ function _attachScanListeners(source) {
log((d.file ? d.file + ': ' : '') + d.error, 'err'); log((d.file ? d.file + ': ' : '') + d.error, 'err');
}); });
source.addEventListener('scan_cancelled', function() { source.addEventListener('scan_cancelled', function() {
if (S._userStartedScan) { // scan_cancelled is the M365 engine's "stopped" event. Clear its running
// flag (leaving it set made later replays/status polls treat the stopped
// scan as live) and, like scan_done, only tear down the UI once no Google
// or file scan is still running.
S._m365ScanRunning = false;
var _anyRunning = S._googleScanRunning || S._fileScanRunning;
if (S._userStartedScan && !_anyRunning) {
S._userStartedScan = false; S._userStartedScan = false;
if (S.es) { S.es.close(); S.es = null; } if (S.es) { S.es.close(); S.es = null; }
} }
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = _anyRunning;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay(_anyRunning ? 'inline-block' : 'none');
_clearProgressBar(); if (_anyRunning) { _renderProgressSegments(); } else { _clearProgressBar(); setLogLive(''); }
setLogLive(''); log(t('m365_scan_stopped', 'Scan stopped.'), 'warn');
log('Scan stopped.', 'warn');
}); });
source.addEventListener('scan_done', function(e) { source.addEventListener('scan_done', function(e) {
var d = JSON.parse(e.data); var d = JSON.parse(e.data);
@@ -401,7 +406,7 @@ function _attachScanListeners(source) {
} }
if (!_anyRunning) setLogLive(''); if (!_anyRunning) setLogLive('');
document.getElementById('scanBtn').disabled = _anyRunning; document.getElementById('scanBtn').disabled = _anyRunning;
document.getElementById('stopBtn').style.display = _anyRunning ? 'inline-block' : 'none'; _setStopBtnDisplay(_anyRunning ? 'inline-block' : 'none');
if (!_anyRunning) _clearProgressBar(); if (!_anyRunning) _clearProgressBar();
document.getElementById('statsSection').style.display = 'block'; document.getElementById('statsSection').style.display = 'block';
document.getElementById('statScanned').textContent = d.total_scanned; document.getElementById('statScanned').textContent = d.total_scanned;
@@ -437,7 +442,7 @@ function _attachScanListeners(source) {
} }
setLogLive(''); setLogLive('');
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay('none');
_clearProgressBar(); _clearProgressBar();
document.getElementById('statsSection').style.display = 'block'; document.getElementById('statsSection').style.display = 'block';
document.getElementById('statsPill').style.display = 'block'; document.getElementById('statsPill').style.display = 'block';
@@ -466,7 +471,7 @@ function _attachScanListeners(source) {
} }
setLogLive(''); setLogLive('');
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay('none');
_clearProgressBar(); _clearProgressBar();
document.getElementById('statsSection').style.display = 'block'; document.getElementById('statsSection').style.display = 'block';
document.getElementById('statsPill').style.display = 'block'; document.getElementById('statsPill').style.display = 'block';
@@ -502,7 +507,7 @@ function _attachSchedulerListeners(source) {
log('\uD83D\uDD50 ' + t('m365_sched_title','Scheduled scan') + ': ' + (d.job_name||'') + '\u2026'); log('\uD83D\uDD50 ' + t('m365_sched_title','Scheduled scan') + ': ' + (d.job_name||'') + '\u2026');
// Show progress UI so scan_phase / scan_progress events are visible // Show progress UI so scan_phase / scan_progress events are visible
document.getElementById('scanBtn').disabled = true; document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block'; _setStopBtnDisplay('inline-block', true);
S._srcPct = { m365: 0, google: 0, file: 0 }; S._m365ScanRunning = true; _renderProgressSegments(); S._srcPct = { m365: 0, google: 0, file: 0 }; S._m365ScanRunning = true; _renderProgressSegments();
_setProgressPhase((d.job_name||'') + '\u2026'); _setProgressPhase((d.job_name||'') + '\u2026');
document.getElementById('progressFile').textContent = ''; document.getElementById('progressFile').textContent = '';
@@ -512,25 +517,32 @@ function _attachSchedulerListeners(source) {
// scheduler_started was missed (e.g. browser reconnected mid-scan) // scheduler_started was missed (e.g. browser reconnected mid-scan)
console.log('[SSE] scan_start received'); console.log('[SSE] scan_start received');
document.getElementById('scanBtn').disabled = true; document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block'; _setStopBtnDisplay('inline-block');
// Ensure at least the M365 segment is rendered (scan_start is M365-only) // Ensure at least the M365 segment is rendered (scan_start is M365-only)
if (!S._m365ScanRunning) { S._m365ScanRunning = true; _renderProgressSegments(); } if (!S._m365ScanRunning) { S._m365ScanRunning = true; _renderProgressSegments(); }
}); });
source.addEventListener('scheduler_done', function(e) { source.addEventListener('scheduler_done', function(e) {
var d = JSON.parse(e.data); var d = JSON.parse(e.data);
console.log('[SSE] scheduler_done received:', d); console.log('[SSE] scheduler_done received:', d);
// The job's stages are all over — scheduler_started set the M365 flag
S._m365ScanRunning = S._googleScanRunning = S._fileScanRunning = false;
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay('none');
_clearProgressBar(); _clearProgressBar();
log('\u2713 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + (d.flagged||0) + ' flagged', 'ok'); if (d.cancelled) {
log('\u23f9 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + t('m365_sched_stopped','stopped') + ' (' + (d.flagged||0) + ' flagged)', 'warn');
} else {
log('\u2713 ' + t('m365_sched_title','Scheduled scan') + ' ' + (d.job_name||'') + ' \u2014 ' + (d.flagged||0) + ' flagged', 'ok');
}
markOverdueCards(); markOverdueCards();
loadTrend(); loadTrend();
}); });
source.addEventListener('scheduler_error', function(e) { source.addEventListener('scheduler_error', function(e) {
var d = JSON.parse(e.data); var d = JSON.parse(e.data);
console.log('[SSE] scheduler_error received:', d); console.log('[SSE] scheduler_error received:', d);
S._m365ScanRunning = S._googleScanRunning = S._fileScanRunning = false;
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay('none');
_clearProgressBar(); _clearProgressBar();
log('\u26A0 ' + t('m365_sched_title','Scheduled scan') + ' failed: ' + (d.error||''), 'err'); log('\u26A0 ' + t('m365_sched_title','Scheduled scan') + ' failed: ' + (d.error||''), 'err');
}); });
@@ -564,7 +576,7 @@ function startScan(resume) {
S._progressCurrentUser = ''; S._progressCurrentUser = '';
_renderProgressSegments(); _renderProgressSegments();
document.getElementById('scanBtn').disabled = true; document.getElementById('scanBtn').disabled = true;
document.getElementById('stopBtn').style.display = 'inline-block'; _setStopBtnDisplay('inline-block', true);
// progress segments rendered by _renderProgressSegments() called above // progress segments rendered by _renderProgressSegments() called above
document.getElementById('progressFile').textContent = ''; document.getElementById('progressFile').textContent = '';
_setProgressPhase(t('scan_preparing', 'Preparing…')); _setProgressPhase(t('scan_preparing', 'Preparing…'));
@@ -594,7 +606,7 @@ function startScan(resume) {
if (label === 'google') S._googleScanRunning = false; if (label === 'google') S._googleScanRunning = false;
if (!S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) { if (!S._m365ScanRunning && !S._googleScanRunning && !S._fileScanRunning) {
document.getElementById('scanBtn').disabled = false; document.getElementById('scanBtn').disabled = false;
document.getElementById('stopBtn').style.display = 'none'; _setStopBtnDisplay('none');
if (S.es) { S.es.close(); S.es = null; } if (S.es) { S.es.close(); S.es = null; }
S._userStartedScan = false; S._userStartedScan = false;
} }
@@ -664,8 +676,30 @@ function startScan(resume) {
} }
// Show/hide the Stop button. Hiding — or passing fresh=true when a new scan
// starts — resets it from its "Stopping…" state, so a new scan always gets a
// normal, clickable Stop button even if a previous scan never reported its end.
function _setStopBtnDisplay(display, fresh) {
const btn = document.getElementById('stopBtn');
if (!btn) return;
btn.style.display = display;
if (display === 'none' || fresh) {
btn.disabled = false;
btn.textContent = t('m365_btn_stop', 'Stop');
}
}
function stopScan() { function stopScan() {
fetch('/api/scan/stop', {method:'POST'}); // Immediate feedback: the scan only notices the stop between items, which
// can take a while (e.g. OCR on a large PDF). Stays "Stopping…" until the
// scan's done/stopped event hides the button.
const btn = document.getElementById('stopBtn');
if (btn) { btn.disabled = true; btn.textContent = t('m365_btn_stopping', 'Stopping…'); }
log(t('m365_scan_stopping', 'Stopping scan — finishing the current item…'), 'warn');
fetch('/api/scan/stop', {method:'POST'}).catch(function(e) {
if (btn) { btn.disabled = false; btn.textContent = t('m365_btn_stop', 'Stop'); }
log('Stop failed: ' + e, 'err');
});
} }
// ── Trend sparkline (#7) ────────────────────────────────────────────────────── // ── Trend sparkline (#7) ──────────────────────────────────────────────────────
@@ -795,6 +829,7 @@ window._attachScanListeners = _attachScanListeners;
window._attachSchedulerListeners = _attachSchedulerListeners; window._attachSchedulerListeners = _attachSchedulerListeners;
window.startScan = startScan; window.startScan = startScan;
window.stopScan = stopScan; window.stopScan = stopScan;
window._setStopBtnDisplay = _setStopBtnDisplay;
window.drawSparkline = drawSparkline; window.drawSparkline = drawSparkline;
window.loadTrend = loadTrend; window.loadTrend = loadTrend;
window.updateStats = updateStats; window.updateStats = updateStats;
+2 -1
View File
@@ -242,7 +242,8 @@ function schedLoadHistory() {
var html = ''; var html = '';
runs.forEach(function(r) { runs.forEach(function(r) {
var ts = r.started_at ? new Date(r.started_at * 1000).toLocaleString() : '-'; var ts = r.started_at ? new Date(r.started_at * 1000).toLocaleString() : '-';
var icon = r.status === 'completed' ? '\u2713' : r.status === 'failed' ? '\u2716' : '\u23f3'; var icon = r.status === 'completed' ? '\u2713' : r.status === 'failed' ? '\u2716'
: r.status === 'cancelled' ? '\u23f9' : '\u23f3';
var jname = r.job_name ? '<strong>' + _esc(r.job_name) + '</strong> - ' : ''; var jname = r.job_name ? '<strong>' + _esc(r.job_name) + '</strong> - ' : '';
html += icon + ' ' + jname + ts + ' - ' + (r.flagged||0) + ' flagged'; html += icon + ' ' + jname + ts + ' - ' + (r.flagged||0) + ' flagged';
if (r.emailed) html += ' \u2709'; if (r.emailed) html += ' \u2709';
+218
View File
@@ -0,0 +1,218 @@
"""
Stop must stop — regressions for "pressing Stop seems to restart the scan".
- SSE replay: a stopped/finished scan's buffered events are never replayed to
a reconnecting browser (that re-showed the Stop button and progress bar).
- run_scan: Stop during collection ends the scan (checkpoint + scan_cancelled)
instead of announcing scan_start / "Scanning…".
- Scheduler: Stop during one stage prevents later stages — in particular the
Google stage no longer clears the Google abort flag and scans everyone.
"""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
import sse
@pytest.fixture(autouse=True)
def clean_scan_state():
"""Restore SSE globals and abort flags so tests don't leak into each other."""
from routes import state
saved_buf = list(sse._sse_buffer)
saved_id = sse._current_scan_id
saved_finished = list(sse._finished_scan_ids)
sse._sse_buffer.clear()
sse._current_scan_id = ""
sse._finished_scan_ids.clear()
yield
state._scan_abort.clear()
state._google_scan_abort.clear()
sse._sse_buffer.clear()
sse._sse_buffer.extend(saved_buf)
sse._current_scan_id = saved_id
sse._finished_scan_ids.clear()
sse._finished_scan_ids.extend(saved_finished)
def _events(msgs: list[str]) -> list[str]:
return [m.split("\n", 1)[0].removeprefix("event: ") for m in msgs]
# ---------------------------------------------------------------------------
# SSE replay
# ---------------------------------------------------------------------------
class TestReplay:
def test_running_scan_is_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
sse.broadcast("scan_progress", {"pct": 10})
assert _events(sse.replay_messages()) == ["scan_phase", "scan_progress"]
def test_stopped_scan_is_not_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
sse.broadcast("scan_start", {"total": 5})
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
assert sse._current_scan_id == ""
assert sse.replay_messages() == []
def test_completed_scan_is_not_replayed(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 100})
sse.broadcast("scan_done", {"total_scanned": 1, "flagged_count": 0})
assert sse.replay_messages() == []
def test_file_scan_done_ends_the_file_scan(self):
# Previously only scan_done cleared the id, so every finished file scan
# stayed "current" and was replayed on each page load.
sse._current_scan_id = "filescan_1"
sse.broadcast("scan_phase", {"phase": "Files — Share"})
sse.broadcast("file_scan_done", {"total_scanned": 3, "flagged_count": 0})
assert sse._current_scan_id == ""
assert sse.replay_messages() == []
def test_untagged_events_still_replayed_after_scan_ends(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
sse.broadcast("google_scan_done", {"total_scanned": 7}) # no scan running → untagged
assert _events(sse.replay_messages()) == ["google_scan_done"]
def test_new_scan_replays_only_its_own_events(self):
sse._current_scan_id = "scan_1"
sse.broadcast("scan_progress", {"pct": 40})
sse.broadcast("scan_cancelled", {"completed": 2})
sse._current_scan_id = "scan_2"
sse.broadcast("scan_phase", {"phase": "Collecting…"})
msgs = sse.replay_messages()
assert _events(msgs) == ["scan_phase"]
assert '"scan_id": "scan_2"' in msgs[0]
# ---------------------------------------------------------------------------
# run_scan: Stop during collection
# ---------------------------------------------------------------------------
class TestRunScanStopDuringCollection:
@pytest.fixture()
def engine(self, monkeypatch):
import scan_engine
from routes import state
sent: list[tuple[str, dict]] = []
checkpoints: list = []
monkeypatch.setattr(scan_engine, "broadcast", lambda ev, data: sent.append((ev, data)))
monkeypatch.setattr(scan_engine, "DB_OK", False)
monkeypatch.setattr(scan_engine, "_load_checkpoint", lambda key, **kw: None)
monkeypatch.setattr(scan_engine, "_save_checkpoint", lambda *a, **kw: checkpoints.append(a))
monkeypatch.setattr(scan_engine, "_clear_checkpoint", lambda **kw: None)
conn = MagicMock()
conn.is_app_mode = True
monkeypatch.setattr(state, "connector", conn)
monkeypatch.setattr(state, "flagged_items", [])
opts = {"sources": [], "user_ids": [{"id": "u1", "displayName": "User"}], "options": {}}
return scan_engine, sent, checkpoints, opts
def test_stop_before_processing_does_not_announce_scan_start(self, engine):
scan_engine, sent, checkpoints, opts = engine
from routes import state
state._scan_abort.set()
scan_engine.run_scan(opts)
names = [e for e, _ in sent]
assert "scan_cancelled" in names
assert "scan_start" not in names
assert not any(e == "scan_phase" and "Scanning" in d.get("phase", "") for e, d in sent)
assert "scan_done" not in names
assert len(checkpoints) == 1 # resumable
def test_without_stop_scan_runs_to_done(self, engine):
scan_engine, sent, _, opts = engine
scan_engine.run_scan(opts)
names = [e for e, _ in sent]
assert "scan_start" in names and "scan_done" in names
assert "scan_cancelled" not in names
# ---------------------------------------------------------------------------
# Scheduler: Stop ends the whole job
# ---------------------------------------------------------------------------
class TestSchedulerStop:
JOB = {"id": "j1", "name": "Nightly", "profile_id": "", "enabled": True,
"auto_email": True, "auto_retention": True, "retention_years": 5}
OPTIONS = {"sources": ["email", "gmail"], "user_ids": [], "options": {},
"file_sources": [{"id": "f1", "path": "/srv/share", "label": "Share"}],
"google_sources": ["gmail"]}
@pytest.fixture()
def job(self, monkeypatch):
import gdpr_scanner
import scan_scheduler
import routes.google_scan as google_scan
from routes import state
sent: list[tuple[str, dict]] = []
monkeypatch.setattr(scan_scheduler, "load_jobs", lambda: [dict(self.JOB)])
monkeypatch.setattr(gdpr_scanner, "_get_db", lambda *a, **kw: None)
monkeypatch.setattr(gdpr_scanner, "broadcast", lambda ev, data: sent.append((ev, data)))
conn = MagicMock()
conn.is_authenticated.return_value = True
monkeypatch.setattr(state, "connector", conn)
monkeypatch.setattr(state, "google_connector", MagicMock())
monkeypatch.setattr(state, "flagged_items", [{"id": "found"}])
mocks = {
"run_scan": MagicMock(),
"run_file_scan": MagicMock(),
"google": MagicMock(),
}
monkeypatch.setattr(gdpr_scanner, "run_scan", mocks["run_scan"])
monkeypatch.setattr(gdpr_scanner, "run_file_scan", mocks["run_file_scan"])
monkeypatch.setattr(google_scan, "_run_google_scan", mocks["google"])
sched = scan_scheduler.ScanScheduler()
monkeypatch.setattr(sched, "_build_options", lambda cfg: {
k: (list(v) if isinstance(v, list) else dict(v) if isinstance(v, dict) else v)
for k, v in self.OPTIONS.items()})
mocks["email"] = MagicMock()
mocks["retention"] = MagicMock()
monkeypatch.setattr(sched, "_send_email_report", mocks["email"])
monkeypatch.setattr(sched, "_run_retention", mocks["retention"])
return sched, mocks, sent
def test_stop_during_m365_stage_skips_file_and_google(self, job):
from routes import state
sched, mocks, sent = job
def _user_presses_stop(opts): # what POST /api/scan/stop does mid-scan
state._scan_abort.set()
state._google_scan_abort.set()
mocks["run_scan"].side_effect = _user_presses_stop
sched._execute_scan("j1")
mocks["run_scan"].assert_called_once()
mocks["run_file_scan"].assert_not_called()
mocks["google"].assert_not_called()
assert state._google_scan_abort.is_set(), "Google abort must not be cleared after Stop"
run = sched._last_runs["j1"]
assert run["status"] == "cancelled"
mocks["email"].assert_not_called()
mocks["retention"].assert_not_called()
done = [d for e, d in sent if e == "scheduler_done"]
assert done and done[0]["cancelled"] is True
def test_uninterrupted_job_runs_every_stage(self, job):
sched, mocks, sent = job
sched._execute_scan("j1")
mocks["run_scan"].assert_called_once()
mocks["run_file_scan"].assert_called_once()
mocks["google"].assert_called_once()
assert sched._last_runs["j1"]["status"] == "completed"
mocks["email"].assert_called_once()
mocks["retention"].assert_called_once()
done = [d for e, d in sent if e == "scheduler_done"]
assert done and done[0]["cancelled"] is False