ASSESSMENT METHODOLOGY

How this assessment works

A customer-readable explanation of what we collect, what we check, what is and is not tested, and how you can independently verify the result.

The 6 stages of every assessment

  1. 1. Scope & Authorization

    You provide the customer_id, tenant_domain, and sign a written consent for read-only access. We document what is in scope, what is not, and the deadline for delivery. No work proceeds without an explicit consent record.

  2. 2. App Registration (read-only)

    A one-time setup in your Entra admin center: an App Registration with the scopes User.Read.All, Directory.Read.All, Policy.Read.All, Reports.Read.All, AuditLog.Read.All, DeviceManagement.Read.All. Admin consent is granted to the app — not to SOC Root. SOC Root never receives tenant-wide admin credentials.

  3. 3. Deterministic Collection

    The audit engine calls Microsoft Graph read-only endpoints (organization, users, conditionalAccess, oauth2PermissionGrants, auditLogs/signIn, security/secureScores, deviceManagement). Each call is logged with run_id, timestamp, and a content hash. We never write, modify, or delete anything in your tenant.

  4. 4. Analysis (deterministic)

    Each of the 25 baseline controls is checked against the snapshot. Findings are PASS, FAIL, WARN, or NOT_APPLICABLE. A control marked NOT_APPLICABLE is one where the data was not collectable (e.g. Intune disabled) — it is not a pass-by-default. A coverage % is reported honestly.

  5. 5. Independent QA

    A 10-check deterministic QA gate runs before delivery: schema, score consistency, finding counts, remediation text, N/A disclosure, secret leakage scan, customer identity, score/grade validity, evidence references, and brand consistency. Delivery is BLOCKED on any failure.

  6. 6. Customer-Readable Manifest

    Every audit produces a small `manifest.json` next to the report and annex. It contains: run_id, engagement_id, engine version, config version, score, grade, coverage %, and SHA-256 hashes of the annex and HTML. You (or your auditor) can hash the annex yourself and confirm it matches the manifest.

What goes in and out

What we collect

Read-only Microsoft Graph endpoints listed in the App Registration step above. No screenshots, no third-party data, no browsing history.

What we do NOT collect

Mail content, files, chat messages, user passwords, MFA seeds, conditional access raw values beyond state, audit log bodies. The collector explicitly does not call those endpoints.

What we do NOT modify

Anything. The V1 contract is read-only. capability_executor hard-blocks destructive capabilities. The deliverable manifest always reports destructive_operations=false.

What is deterministic

Collection, control checks, score calculation, severity classification, QA gate, manifest hash binding — all are deterministic code. LLM is not used in the M365 path. llm_used in the manifest is always false.

What a manifest looks like

A small JSON file, one per audit, written next to the report and annex. The customer can hash the annex file themselves and confirm it matchesmanifest.annex_sha256.

{
  "run_id": "20260831_101530_a4f2",
  "engagement_id": "eng_acme-demo_2026Q3_v1",
  "client_id": "acme-demo",
  "engine_version": "1.0",
  "config_version": "v1",
  "input_hash": "8a4f2b...c1d3",
  "annex_sha256": "b3a7f2c1...e9d4",
  "html_sha256":  "7c8b9a01...2f5e",
  "controls_total": 25,
  "controls_assessed": 23,
  "coverage_percent": 92,
  "findings_count": 25,
  "by_severity_fail": {"critical":1,"high":5,"medium":6},
  "score": 67,
  "grade": "D",
  "llm_used": false,
  "destructive_operations": false,
  "assessment_timestamp": "2026-08-31T10:15:30+00:00"
}

A coverage_percent of 92.0 means 23 of 25 controls were actually assessable. The remaining 2 wereNOT_APPLICABLE — for example, an Intune control when the tenant has no Intune subscription. NOT_APPLICABLE is not a pass.

Limitations

An assessment is a point-in-time snapshot. It does not monitor, alert, or remediate. It does not cover every M365 control surface (the V1 baseline is 25 controls; Intune deep-dive, Purview/DLP, and Azure infrastructure are out of scope and require separate engagements).

The score reflects what the audit could verify from the data it was able to collect. A control that cannot be assessed because the customer's App Registration does not have admin consent for that endpoint is reported as NOT_APPLICABLE, not as a pass.

Verify a result yourself

  1. Open the manifest.json file in your delivery.
  2. Hash the *_m365_annex_*.json file with sha256sum on your machine.
  3. Compare to manifest.annex_sha256. They must match.
  4. Cross-check manifest.engagement_id against your task contract.
  5. Cross-check manifest.destructive_operations is false.