When a large customer runs a vendor security assessment, the security team usually asks for five kinds of login evidence after the questionnaire is answered: a record of who signed in, when and from where, proof that access is removed when someone leaves, proof that multi-factor authentication (MFA) is enforced rather than optional, a record of who changed identity settings, and a demonstration of how one customer’s users are kept apart from another’s. Each of these can be a log export, a configuration screenshot or a short written procedure, as long as it is real and dated.
If your first large customer has just sent a vendor risk review, this post explains what each request means, where the evidence normally lives, and what to decide before the follow-up call arrives.
Why does a vendor security assessment ask for evidence and not just answers?
A security questionnaire records what you say you do, and the evidence step lets the buyer’s security team check that the statements are true. Questionnaires are self-reported, so reviewers often pick a handful of answers, frequently ones about access control, and ask for something that proves them. A “yes” without a matching artifact tends to become a follow-up call or a delayed approval.
The buyer is trying to confirm three things. The first is that the control exists, for example that MFA is configured. The second is that it applies to everyone it should, not only to the account you showed in a demo. The third is that it has been working over time, which is why dated logs carry more weight than a screenshot taken yesterday.
This is also how SOC 2 reports are built. The AICPA’s 2017 Trust Services Criteria (with revised points of focus, 2022) group logical access controls under the CC6 series, and in a Type 2 examination an auditor tests those controls by sampling evidence over a period rather than by accepting descriptions. A buyer asking for login evidence is applying the same logic to your product without waiting for an auditor. If you are weighing that route, our post on what to do when a prospect asks for SSO and SOC 2 covers how the two requests relate, and why SSO alone doesn’t pass procurement explains why the review rarely stops at a feature checklist.
What login evidence will an enterprise security team ask for?
The five requests below come up often, though the exact wording depends on the buyer’s template. Questionnaire formats such as the Cloud Security Alliance’s Consensus Assessments Initiative Questionnaire (CAIQ, which is built on its Cloud Controls Matrix) include identity and access management and logging as domains, so these topics are standard rather than unusual.
A record of who signed in, when, and from where
The first request is usually an export of authentication events for a sample period, showing the user, the timestamp, the source IP address, the result (success or failure) and the method used. The buyer wants to see that sign-ins are recorded at all, that failures are recorded as well as successes, and that the records are kept for a stated period.
NIST SP 800-92, the Guide to Computer Security Log Management (2006; a revision is in draft), is a useful reference for what a reasonable log program includes, such as consistent timestamps, protected storage and a defined retention period. You don’t need to cite it to the buyer, but the questions you’ll receive about retention and tampering follow the same ideas.
Proof that access is removed when someone leaves
Reviewers want to see that offboarding works for your own staff and, more importantly, for the customer’s users. For your staff, this means a list of people who left in the last year alongside the date their accounts were disabled. For the customer’s users, it means showing that when the customer disables someone in their identity provider, that person loses access to your product, either automatically or within a stated time.
This maps to SOC 2 criteria CC6.2 and CC6.3, which deal with registering and authorizing users and with removing access when it is no longer appropriate. Auditors typically ask for a terminated-user sample and compare HR dates with account-disable dates. Our explainer on what a user access review is describes the periodic check that sits alongside offboarding.
Proof that MFA is enforced, not optional
Saying “we support MFA” and showing that MFA is required are different statements. A buyer will usually ask for a screenshot or export of the policy setting that requires a second factor, and sometimes a report listing any accounts that don’t have one enrolled. Administrators and anyone with access to production data get the most scrutiny.
For reference, NIST SP 800-63B, the digital identity guideline on authentication, defines three authentication assurance levels (AAL1 to AAL3) and which kinds of authenticator are acceptable at each one. If a customer names a level in their questionnaire, read it in the NIST text, since the level decides which factors they will accept.
A record of who changed identity settings
Changes to the login configuration are as sensitive as logins themselves. Examples are disabling MFA for a group, adding an identity provider, changing a session timeout, or granting someone an administrator role. The buyer asks for an administrative change log that shows who made each change, what changed and when.
Reviewers usually treat this as change management for the identity system (SOC 2 CC8.1 covers changes to configuration), and the evidence is the same sort of log as for sign-ins but with a different source. In Keycloak, for instance, user events (sign-ins, failures, logouts) and admin events (changes made through the admin console or Admin API) are configured separately under Realm settings, Events, and neither is saved by default. Admin events also record only who changed which resource unless you turn on “Include representation”, which stores the new values. Either gap is a common reason a log turns out to be empty or thin when the request arrives. Skycloak’s audit logs cover both sign-in and admin events. Whichever system you use, check both.
How one customer’s users are separated from another’s
For a multi-tenant product, the reviewer wants to understand how a user at one customer is kept from seeing another customer’s data. In identity terms, this means showing how each customer’s users are scoped, whether through separate realms or organizations, or through a tenant identifier carried in the token and checked on every request.
The usual evidence is an architecture diagram, a short written description and, where possible, a test result showing that a token from one tenant is rejected by another. This is the item most likely to need an engineer’s explanation rather than an export, so it helps to have the diagram ready before the call.
Where does this evidence live, and why is it hard to collect?
Login evidence is hard to collect mainly because it is spread across several systems and nobody is assigned to gather it. Sign-in events may sit in your identity provider or application database, offboarding dates in an HR tool, MFA policy in an admin console and tenant isolation in code and architecture notes. Assembling one package means querying four or five places in the same week.
A common failure is that the process depends on one engineer remembering where everything is. That person writes a query, exports a file and pastes a screenshot, and the next request takes the same effort again because nothing was written down. If that engineer is on leave while a deal is waiting on the answer, the review stalls until they are back.
A second failure is that logs exist but cover too short a period. Many systems keep authentication events for a limited time by default, and some require you to enable administrative event logging before it records anything. You can’t create last quarter’s records after the buyer asks for them, so it’s worth checking retention settings now.
Here is a rough map of where each item normally comes from and who can produce it.
| Evidence | Typical source | Who usually holds it |
|---|---|---|
| Sign-in records | Identity provider or application auth logs | Engineering |
| Offboarding proof | HR system plus identity provider account status | People ops and engineering |
| MFA enforcement | Identity provider policy and enrollment report | Engineering or IT |
| Identity setting changes | Admin or audit event log | Engineering |
| Tenant separation | Architecture documents and test results | Engineering lead |
Identity management as a service, on open source
Skycloak does what Auth0 and Okta do, SSO, MFA, SCIM, audit logs and enterprise federation, on an open source core. Unlimited users and applications on every plan, no charge per monthly active user, and you can export and self-host whenever you want.
What should you decide after reading this?
Before the next call, settle three things: what you can produce today, who owns collecting it, and what you will ask an identity vendor to supply. Doing this once, in writing, means the next review takes hours of work instead of days, and it also tells you which gaps to close first.
What you can produce today. Try to generate each of the five items for a recent 30-day window. Note which ones you can export in under an hour, which ones need manual work, and which ones you can’t produce at all. The last group is your remediation list.
Who owns collecting it. Name one person, with a backup, and store the answers and exports in a shared folder with dates. The goal is that a sales engineer can find last quarter’s evidence without asking the person who built the system. If a SOC 2 report is on your roadmap, this folder becomes the starting point for your auditor, and our comparison of SOC 2 Type 1 or Type 2, which first helps you plan the sequence.
What to ask an identity vendor to supply. If you use a managed identity service, ask which of the five items it can export directly, how long it retains sign-in and admin events, whether you can stream them to your own log store, and what it can say in writing about its own controls. Skycloak’s published position on its own controls is on the security page, and you should read any vendor’s equivalent page before relying on it in your own answers. Be wary of promising a buyer something your vendor hasn’t agreed to provide.
It also helps to be realistic about timing. Evidence collection is one of several tasks on the path to being ready for enterprise buyers, and how long it takes to become enterprise-ready on login lays out the order of that work.
Checklist to forward to your head of engineering
- Export 30 days of sign-in events with user, time, source IP, result and method.
- Confirm the retention period for sign-in and admin events, and write it down.
- Produce a list of accounts disabled in the last year, with the matching departure dates.
- Capture the MFA policy setting and an enrollment report that lists accounts without a second factor.
- Export the administrative change log for identity settings and confirm admin events are switched on.
- Write a one-page description of how tenants are separated, with a diagram.
- Name the owner and a backup, and choose where dated evidence is stored.
- List what your identity vendor can export and what it cannot.
Frequently asked questions
What is a vendor security assessment?
A vendor security assessment is the review a customer’s security team runs on a supplier before approving it. It normally includes a security questionnaire, a request for supporting documents such as a SOC 2 report or policies, and sometimes a call or a request for specific evidence like the login records described above.
How is a security questionnaire different from the evidence request?
The questionnaire collects your written answers, often in a spreadsheet, and the evidence request asks you to prove selected answers with artifacts. The questionnaire is usually the quicker part, and teams often spend longer on evidence, because the artifacts have to be pulled from several systems.
What counts as SOC 2 evidence for login controls?
SOC 2 evidence is whatever shows a control operating, such as sign-in logs, a terminated-user sample, an MFA policy export, or a change log for identity settings. The relevant criteria sit mainly in the AICPA’s CC6 series on logical access. Your auditor decides what is sufficient for your scope, so ask them what they will sample.
Do we need a SOC 2 report to pass a vendor security assessment?
Not always. Some buyers accept a completed questionnaire plus evidence, while others require a report before signing. If you don’t have one, a written commitment with an auditor and a target window, along with the login evidence above, is often enough to keep the review moving, though this depends on the buyer.
Can our identity provider produce this evidence for us?
Partly. An identity provider can usually supply sign-in events, administrative events and MFA policy settings, but it can’t supply your HR offboarding dates or your application’s tenant isolation tests. Ask the vendor which exports exist, how long data is kept, and what is available on your plan.