Why Microsoft 365 Audit Logs Lack Proper Fit and Finish

Audit logs are incredibly useful tools for troubleshooting issues and investigating security incidents. That is, when they work as expected. In this article, we discuss a recent example where the Azure AD audit log failed to deliver. While there is nothing earth-shattering in these observations, they demonstrate that Microsoft needs to continuously monitor existing functionality to make sure everything performs as advertised, instead of constantly moving to the “next big thing.”

Setting the Stage – Auditing New License Assignments

In this article, we focus on the reliability of capturing Azure AD audit events when a license is assigned to an Azure AD user. Apart from the obvious cost of assigning licenses in terms of how much your organization pays Microsoft, licensing also controls access to functionality across the Microsoft 365 suite. For both reasons, being able to audit licensing changes is important.

When I reviewed the new auto-claim policies, I wanted to check what the audit trail looks like. To my surprise, performing a search for change user license events found no entries, even though I could clearly see the operation was successful and the license assigned. I decided to wait to account for potential latency in populating the audit log. A week or so later I circled back to find no events, so I asked Tony Redmond to check whether he experienced the same behavior. He confirmed that some events were missing, so we then proceeded to ask other if this is the expected behavior.

The Plot Thickens

To ensure I missed nothing obvious, I looked for the corresponding events in both the Azure AD Audit log, the Office 365 audit log and MCAS (which seems to get some additional information, at least for some events). I also tested some other scenarios, including assigning licenses to accounts via various admin endpoints and using the group-based licensing feature. Some oddities were immediately visible. For example, when assigning a license via the Microsoft 365 Admin Center, Azure AD generated the corresponding change user license event. However, the audit event contained no information about the license assigned to the account. Examining the “raw” event data showed the “modified properties” value were empty (Figure 1).

In effect, while we can see that a license change occurred for the user, there is no way to tell exactly which license was assigned. To confirm, I tested the same operation in another tenant, and asked fellow MVPs to do the same. Everyone reported the same behavior. Microsoft’s documentation (Figure 2) says that I should check the Updated user audit event.

However, I found no trace of any Update(d) user events when I assigned a license (as confirmed in Figure 1). In contrast, assigning a license using the Azure AD portal results in an Update user event without any Change user license event (in this case, the “modified properties” contain the expected information). The lack of Change user license event makes it harder to find/filter just the relevant licensing events. Testing further, when I assign a license using the Set-MsolUserLicense cmdlet from the MsOnline module, the action generated both event types. In effect, each endpoint behaved differently, with the Microsoft 365 Admin Center being worst.

Group-based Licensing Joins the Fray

I found another oddity when using the group-based licensing feature. Accounts received license assignments as expected, but no Change user license audit events were captured. Technically speaking, you should be able to “reconstruct” the missing events based on the information stamped on the group object and the returned GroupManagement events. However, getting a point-in-time list of members for a group is far from trivial, especially in any large organization. This seems to be yet another example where the Azure AD Audit logs failed to capture the required audit events for license assignments. A week later, those events are still missing, but a similar group-based licensing operation performed on April 2 created the expected events (Figure 3).

Surfacing auto-claim policy events

Circling back to auto-claim policies, I decided to perform a more expansive search of the audit log and found several Update User entries containing details of the licensing changes. No accompanying Change user license events were generated, which probably explains why the initial query was unsuccessful. Aimed with the knowledge obtained from reviewing these events, it is possible to create a search filter to find events for license assignments by auto-claim policies. The easiest way to do this is to filter for events initiated by a specific service principal, ead24062-d380-4260-9d1b-3891efe04ccc, also known as as Signup (Figure 4).

If you search the Office 365 audit log through the Compliance center, enter the value ServicePrincipal_ead24062-d380-4260-9d1b-3891efe04ccc in the Users box. Or you can also use the following PowerShell command:

Search-UnifiedAuditLog -StartDate "2 Mar 2021" -EndDate "3 Apr 2021"  -UserIds ServicePrincipal_ead24062-d380-4260-9d1b-3891efe04ccc

Licensing Events Need Some Attention

Through persistence, audit events for licensing changes executed by the auto-claim policies feature are available. On the other hand, a week after assigning a license from the Microsoft 365 admin center, no additional audit events or data is available, leaving you unable to tell which license(s) were assigned. Interestingly, if you assign a new SKU with some of the service plans disabled, changing an existing license assignment or removing a license from an account create a proper audit trail. And the changes made via the group-based licensing feature on March 30^th are still missing the corresponding Change user license events, signaling another (albeit temporary) issue with audit event generation. Overall, the inconsistency in audit event generation across management endpoints is infuriating.

This might not sound like a big dedal, but our investigation focuses only a specific event, and only at events created in the Azure AD log. If we expand our examination to cover the Office 365 audit log and broaden the set of events and workloads, a whole new set of issues pops up quickly, even if we concentrate solely on the trail generated by auto-claim policies. For example, even though the “claiming” of license happens after a user first logs into Teams, no TeamsSessionStarted events exist for any of the users we tested this functionality with. In contrast, there are 30+ such events for my own account during the month of March.

…And So Does the Entire Audit Experience

And that’s just one example. Missing events, missing data within events, “unknown” objectIDs, missing IP address information, missing client information, GUIDs instead of “human readable” identifiers are just a few of the issues visible by examining this specific scenario. If we are to consider the full scope of the Office 365 audit log, the list grows substantially. A personal favorite of mine are all the “system” or “backend” events, which often pollute the audit log with useless information, and are sometimes sufficiently cryptic to make you wonder whether someone compromised your tenant.

There is no denying that Microsoft keeps on improving the auditing functionality. They have polished the overall experience over the years, decreased latency for event generation, and incorporated important scenarios such as capturing events for service principal sign-ins. More and more workloads and events are supported. Yet, we continue to see this trend of neglecting obvious issues and a lack of attention to detail. As a reminder, it took Microsoft several months to fix the infamous truncation issue, years to add SharePoint Online admin operations, and now for sensitivity labels. And there are still workloads and operations within the broader Microsoft 365 suite that simply do not generate audit events.

Finally, there is the trend of putting auditing information behind a license paywall. Azure AD has traditionally required an AAD P1 or equivalent license to access some of the audit and reports data, which means that even Office 365 tenants (which get the Azure AD for Office 365 tier) cannot access the data or see a limited subset of what’s available. Such organizations can take advantage of the Office 365 audit log instead, but that’s not always a solution. Microsoft has also introduced the concept of paying extra for access to audit events for Office 365 activity with high value audit events. This is a pity, given the importance such “premium” events play in investigating account compromises. Before looking to charge customers extra for audit data, Microsoft should take care of the basics and make sure that all workloads generate audit data on a consistent and predictable basis.