Microsoft’s Spectre/Meltdown Guidance for Exchange Server and Office 365 Customers
Post author:Written By Paul CunninghamPost published:January 18, 2018
Table of Contents
Spectre and Meltdown are attacks against previously undisclosed vulnerabilities in modern processors (CPUs). The names “Spectre” and “Meltdown” were chosen because they exploit vulnerabilities in “speculative execution” (a technique that allows processors to work really fast) and “melt security boundaries”.
Meltdown and Spectre exploit critical vulnerabilities in modern processors. These hardware vulnerabilities allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents.
Meltdown and Spectre work on personal computers, mobile devices, and in the cloud. Depending on the cloud provider’s infrastructure, it might be possible to steal data from other customers.
This means that a malicious app on your phone could steal data from your other apps. Or a malicious program on your computer — maybe one running in a browser window from that sketchy site you’re visiting, or as a result of a phishing attack — can steal data elsewhere on your machine. Cloud services, which often share machines amongst several customers, are especially vulnerable. This affects corporate applications running on cloud infrastructure, and end-user cloud applications like Google Drive. Someone can run a process in the cloud and steal data from every other user on the same hardware.
If that sounds bad to you, it’s because it is bad. There is really no upside to this one. However, although these vulnerabilities do exist, they are not considered to be actively exploited at this time (or rather, there have not been attacks attributed to these exploits). This means you don’t need to panic in your response, and should instead take the time to learn about the available fixes and mitigations, and roll them out to your devices and infrastructure.
Mitigation for the known vulnerabilities comes in multiple parts:
Software configurations (KVAS and IBC, described in the article)
Formulating a response is complex, because full mitigation involves multiple updates/changes working together. For example, IBC mitigates one variant of Spectre, but only if the required firmware updates from your hardware vendor have been applied. One of the other mitigations, known as KVAS, is expected to have a negative performance impact. More on this shortly.
Spectre/Meltdown for Exchange Server and Exchange Online
This week, Microsoft has released their Spectre/Meltdown guidance for Exchange Server customers. It’s a long and dense read, which you can find here. You can also find Microsoft’s general guidance for Windows Server operating systems here. The Windows Server guidance is a key part of the response for Exchange Server customers, because this is not something that can be mitigated with an Exchange Server update.
There are five scenarios covered by Microsoft’s Spectre/Meltdown guidance for Exchange Server customers:
Office 365 customers – Microsoft has “taken action to help secure our cloud services”. If you are a 100% cloud-based customer, Microsoft has dealt with the “server stuff” for you, and your focus can be on dealing with your other devices such as desktops, laptops and mobile devices. If you still run some on-premises server infrastructure, there’s more to consider.
Exchange on hardware (bare metal) – Microsoft recommends applying all updates following your normal testing and deployment procedures. They do not consider KVAS to be required in this scenario, but you might (see discussion of “trusted” code later in this post).
Exchange on a VM in public environment – For Azure, Microsoft has detailed their mitigations here and guidance for what to do with your VMs (basically, install updates and consider enabling KVAS). If you use a different cloud provider, they recommend you check with them for more information (AWS here and here, Rackspace, DigitalOcean).
Exchange on a VM in a private environment – Microsoft recommends applying all updates following your normal testing and deployment procedures. Consider enabling KVAS based on your assessment of the risk of untrusted third party code execution.
Exchange on a VM or bare metal and is not isolated from other application logic on the same system – Microsoft recommends installing all updates following your normal testing and deployment procedures. Consider enabling KVAS based on your assessment of the risk of untrusted third party code execution.
Trusted vs Untrusted Code
The “trusted” vs “untrusted” code question is a complex one. As some of my fellow MVPs have argued in our discussions on this topic, to many infosec teams, all third party software (including Microsoft’s) falls into the “untrusted” category. And even if you trust Microsoft software, you still need to think about every other agent running on a server (antivirus, backups, monitoring, etc) and whether those could ever become compromised in a way that lets an attacker exploit the Spectre/Meltdown vulnerabilities.
If you’re in a public cloud or multi-tenant environment, it’s not just the code on your own systems that you need consider trusted or untrusted. It’s also code on other systems running within that environment, systems that you don’t control or have any visibility into.
KVAS, IBC, and Performance Impacts
Microsoft recommends enabling the mitigations known as KVAS and IBC in pretty much any shared environment, and any environment where untrusted code is allowed to run alongside Exchange. If it were simply a matter of rolling out all the fixes with no detrimental impact, then this whole issue would be slightly less messy. No such luck, unfortunately. Full mitigation of the Spectre/Meltdown attacks, as they’re understood today, will have an negative performance impact on Exchange Servers.
The full extent of the impact is unknown. I suspect this is mainly due to the number of variables involves. With Exchange running on so many different hypervisors, hardware platforms, different CPU vendors and specs, different OS versions, and so on, it’s impossible for Microsoft to provide specific numbers. What they do say is this:
We have measured the effect of Kernel Virtual Address Shadowing (KVAS) on various workloads. We have found that some workloads experience a significant decrease in performance. Exchange Server is one of those workloads that may experience a significant decrease if KVAS is enabled. Servers that show high CPU usage or high I/O usage patterns are expected to show the largest effect. We highly recommend that you first evaluate the performance effect of enabling KVAS by running tests in a lab that represents your production needs before you deploy into a production environment. If the performance effect of enabling KVAS is too high, consider whether isolating Exchange Server from untrusted code that is running on the same system is a better mitigation for the application.
In addition to KVAS, information about the performance effect from Branch Target Injection mitigation hardware support (IBC) is detailed here. A server that is running Exchange Server and that has an IBC solution deployed to it may experience a significant decrease in performance if IBC is enabled.
Ouch. But is that how it’s panning out in the real world so far? Unfortunately, yes. In a blog post, Solarwinds shared the impact they measured on their AWS infastructure as updates were rolled out.
We saw a window as short as just six hours between different availability zones within us-east-1, a relatively small time window between AZs for anyone running large multi-AZ workloads. CPU bumps like this were noticeable across several service tiers.
Image via Solarwinds at blog.appoptics.com
But it’s not all bad news. Solarwinds has also reported that they’re seeing improvements in CPU usage, perhaps as more efficient patches are applied to the underlying AWS infrastructure.
New EC2 hot patches for Meltdown/Spectre rolling out? Previous CPU bumps appear to be dropping off starting after 10:00 UTC this morning. pic.twitter.com/OL5N5TNG1s
The implication is that over time, vendors might get better at mitigating the vulnerabilities without the significant performance impact. Only time will tell.
What Next?
This is not a problem that you can ignore, nor is it a problem that you should panic about and rush a response.
Start by reading the vendor guidance I’ve included in this article (I will summarize links below as well), and seeking guidance from your hardware or cloud providers.
Perform an assessment of the risks to your environment, and the mitigations required to address those risks.
For Exchange servers, take a performance benchmark before you implement any updates or mitigations.
Create a plan to deploy the updates and mitigations, beginning with test or non-critical infrastructure.
Continue to implement your plan while monitoring for additional guidance from vendors as the impacts are better understood.
Resources
Here’s a list of the resources I linked to throughout the article above, as well as additional sources I looked at while writing this.
Additional note: I originally considered not writing about this, but ultimately decided to publish this post. I’ve made every effort to make this blog post accurate and relevant for Exchange Server admins and customers. If you find any errors or points that need clarifying, please leave a comment below to let me know. If your comment includes a link it will be held in moderation temporarily until I am able to approve it.
Paul is a former Microsoft MVP for Office Apps and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul no longer writes for Practical365.com.
On the show this week, Rich and Steve are joined by Distinguished Engineer & AI Security Evangelist, Mike Wilson to discuss Microsoft's focus on AI, AI Security, the new Microsoft Planner, and more!
Just over two weeks ago at Ignite 2023, Microsoft confirmed features that made the cut for Microsoft 365 Copilot, and what features are coming further down the line. With full access to the GA version of Microsoft 365 Copilot, Steve Goodman takes a look at how well it works in the real world, and how it interacts with data in your Microsoft 365 tenant.
Exchange Online keeps message trace data online for 10 days. If you want an email traffic report covering longer periods, you need to retrieve historical message data (for up to 90 days) and report that data. That's the easy description. This article explains some of the complexities involved in generating a 90-day email traffic report for a Microsoft 365 tenant.
