A Serious Worldwide Attack Against On-Premises Exchange
The news that Microsoft Exchange on-premises servers have been hacked before and following the disclosure of four flaws patched by updates released by Microsoft on March 2 creates a serious problem for organizations running on-premises Exchange. Apart from ongoing attacks, it seems that attackers are leaving behind a “web shell” to allow them to continue to have administrative access to servers.
KrebsOnSecurity.com, which reported the widespread infection of Exchange servers, doesn’t give a verifiable source for its claim that “At Least 30,000 U.S. Organizations Newly Hacked Via Holes in Microsoft’s Email Software.” Similar claims are made by TheVerge.com and Wired.com while Bloomberg doubles the estimated victims “to least 60,000 known victims globally.” In this instance, the cited authority is “a former senior U.S. official with knowledge of the investigation.” Clearly someone who keeps close track of Exchange on-premises servers.
The number of affected servers are educated guesses on the part of security researchers and other observers, possibly informed by running network traces to find vulnerable servers. There’s no way of keeping score of the exact number of Exchange servers or organizations penetrated by attackers. Even so, there’s no doubt that the problem exists and is very real. And it is a worldwide problem that isn’t unique to the U.S.
Here are how the experts combat Hafnium attacks and security flaws
A Developing Attack
Here are some updates gathered to track the development of attacks against Exchange servers. The underlining message is clear: unpatched servers are at immediate and imminent risk of compromise. In many cases, a China Chopper web shell is left on a server to allow attackers future access to the server.
Update March 8: Microsoft says that they “see multiple actors taking advantage of unpatched systems to attack organizations with on-premises Exchange Server.”
Update March 9: Palo Alto Networks estimate that there are “over 125,000 unpatched Exchange Servers.”
Update March 10: Wired.com reports that multiple groups of hackers are busily figuring out how to exploit Exchange server vulnerabilities, including hackers preparing to use Exchange servers to mine cryptocurrency. They also report that “public scans still show more than 10,000 Exchange servers that are vulnerable to attack” and note that at least one proof of concept of an attack is circulating (Figure 1).
Update March 11: ZDNet reports that multiple attacker groups are scanning for vulnerable Exchange servers with the intention of compromising the servers before administrators apply patches.
Update March 12: Check Point Software says that “the number of exploitation attempts […] doubled every two to three hours” and the countries most attacked are Turkey (19% of exploit attempts), the U.S. (18%) and Italy (10%). On March 15, the UK National Cyber Security Center (NCSC) said that over 3,000 Exchange servers used by UK organizations had still not been patched.
Update March 12: Bloomberg reports that large-scale attacks started on Exchange servers a week before Microsoft released patches on March 2. The theory is that a leak occurred somewhere (see below for a possible source) to warn the attackers that the gap they wanted to exploit was about to close.
Vulnerability Known Since January
According to security researchers, the exploit has been ongoing since early January with an uptick of activity since February 26 (see Bloomberg report cited above). Microsoft needed the time between the initial reports and March 2 to analyze the attack and prepare the security updates. On February 23, Microsoft shared a proof of concept for the attack with the partners in the Microsoft Active Protection Program (MAPP) to give security vendors a heads-up and prepare updates for anti-malware updates. Some suspicion exists that the attackers learned that Microsoft was working with partners and decided to ramp up their activity, which then exploded after Microsoft released its patches.
Since the news broke, Microsoft support has struggled to deal with the volume of customer queries. Microsoft has issued scripts, tools, and advice as the days passed (see our page reporting the release of the patches), but it would have been better to have this help lined up and available on March 2.
The critical nature of the problem is underlined by statements by the Microsoft Security Response Center that the available mitigations “are not a remediation if your Exchange servers have already been compromised, nor are they full protection against attack” and that actions will “not evict an adversary who has already compromised a server.”
The current situation is that if you have not updated servers and you allow OWA access from the internet, you run an extremely high risk of server compromise. I know of several companies who have patched and since discovered that servers still show signs of infection. In many cases, the only solution is a complete server rebuild from scratch (like this example reported by MVP Jaap Wesselius).
Exchange Online
I consider the attack by Hafnium a wake-up call for those still running on-premises servers. It should provide the impetus for a push to move email from on-premises Exchange to Exchange Online. We’ve been on this road for a while and it’s time to get the job done.
Moving Exchange to the cloud began in 2005 but only became mainstream after the release of Office 365 in 2011. I spoke about the perils of moving to Exchange Online at the Exchange Conferences of 2012 and 2014. On-premises servers were still attractive in 2014 but the situation is very different now, both in terms of the threat to on-premises servers and our knowledge of what it’s like for companies to run email in the cloud.
According to data shared at the TEC 2020 conference, Exchange Online supports 5.5 billion mailboxes. That number seems enormous in the context of 250-odd monthly active Office 365 users, but more reasonable when you consider that the figure includes Outlook.com users (400 million switched over to use the Exchange Online infrastructure in 2017), shared mailboxes, group mailboxes, resource mailboxes, and a very large number of system mailboxes used by the Microsoft 365 substrate. Exchange Online is a massive online service running on 275,000 mailbox servers. The attack penetrated none of these servers.
Whatever the number of mailboxes running in Exchange Online is, there’s little doubt that most of the email processed by Exchange is now cloud-based. I doubt anyone knows exactly how many on-premises Exchange servers remain today. Microsoft could hazard a guess, but I think there would be plenty of wriggle room in the answer.
Join our webcast Exchange Server Exploits: Why Cloud Email is More Secure and How to Finally Move
The Joys or Not of On-Premises Server Management
The crisis reminded me how much I had forgotten about running on-premises Exchange. And the reports of having to rebuild OWA virtual directories, of failed updates because of anti-virus products, and problems occurring because no one had applied .NET and C++ updates to servers for a while did not make me want to rekindle my experience. Email is a commodity service, and I am happy for Microsoft to do the heavy lifting for regular server maintenance.
But Exchange Online is not all happiness and light. Many administrators don’t like it when Microsoft imposes new restrictions, like when they clamped down on automatic mail forwarding in November 2020. The restrictions exist for a reason, but the loss of control can be irksome, as can dealing with Microsoft Support if the agent you speak with isn’t competent with email.
The point is that you don’t get to twiddle the knobs for a utility service. There’s plenty of administrative work to do, not least to keep up with the volume of changes Microsoft makes across Office 365, which means that you’ll use the time otherwise occupied by server upgrades better.
Microsoft Fails Hybrid Deployments
That is, unless you run a hybrid deployment and need an on-premises server to act as the link between Exchange Online and Active Directory. Despite raising hopes over several Ignite (in-person) conferences, Microsoft has failed to deliver a way to allow Active Directory act as the master for email settings on cloud mailboxes and remove the last on-premises server after all mailboxes run in the cloud. The result is that more on-premises hybrid servers exist today than should be necessary, and those servers need maintenance. Indeed, the current crisis has shown that some hybrid servers don’t get updates because “they don’t do a lot of work.” Leaving any server in an unmaintained state is not a good thing to do.
Three Reasons to Move Email to the Cloud
The bottom line is that if you can move email off on-premises Exchange to the cloud you should do so as quickly as you can. When I’m asked about this topic, I ask CIOs and CTOs to consider three points:
- The increasing sophistication of attack. This will continue. The attacks will be more serious and will consume more resources. The latest example underscores the point.
- Better protection is available in the cloud. Some organizations might be able to match the defenses erected and maintained by Microsoft, but most cannot. Moving workload to the cloud should encourage better user-level defense through multi-factor authentication, the elimination of basic authentication, and use of conditional access policies. For those worried about data privacy and security, an array of developments such as go-local datacenter regions, customer key protection for data at rest, sensitivity labels, and administrative controls like privileged access management and privileged identity management are available to help.
- More functionality is available in the cloud. Microsoft’s eyes have been on Exchange Online since they shipped Exchange 2010. Exchange Online is more functional than its on-premises counterpart, and it is a full participant in the Microsoft 365 ecosystem alongside SharePoint Online, OneDrive for Business, To Do, Teams, Planner, and Stream. In short, things have moved on from pure email.
Move. Don’t Delay
Some organizations can’t move or believe they can’t move email to the cloud. A lack of connectivity (a problem disappearing fast in many parts of the world) might be a constraint or a particular security regime. But then Hafnium comes along to prove that the security for tens of thousands of on-premises Exchange servers isn’t so good when put to the test by a nation state. As Microsoft CTO for Modern Workplace Jeffrey Snover asks (Figure 2), does it really make sense to run Exchange on-premises servers anymore?
It’s time to move Exchange to the cloud as quickly as possible before attackers discover and exploit the next set of vulnerabilities. This attack is regrettable, but organizations can rescue some benefit if they use the chaos as a wake-up call to examine just why they continue to run an on-premises email service. It might have been a good call to remain on-premises several years ago. It’s not now.
I appreciate you Tony for sticking up for us Exchange Admins. In all honesty, what aggravates me the most being an Exchange Admin for past 12 years on articles like this is, sure there is some value and benefit to moving to the cloud in some scenarios, for others on-prem it’s still more cost effective for one to stay on-prem, and while the cloud software may be watched closer and patched more frequently, the security argument seems to only apply to the product itself. We were protected from Hafnium because of Geo-Protect and IPS on our firewall. If you have in place the right stuff, it’s just as secure as the cloud if not even more secure. I mean, someone is going to find a hole in a product whether it’s cloud or on-prem. There are always vulnerabilities.
But what does truly aggravates me even more with an article like this stating O365 is the ONLY option and not vouching for making a better more secure on-prem solution. I mean if MS had the model of “you’re the customer, you have the choice of our cloud or on-prem”, no matter which one you choose they will encourage based off of what you the customer wants and will listen to their customers. MS seems to have forgotten that concept when there are so many of us still looking for on-prem because of it’s benefits over the cloud. It’s aggravating to not see both options pitched and helping the customer decide which is better. Their model is O365 is the better choice when it always isn’t.
After working with Exchange since 1995 or thereabouts, all I can do is report the situation as I see it.
Yes, there are organizations capable of running secure Exchange on-premises deployment that suit the requirements of their users very well. There’s no doubt about that. But what has become horribly obvious over since March is two facts. First, there’s a very large set of Exchange on-premises servers which are not being well managed and are increasingly vulnerable as time passes. The users of those servers would be better off in the cloud. At least they’d be safer.
Second, Microsoft’s failure to deliver the committed next version of Exchange Server leads me to think that maybe the sands of time are running out faster than anticipated for these servers.
Office 365 is not for everyone. It is frustrating as hell at times and Microsoft’s quality in some parts is poor. Unbelievably poor. But overall, I consider it the best solution for email today. Again, I can only say it as I see it.
https://practical365.com/blog/azure-ad-outage-march-15-2021/
🙂
Fair point. However, I still hold to my position that many companies would be better to run email in the cloud. We’ll discuss the issues on Thursday at this online event: https://www.binarytree.com/event/exchange-server-exploits-experts-discuss-why-cloud-email-can-be-more-s8147931/
The push for the cloud will probably keep me even busier than usual this year, but I’m not comfortable with MS owning everything. M365 itself can and will likely be exploited. It’s a matter of time. Can anyone compete with M365’s security and feature sets at the perimeter? Probably not.
It’s obvious that MS is wanting out of the on-premises Exchange world with the release of 2019 and licensing requirements even if just running hybrid.
This has me considering researching some open source mail platforms, because I have a feeling those will also be embraced by firms who see this exploit as a failure of MS themselves and will be against migrating to their cloud environment.
It’s always good to consider the alternatives which are available and to understand the costs and technical implications of moving platforms.
The problem for the on-premises base is that it’s declining. Microsoft is a commercial organization and its focus is therefore on the part of the business which generates returns for shareholders, and that’s the cloud. As the number of on-premises seats declines further, Microsoft has an interesting balancing act to perform to keep some large on-premises organizations happy while not investing too high an amount of engineering resources. We’ll see how that goes over time.
You are very true that, this case unfortunately highlighted that companies have troubles to keep their servers updated. Reasons for that are many: money, time, skills, priorities. I believe, this incident gives them a tiny boost to patching all. But that will not solve the fundamental issues.
There are also other services where remote control is still based on default passwords, databases are open to internet etc. We should not forgot them either. We perhaps could ask also, how many of those Exchange servers being open to anywhere to internet must be open (port 443)? Or have they just followed Microsoft best practices?
Can I also ask, have you been too long time surrounded by Microsoft speaks, and having hard to see no more than O365 as a solution? ;-D But seriously, perhaps there should be some reminders of how companies are able to keep their platforms updated as they should.
ps. you should have counter which shows how long messages we could write on here 🙂
I don’t know how long a message can be. But in any case, short and sweet is the best way to get your point across.
It’s true that I have been focused on cloud services since 2015 or thereabouts, but I have worked with on-premises technology since my first email server in 1982, so I think I have balance. I understand the challenges that you outline, but I do think that companies who can’t maintain on-premises servers should be in the cloud because sooner or later they will have a security problem.
I would also point out that although I have close connections to Microsoft, I never hesitate to call them out when things go wrong. See https://practical365.com/blog/exchange-hcw-replaced/ as an example.
Not sure how do you see this, but I feel it might be hard to see when looking for from US side, why companies are not moving to cloud services when all of the cloud vendors are US based companies. The risks that US government overwrite rules in future is a real risk which all (non US) companies has to be taken, if they really owns their data in a future.
Another risk category for competition between cloud vs. online is money. If you close your eyes and looking to the future were no on-prem exchange servers exists anymore. Then that basically means Microsoft have full monopoly for Exchange services. After that, MS is able to start lifting the prices as they wishes, as their customers have no rollback options anymore. Yes, you could argue that MS have no monopoly because of Google. But when you are locked into one cloud cage, the price of migration to Google keeps you as a prisoner on EXO. And is it really everyone’s wishes to have only to major vendors for offering email services?
Couple of things to consider:
First, I don’t think Microsoft has shown much appetite for increasing cloud prices for basic services like EXO. They haven’t since 2011. What they will do is try to upsell higher price plans and add-ons, just like EA Games will try to sell you a new player in FIFA 2021.
Second, there’s a bunch of “go local” datacenters deployed at country level which allow organizations to keep their data in that country. Sure, the US government might try to do something bizarre, but that might also affect on-premises software. It’s not so long ago since the encryption technology in Outlook was considered a state secret in some countries…
The thing I can never seem to find a concrete answer to is: “In a small business with only 34 mailboxes, what do I give up if I just do a cutover migration and don’t keep a hybride-mode Exchange host onsite?”. Such very minimal, simple email needs. We need an alias or two on some mailboxes. We have five shared mailboxes, and maybe 12 distro lists, tops.
All of those things seem so very easily and readily manageable via the o365 admin console. We couldn’t care less if someone’s AD password matches their o365 password for their email. Treat them as different things, every single email thick or mobile client readily remembers your password, so it’s a one-time entry and you’ll never, ever need your o365 password again if you’re a normal mail user.
We just really, really do not want a hybrid server and would more than happily manage aliases/distros/etc. via the admin console. But everyone says, “No, no way, if you don’t have hybrid mode it’s a nightmare”. OK, why?
It’s a nightmare if you don’t have a hybrid server and plan to run hybrid for the long term. In your situation, I imagine that server could be removed. There are ways to do this (like https://www.itpromentor.com/remove-hybrid-keep-sync/) and it’s been debated extensively (like https://practical365.com/exchange-server/how-to-remove-the-last-exchange-server-dilemma/). At the end of the day, if you want to remove that last server you can… but Microsoft won’t support you, which is the problem.
Any chance this will motivate Microsoft to speed up work on removing the requirement to have a hybrid Exchange server for AAD Connect support? It sure would be nice to get rid of those last Exchange servers. Also, do you know of any reason we would need to continue to allow access to Exchange web services (TCP 80 and TCP 443) on hybrid servers? Blocking access to those ports would limit our exposure to future vulnerabilities such as this.
I absolutely hope so. This is a commitment Microsoft has abjectly failed to deliver on despite many commitments made at Ignite conferences.
I’d love to know what bad things will happen if I decommision my local exchange server that has no mailboxes on it and continue to use DirSync. This seems like such a simple question but it must not be supported?
I think the problem is that Microsoft won’t support this scenario.
I find it hard to stomach that the exchange servers running o365 were not affected taking in mind they are running a similar verison. Furthermore it seems patching quality from ms has dropped off the scale . So its only a matter of time before o365 emails get a canning. When its on prem u can manage this risk. Where as with o365 u are hoping ms do there bit . Which im yet to be full happy with
Well, there’s no evidence that the EXO servers have been affected. Couple of points to consider:
1. EXO doesn’t run the same code base as on-premises servers. They used to, but that stopped with Exchange 2016.
2. Microsoft had time to prepare its defenses after security researchers identified the issue in January. I’ve no doubt they focused on EXO because they needed to protect the 250-million plus daily active users of that service.
3. You can certainly do your utmost to defend an on-premises server against attack. Good luck. Microsoft has a bunch of people defending EXO. It’s just one less thing for an organization to have to deal with.
4. I agree that Microsoft has some issues to deal with in terms of patching quality. Is everything running so fast now that we all have less time to do a good job? I wonder this all the time when I see the quality of some of the software features introduced in the cloud.
Another thing to consider is that this attack requires an untrusted connection whereas Office 365 doesn’t allow untrusted connections.
I would say many companies are using exchange online but in hybrid mode, which makes administration easier. I think this is best practice per Microsoft. So I’m not sure what those organizations are to do.
Threats are everywhere. Didn’t Microsoft just have code stolen through the solar winds ordeal? We will probably never truly know the extent of that breach. Does that make the Microsoft cloud safer? No!
As another poster had mentioned – Microsoft has a huge attack surface due to their market share. So just like with windows, they become a primary target for hackers. I see lots of phishing emails coming from office 365 accounts.
I think every organization needs to perform their own risk assessment and decide what systems and processes provide the best value with the least risk.
Even if hackers stole some code from Microsoft during the Solarwindows exploit, no one knows if a) it’s Exchange Online code and b) the code still runs in the service.
Yes, Microsoft represents a massive attack surface. However, that’s balanced by the resources they deploy to protect their services.
As always, it’s a balancing act. If you think you can do better on-premises then stay on-premises. But one lesson we have learned from this exploit is that many organizations leave their servers sadly under-maintained and unpatched, which then creates a very exploitable attack surface for hackers.
Nothing about cost/price in this entire article.??? With 400 users, how much will it cost us to run for 5 years in the cloud?
How much will it cost to have servers melt down because of an attack?
Look, every company will have a different cost-benefit analysis. For some, they absolutely need a guarantee of email availability no matter what the threat horizon happens to be. I think the cloud is the best way of getting this today.
Others will be more cost sensitive and yes, you can definitely run on-premises servers into the ground, but eventually you are going to need to reinvest and Microsoft has already said that they are moving to a subscription model for new versions of Exchange, so you’re going to be paying a per-month fee anyway.
Do your costs, understand the threat, and make a decision. That’s all anyone intelligent can ask in the absence of knowledge of the individual circumstances of a company. As I keep on saying, ten or five years ago, I would have been more prone to stay on-premises. Now I am not.
As Tony says, the landscape has changed and continues to do so – including an ever-growing risk if you’re trying to maintain your own environment. The cost and benefit depends upon your requirements – do you need kiosk licenses or E5 plus all of the additional options? He’s also right about the cost of doing nothing – getting breached will likely cost you more than any “extra cost” of moving to the Cloud (and you’ve really got to consider the hidden costs of maintaining infrastructure and skills when you’re doing that calculation) – and that’s before you consider the cost of any fines if you have data leakage. Speak to a company that specializes in Cloud migration to really understand the cost / benefit analysis – you’re not committing to anything and you’ll get a far better idea of your specific situation.
The cloud is also a huge target. 275,000 servers? No thanks. Microsoft and other companies need to keep their software up to date and stop leaving on-premise lagging behind the hosted versions in security updates.
All clouds are huge targets, but they’re also staffed by thousands of people who are dedicated to security and protection. I don’t know about you, but I warrant a guess that they know a lot more about resisting attacks that I do, so I am happy to let them go at it.
Also, about 10% of the servers are offline at any time under maintenance. Microsoft doesn’t apply patches the way people do on-premises. They strip servers to bare metal and lay out a complete kit of Windows, Exchange, and any required patches. There is less chance of missing an important fix or patch that way.
Running your own on-premises environment is rarely the right answer. Is your core business managing infrastructure and services? If not, consider what it will cost you to have the same quality of service as you’ll get from M365 and what kind of a distraction that is from focusing on your core business. You probably don’t generate your own electricity and trust this to a specialist. If you’re dependent upon on-premises servers and simply can’t move to the Cloud then I sympathize but I honestly don’t see this very often. It’s frequently down to misinformation and a lack of clarity about what can be done.
thanks Tony im no trying to be negative and i too agree that for lots of businesses like small businesses that dont have the IT resources and historically that is the small business that had servers in cupboards and ran IT on a shoestring budget of course Exchange Online is a no brainer
But i just wanted t o highlight i think not just yourself but the general commentary from MS is that as you should be in the cloud at all costs, etc
i would imagine companies would have already moved en masse to Exchange Online have already done so with the holdouts probably still have a legitimate reason so the tone should change slightly to support these users rather than the repeated mantra above that you cant or why would u still have to patch your servers and maintain them so just move to Exchange Online – i would imagine most companies \ admins are well aware of that and they have good reasons to remain that way
Just an opinion and just would like to thank you for all your work contributing to exchange server over the years just some feedback
Great article Tony but this rhetoric that if your on premise you cant be trusted to do your job and patch and maintain yours servers so just go to Exhange Online just irritates me no end
Not every company fits in the MS hosted model some just need good reliable email without the bells and whistles and some prefer to keep data on their servers where they are responsible for it
And also at what point if its your job as an Exhange Admin then you should manage and maintain your environment which is function on how well you do your job
The rhetoric really is offensive to good admins out there who still take pride in there environment and really needs to change
The impression that on-premises administrators don’t know their job is not what I intended to convey. Remember, I’ve been sticking up for Exchange admins for 25-odd years, so it would be strange if I said something like that.
But it is clear that there are many Exchange on-premises installations which would get better service and better security if they were moved to Exchange Online. That’s not rhetoric. It’s fact. Ten years or even five years ago I would have argued differently. Given the world of threat, the available functionality, and my lack of desire to keep on patching servers, I see things differently now.