Rackspace hosted Exchange suffered a catastrophic interruption starting December 2, 2022 and is still continuous as of 12:37 AM December 4th. At first referred to as connectivity and login problems, the assistance was ultimately upgraded to reveal that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be resolved.
Consumers on Buy Twitter Verified reported that Rackspace was not responding to support emails.
This has been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not exactly sure the number of companies that is, however it’s significant.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t knowledgeable about the bounce for several hours.”
The main Rackspace status page provided a running upgrade of the outage however the initial posts had no info aside from there was an outage and it was being investigated.
The first official update was on December second at 2:49 AM:
“We are investigating a problem that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace began calling it a “connectivity issue.”
“We are investigating reports of connection problems to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates explained the ongoing issue as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the blackout, still attempting to figure out what failed.
And they were still calling it “connectivity and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later Rackspace described the situation as a “substantial failure”and started using their consumers totally free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.
The main guidance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any further concerns while we continue work to bring back service. As we continue to resolve the origin of the problem, we have an alternate option that will re-activate your capability to send and receive emails.
At no cost to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until additional notification.”
Rackspace Hosted Exchange Security Occurrence
It was not until nearly 24 hr later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was suffering from a security incident.
The statement further exposed that the Rackspace service technicians had actually powered down and disconnected the Exchange environment.
“After additional analysis, we have figured out that this is a security occurrence.
The known effect is separated to a part of our Hosted Exchange platform. We are taking essential actions to assess and protect our environments.”
Twelve hours later that afternoon they upgraded the status page with more info that their security group and outdoors experts were still dealing with solving the blackout.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not launched details of the security event.
A security occasion normally involves a vulnerability and there are two severe vulnerabilities currently in the wile that were patched in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy has the ability to run harmful code on a server.
An advisory published in October 2022 described the impact of the vulnerabilities:
“A validated remote assaulter can perform SSRF attacks to escalate opportunities and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mailbox server, the assaulter can potentially access to other resources by means of lateral movement into Exchange and Active Directory environments.”
The Rackspace failure updates have actually not indicated what the specific issue was, just that it was a security occurrence.
The most existing status update since December 4th stated that the service is still down and customers are encouraged to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in dealing with the incident. The schedule of your service and security of your information is of high importance.
We have dedicated comprehensive internal resources and engaged world-class external know-how in our efforts to minimize negative effects to clients.”
It’s possible that the above noted vulnerabilities belong to the security occurrence affecting the Rackspace Hosted Exchange service.
There has been no statement of whether client info has actually been jeopardized. This occasion is still continuous.
Featured image by Best SMM Panel/Orn Rin