Originally posted on: https://agencyintelligence.co/wpx-hosting-steadfast-networks-outage
On the morning of August 2, 2021, Steadfast Networks' datacenter in Chicago, the Lakeside Technology Center, went down for 4 hours (for some, up to 5 hours). Several prominent web hosts were affected ... especially WPX Hosting.
One year ago, when the once-heralded SiteGround made huge cuts to their scope (and quality) of support, and raised prices when nobody asked for it, we found Terry Kyle’s WPX Hosting the most sensible alternative.
From partners’ recommendations, to the consensus in closed-group SEO masterminds, to independent tests from authorities in the SEO community, all roads led to WPX as the fastest & most reliable managed WordPress hosting, with the best support, for the best price.
Right away, we made the switch—and until this catastrophic outage, we were more than happy campers.
WPX Customer Support
One thing I have to say, their support has been amazing. Here are some things we've noticed dealing with them over the past year:
- I'd hop on chat, and within half a minute or less, someone would be online. 95% of issues would be resolved within no more than a half-hour; often, only a few minutes on live chat.
- When transferring an infected site, never rely on automatic antivirus scanners alone! In addition to free transfers, WPX provides their own pre-transfer scanning also for free; just ask for it. We transferred a client from A2 Hosting to WPX, as A2 failed their uptime commitment. Their site still had active traces of an old hack from the "mplugin.php" malware. Chat support performed an auto-scan which came up clean—and yet we knew the virus was still there. We requested a more advanced scan, which required escalation to a ticket. In this advanced scan, all traces of the malware were found and SUCCESSFULLY REMOVED. (We had another client who was hacked via WP File Manager, and SiteGround—who used to provide cleanup for free—asked him $200-300 USD for a similar service. He came back to us, and we cleaned it up at no cost. He gave us $100 for our efforts, and we didn't even ask!)
- The WPX hosting service and CDN are so blazing-fast (response time, TTFB and everything) that we never needed caching plugins for any of our sites. Ever. At first, we briefly used their recommended settings with W3 Total Cache, and found it introduced too much editing latency for our liking. We quickly found that our sites, all less than 30 pages, ran well even without it, provided they're built lightweight. (Our sites were rebuilt in Oxygen Builder, which we now do for our clients--and you, too.)
- Should their security block an external service, such as ShipStation emails for e-commerce, we'd send WPX the relevant data, and they would permanently unblock the process.
So What Did WPX Do Wrong? (Or what DIDN'T they do?)
After the outage at "The World's Largest Technology Center" (WPX's own words), not only did WPX customers (including myself) suffer for hours, WPX.net itself was down too, so we all had no way to get on our knees and beg support other than on Facebook and Twitter.
Search Engine Journal covered the incident. They made some glaring omissions:
- WPX has a CDN with 25 endpoints worldwide (10 in North America) to deliver their clients' content and reduce latency. While it's optional, the WPX CDN was & is enabled for all our sites. We were under the impression that there was built-in failover: that even if a datacenter fails, the CDN endpoints would deliver a live backup, and we'd still be good. This was NOT so. Instead, they only relied on Steadfast's redundancy service, which also failed. (If it was me, I'd have my own backup for EACH datacenter I'd use!)
- Communication is the best thing to have in a disaster. While WPX was down, support was pretty much nonexistent, except for a few messages.
- WPX hosts on 3 datacenters, the other two in Sydney and London. However, the Chicago outage affected customers in OTHER continents—Asia, the Middle East, Europe, and likely more. I'd think they would choose the Sydney or London datacenters due to proximity. But no matter where in the world, there was not a single report that anyone was online. On Twitter, someone in Munich told me that everything was down for him, too. Either one datacenter took down the other 2 with it, or some European clients chose the North American datacenter. There are many things that could have happened; it's anyone's guess.
- A Facebooker referred to a tweet that BigScoots, another web host, also relies on Steadfast / Lakeside, and experienced the outage ... but BigScoots ensured uptime using their own efforts at redundancy. (One reply: "I've been working on my sites all day lol. Didn't even know they were down.")
- SEJ preaches "patience is a not bad advice"—but when it IS possible for a host to properly prepare for a server outage, and users are running ad traffic to WPX sites, and they're risking being banned by Facebook & Google due to a website outage... can you really be patient? While it's still best to be calm, there's a lot to lose in this situation.
Huge props to @BigScoots this morning: Their datacenter is down right now (along with many other hosting companies)...but they are still online, thanks to their multiple, redundant tie-ins to the 'net.— Andrew Wilder (@NerdPress) August 2, 2021
Nice work, team! pic.twitter.com/r1RNYibVMO
This is hardly the first time we’ve felt the burn. (And no, I don't mean Bernie! 🤣) Year after year, hosts who were once king face a scandal, either due to a decline in service, or failure to prepare for a disaster—even when all the signs have said otherwise.
Our big takeaway from WPX and SiteGround?
Even when you find a host that must be “THE ONE”—the marketing shows it, all the (unbiased) reviews confirm it, and the (unbiased) tests solidify it—NEVER treat it as a guarantee that they’ll stay that way forever. Server outages are inevitable; how the hosts handle it is the best test of their reliability.
So what do you do?
Here's a real-world example. Every time I’d move into an apartment, I’d read the lease agreement in detail. The agreement would have an addendum about the included security features, and it would start by saying:
“Even the world’s most advanced security systems are not failsafe. Security systems can’t guarantee that someone won’t break in. Always use your own common sense and judgment, and act as if there IS no security system.”
Always have this same mindset when choosing your web hosts. No matter how good they are (and look), never assume they're failsafe.
Don't put all your eggs in one basket. In fact, you can spread and reduce your risk in SEVERAL ways:
1. Veto Your Host
Before signing up with a hosting provider, ask at least some of these questions:
- Do you use more than one datacenter in the country or continent?
- To ensure uptime, what do you do in the event a datacenter fails?
- Do you rely on the datacenter for redundancy, or do you have your own redundancy?
- If my website is hacked, what can you do about it? (Hacks happen more often than not.)
Most North American datacenters are centrally located in cities such as Chicago, Dallas and Toronto, etc.
2. Spread Your Sites Across Multiple Hosts
Most webmasters, entrepreneurs & small business owners have more than one website. I highly recommend having at least TWO (2) web hosts to spread your sites across. The more, the merrier.
The result: If and when one site goes down, it won't take down the rest with it.
Do your research, and find 2, 3, even 5 or more hosts!
3. Get Your OWN Auto-Backups
When even your hosting dashboard is down, what can you do about the backups you're paying for?
Having a secondary tool to also schedule auto-backups at the same time to outside storage—Google Drive, OneDrive or Amazon S3, for example—will ensure you can find and use a backup when you need it. Three of the most popular solutions for WordPress are UpdraftPlus (freemium), BackupBuddy (paid only) and BlogVault (paid with free trial.)
I am not affiliated with any of them; I'm just showing the options out there.
4. Have an Uptime Monitor
I don't care if anyone tells you "ALL uptime monitors are inaccurate." They haven't tested everything.
We use an external uptime monitor*, not a WP plugin or a plugin of any kind, that pings the site from 3 out of 4 continents of our choice (North America, Europe, Asia and/or Australia). We can set it to check in intervals of 30s to 5m+ & up.
The most important thing: when a site goes down, you're notified RIGHT AWAY, and you can confirm by going to your site.
And because it's 100% external—NOTHING running on or in your own server—bandwidth usage is practically nil.
We always aim that a site is never down for more than 15 or 20 minutes.
*OK, yes, that's an affiliate link. All this is simply from my own experience.
5. When You're Running Ad Traffic, Have a Backup Destination
In turn, you'll have more time to switch your ad campaigns so they don't go to inactive URLs.
When you plan an ad campaign, jot down a backup URL that you can fall back to, such as a Facebook page. If your site goes down, you have something to switch to. Sure, you may lose a bit of revenue, because it's not the primary target, but you won't lose as much as when your ad's going to a URL that's not working at all!
6. Host Your Site on TWO or More Hosts at Once
"Say what!?" Yes, I feel this is the ultimate solution. Why not have a carbon copy of your site on another web host? When one goes down, simply repoint your domain nameservers to the other, and be done.
Of course, that is the ideal situation. Most people will need to upload a backup to the other host. We are researching solutions to automate this.
Nonetheless, with any measure like this, even when it's imperfect, outages lasting hours can be easily mitigated. We're pretty sure there's more than one way to do this.
6+ years ago, when I was writing for a music reviews site, their host also proved unreliable. Yet not only did the webmasters have their own daily backups...even during an outage, they quickly went live again, running off a backup. If I remember correctly, they continued to do so until they located an alternative web host.
WPX, despite a history of excellent service, failed to prepare for a disaster. In some areas, they even put all their eggs in one basket, which went against their core business philosophy.
It's right to say that hosts don't have control over a datacenter mishap. That said, other hosts using the same datacenter did successfully keep their customers up by taking redundancy to themselves.
Even the best hosts can eventually make a big mistake. No matter how good they are, and no matter how much corrective (and preventative) action they take, you've got to prepare for a disaster yourself. It is unconditional.
Especially when you have a high volume of visitors & customers, having your own emergency measures will help you to keep serving your customers, and save potential revenue from being left on the table.
Not tech-savvy? Can't take website maintenance to yourself? No problem. You can talk with a website maintenance specialist about disaster preparation, and ask them to help you set up these measures.
And when that's not something they can offer, reach out to us. On top of disaster prep, you can get a custom maintenance package so your Web needs are looked after—from content, to updates, to speed, and more.
For more detailed advice about specific disaster prevention techniques, stay tuned in the coming weeks.
Alexander Vee, founder of Agency Intelligence (AI), Triple Agent Digital Media and Long Live The Hemp, is best known as a marketing consultant who helps Entrepreneurs & Professionals become positive agents of change—using his unique formula, “The ABC’s of Marketing.”
The young award-winning speaker & bestselling author has shared his message on the same stage as Martha Stewart and Ice-T & Coco at Carnegie Hall; on the same stage as Suzanne Somers & Caitlyn Jenner at the Harvard Club of Boston (where he made his public speaking debut at 19 years of age); at Nasdaq, Coca-Cola and Mercedes; and even at West Point, where his message to the leaders of America's future was: "If you're a 'best-kept secret', it's about as good as being bankrupt."
Alexander has been featured & quoted in ABC, NBC, CBS, FOX, Telemundo, The CW, plus other news publications all over North America - something he also helps his friends & clients to achieve!