When I visit Spokane, the Historic Davenport Hotel is by far my favorite place to stay. The architecture and craftsmanship of this 109-year old building is hard to match anywhere in America. And combined with its well-earned 4-star rating, amenities, excellent restaurants, cleanliness, and level of service, I find the value to be irresistible. I am a frequent guest and always stay here when visiting our Spokane-based customers.
This time however, I witnessed an I.T. fiasco that was article-worthy and ought to once again exemplify the foully of poorly-planned and misunderstood cloud implementations.
As I was sitting in their beautiful lobby, working on reviewing a contract for a meeting I had the following day, the hotel WiFi went down. I chuckled under breath as I turned on my cellular hotspot and went about my work. But my ears couldn’t help but overhear the conversations all around me. It seemed that it wasn’t just the guest WiFi – the whole the internet connection was down throughout the hotel. The front desk people were apologizing profusely as they asked people coming in with reservations to wait in the lobby because they were unable to check them in. The hotel is host to a sizeable restaurant, a sperate bar lounge, a lobby coffee stand, and a spa. Apparently none of these could operate, even with cash, and I watched on as managers instructed staff to put up “closed” signs everywhere. I didn’t take an exact survey, but I saw about 35 employees, effectively pacing and apologizing to customers for roughly six hours! I’d figure that the kitchen and back of house had to have had at least another 10 payrolls, or so.
My mind boggled at how an operation of this size, that is so utterly dependent on what is obviously a cloud-hosted system, could be so unprepared and ill-equipped for a simple internet outage.
I wonder who sold their cloud system to them and whether the question of, “what happens when the internet goes down?” was simply never asked, or if it was skillfully deflected? But here it was. Doing simple math: ~45 employees let’s say making an average of $20/h., with employment taxes and benefits overhead, for 6 hours… That’s about $7000 in just payroll losses. What their operational losses and losses of business opportunity might have been, is anyone’s guess. But rumors floated that their two sister hotels located just a few blocks away, were having the same problems.
To add insult to injury, when internet service was finally restored, the cloud-based system links did not come back up. There was a technical problem and I watched as hotel managers jogged back and forth frantically rebooting computers to no avail. I’m guessing that their gateway VPN was not configured correctly and did not automatically reinitiate the encrypted link with “the cloud”.
The losses from this one incident alone could have likely paid for six months to a year of redundant internet connectivity, equipment, and engineering time to design and implement a system that would be immune to this kind of outage. But the salespeople who sold them the cloud solution likely wanted the on-paper costs to seem that much more lucrative to make their pitch of “saving money” pencil into their pitch. Retaining UTS to evaluate that proposal would’ve without a doubt prevented this incident and cost a fraction of the loss. Who wants to bet me that this was not the first, nor the last time this will happen there?
The internet is back on and seven hours later, the world is right again, and I am able to order myself a beer.
Cheers!
