HN Gopher Feed (2017-08-26) - page 1 of 10 ___________________________________________________________________
If the Waffle House is closed, it's Time to Panic (2016)
249 points by brudgers
https://fivethirtyeight.com/features/if-waffle-house-is-closed-i...-closed-its-time-to-panic/___________________________________________________________________
jsjohnst - 6 hours ago
Craig Fugate told his Waffle House postulate in Nov 2009 at the
first Random Hacks of Kindness hack day.Video:
https://www.flickr.com/photos/jeremyjohnstone/4110293316/
toomuchtodo - 6 hours ago
Thank you so much for posting this. I recount this rule of thumb
frequently, and its nice to have a citation to refer to when
sharing with others.
impendia - 4 hours ago
I was in downtown Columbia, SC for this rainstorm:https://en.wikipe
dia.org/wiki/October_2015_North_American_st...The next morning, my
girlfriend and I ventured out on foot to attend a yoga class. The
yoga studio was closed, so we went to the Waffle House next door
and got something to eat. They told us that their water supply had
been compromised, but had made special arrangements so they could
make coffee safely. We enjoyed a hearty breakfast.We learned only
later that some areas of town had gotten two feet of rain, that
houses and businesses were underwater, that multiple dams had been
breached, that we should boil all our tap water for a week, and
that most people in town were without water entirely. My employer
(the University of South Carolina) shut down for a week.Indeed.
Waffle House doesn't kid around.
quickthrower2 - 2 hours ago
If you can't do yoga ... Eat waffles. You sound like me.
cjslep - 3 hours ago
I suppose it wasn't the Waffle House by Garner's Ferry Road and
I77? I have relatives near that part of Columbia and was supposed
to see them that early October but wasn't able to due to that
massive amount of flooding. It was unreal. Luckily, they love on
a hill and avoided a lot of damage.
impendia - 1 hours ago
The one on Harden Street in Five Points. That part of town, and
the whole downtown area, was mostly unscathed.
Shivetya - 2 hours ago
While not in the food industry; I work for a national auto parts
supplier which of course has more than just auto parts; large scale
natural disasters are something to behold. From the store teams who
check both company and privately owned stores for needs to staging
the warehouse with goods the local community will need as well as
the warehouse as well; that can be as little as a semi trailer
based generator.still as with all organizations what also comes up
is spinning off demand to other stores and warehouses. getting
those outside the affected area ready to help those affected and so
on.the key though is communication and you must develop a well
defined plan of how to communicate. this means from who, how often,
the methods of delivery, and with email it can me using specific
templates to make it readily apparent everyone is on
board.anecdotal, with family and friends of who volunteer for fema,
there are just "constants" that they also use for knowing when
things are getting better, some are restaurants and other are
simply commodity foods and goods.
alister - 1 hours ago
The mention of the Waffle House Index reminds me of the Big Mac
Index used to compare currencies. The idea is that a Big Mac, being
something of commodity made from common ingredients, should cost
the same everywhere in the world. If it is significantly more
expensive or cheaper in another currency, than that currency must
be over or under valued.https://en.wikipedia.org/wiki/Big_Mac_Index
jacquesm - 6 hours ago
That's a lovely study in contingency planning. This is something
that always strikes me when doing research in line of my work, how
many companies are simply totally unprepared for even the most
obvious things that can go wrong. It's logical, paying attention to
the happy path is where the business is, it is where you grow. But
if you only concentrate on the happy path you're a small step away
from a disaster.Now, natural disasters such as this one are much
harder to plan for and deal with than the issues that could affect
your average start-up. When you're moving and selling atoms the
chain from raw materials to revenues is a very thin one with many
weak links just waiting for the right conditions so they can
break.In contrast with that, in IT most disasters are man made and
easily protected against. And yet, few companies do.
[deleted]
protomyth - 5 hours ago
I get the feeling that national disasters are actually the easy
thing to anticipate. If you are on the coast, hurricane or
flooding. North Dakota is tornadoes and flooding in river
basins. Fire is an everyone problem.Its the weird crap that
really gets your goat. I remember finding out the power main to
our current place is located next to a T-section of road. Slide
forward too far and hit the power main. Brilliant placement. Or
when the primary, cannot fail contingency fails. A place I once
worked at had a >$1 Million UPS (generator / batteries lasting
weeks) for their data center. Semi hit the power main, and the
UPS failed to turn on.I actually got wrote up[1] as being
"paranoid" in my backup strategy for a government grant we had in
the 90s. The joke of it was that the site visitor who wrote me
up was a project director at another site. That site experienced
a flood and required extensive efforts to recreate their data. I
was young, and a bit vengeful and might have made some comments
during the next national meetup.Sometimes, a lot of disaster
recovery and contingency planning is politics above the level of
the people who should know better.1) They thought it was a
problem during a site visit and we had to write a response that
had to get approved.
jacquesm - 5 hours ago
> I get the feeling that national disasters are actually the
easy thing to anticipate.Their presence, yes. But the frequency
and intensity are hard to put any bounds on.> I remember
finding out the power main to our current place is located next
to a T-section of road. Slide forward too far and hit the power
main. Brilliant placement.Something exactly like that happened
with a gas line in Slupsk, Poland. The effects were
incredible.> A place I once worked at had a >$1 Million UPS
(generator / batteries lasting weeks) for their data center.
Semi hit the power main, and the UPS failed to turn on.Just
having a generator is not enough, you need batteries for the
ride-through period and you need to do periodic tests. Maybe
they did and it still didn't turn on?I'd take being called
paranoid as a badge of honor, better to be paranoid and up and
running than down and searching the yellow pages for data
recovery services.
vacri - 7 minutes ago
> down and searching the yellow pages for data recovery
services.Back when I was a medical tech, we lost a hard drive
with patient data on it. I ran a chkdsk with no luck, and out
of ideas, called a data recovery company. They stated it was
$3-5k to attempt to recover, and asked if we'd done anything
to the drive. "Just a chkdsk, nothing else". The guy on the
other end of the phone was genuinely annoyed by this. "I wish
you people wouldn't do that" delivered with that tone. Yes,
this guy's position was that if you ever have a disk problem,
your first port of call is to drop $3k+ at a data recovery
place, don't even attempt to do some preliminary checking.
protomyth - 5 hours ago
Oh, they had the whole line. UPS in the server room, with
batteries and diesel engine outside. Tested quarterly, I was
told. In server room batteries did their job, but the stuff
outside didn't. Took 4 hours to restore power and another 32
hours to restart the servers and verify their data.[edit:
about the site visit calling me paranoid] I was a bit ticked
off, but I guess someone who didn't even do the basics, as we
later found out, would not get the need.
jacquesm - 5 hours ago
Ok. Even a small DC that I'm intimately familiar with has a
back-up diesel that they test religiously on the 1st of
every month, they have the whole building running on the
genset for an hour or two and then they shut it all down
again. Iirc it failed to start twice in the years that this
setup has been there, which means the chances of a problem
will never be quite '0'.When I ran a gas station in North
America one of the first things I did was install a diesel
generator. My reasoning was that with on average 50K liters
of diesel in the underground tanks and the genset powering
the pumps it should be good for most problems that we'd
face. And in fact it was, which we managed to prove beyond
any doubt when all of the North-West of the US and a good
chunk of Canada went without power for a couple of days
within a year of installing that genset.Sometimes being
prepared is a waste of money and time in retrospect, but it
helps you sleep well, and sometimes it is the difference
between a community continuing to function or people dying.
Animats - 1 hours ago
The classic Bell System standard was that generators were
started once a month and run for several hours. Once a
year, each central office ran on emergency power for 24
hours, with external power shut off.In the entire history
of the Bell System prior to electronic switching, no
central office was ever offline for more than 30 minutes
for any reason other than a natural disaster.
aidos - 5 hours ago
A place I once worked had their servers in-house. They had UPS
on the roof, duel everything (fibre links etc). Apparently
there wasn't a single point of failure in the system.One day,
the servers just vanish. We go into the server room to check.
Someone had plugged all the machines into the same multi box /
power extension cable and the fuse in it had popped.Sometimes
all your planning can be undone - especially once humans are
involved :-)
jacquesm - 5 hours ago
> Apparently there wasn't a single point of failure in the
system.Redundant power supplies on alternate phases is a
pretty common element in designing any kind of server setup.
So this wasn't a setup with a 'single point of failure', it
was a setup with a very explicit single point of failure
(several, in fact).Someone linked to this excellent slide
deck yesterday:https://c3.nasa.gov/dashlink/static/media/othe
r/ObservedFail...Worth an hour of your time, not to make you
paranoid but more to make it plain how hard this sort of
planning really is.
sogen - 1 hours ago
Thanks, great stuff, like the story of the email that
couldn't be sent farther than 500 miles.
protomyth - 5 hours ago
I remember a data center that supposedly had dual fibre links
to the outside. Found out later (after someone cut the
fibre) that they actually linked back up just outside of
town. I understand lawyers were needed.
makomk - 1 hours ago
A certain important data center near here has dual fibre
links, as mandated by their redundancy requirements.
They're installed along both sides of the same major road.
dmurray - 1 hours ago
That's not so bad. The typical mode of failure for fibre
lines is not flooding or tornadoes or landslides or
anything else that would affect both sides of the road at
the same time. It's someone digging up the road, which is
unlikely to happen on both sides simultaneously.
FooHentai - 54 minutes ago
'Fully Diverse Paths' is something we take great pains to
verify with our WAN provider, for exactly that
reason.Because it is said that if you are heading into the
woods and worried you might get lost, take a length of
fiber with you. If you get lost, simply bury the fiber in
the ground. In five minutes a man in hi-viz with a backhoe
will come along and dig it up.
jacquesm - 36 minutes ago
In a pinch, a piece of CAT5 will do.
hexane360 - 3 hours ago
>Fire is an everyone problem.That's not even remotely true.
This is another one of those weird HN tunnel
visions.http://lakedillonfirerescue.org/wp-
content/uploads/2013/03/w...
protomyth - 3 hours ago
Last I checked, everybody's data center or building can catch
fire or have a fire inside. I would love to see the building
that has computer equipment in it that cannot catch on fire.
You are one bad UPS / power supply away from a rack fire.
Never mind the electrical panel itself.
ghaff - 3 hours ago
The context was natural disasters though--so pretty much
wildfires.
protomyth - 1 hours ago
And every place even those listed on the map can get them
unless you build your data center on some terrain with
absolutely no plan life around. You need to deal with
fire outbreaks even if you are not on the map.
jacquesm - 1 hours ago
And even then there is lightning induced fire. Plenty of
natural fires start that way and there is no reason why
your building can't be the starting point of one. Though
fortunately most DCs take lightning protection quite
serious and are probably amongst the harder buildings to
ignite.
FireBeyond - 2 hours ago
It's not your problem until it is.I live in a "green" area,
nice low threat.I'm also a volunteer firefighter. Last week I
was at a brush fire in a semi urban area that got so bad so
quickly that it prompted three choppers and the first time in
this half of the state that a DC10 VLAT (very large aerial
tanker) dropping retardant was deployed due to the threat to
multiple residential and commercial structures, several of
which were still lost.In the lowest threat classification
area.
thesmallestcat - 5 hours ago
> In contrast with that, in IT most disasters are man made and
easily protected against. And yet, few companies do.If I'm
competing in a high growth industry against somebody who is busy
planning for the worst, I'll move twice as fast and put them out
of business before their plan can matter.
Applejinx - 3 hours ago
And this is why Silicon Valley isn't always trusted.Yes, you
probably can, and more often than not your gamble will work.
You'll move fast and disrupt and break stuff and odds are you
can even say, up front, 'I'm moving fast to give you more
without that boring planning stuff!' and may still be able to
put the other guys out of business: depends on how well they
handle it, and how hard you can push.And then the natural
disaster strikes and you're moving fast again, to make an
insurance claim (or get out of town!) and the people who in
turn gambled on you, are boned. Perhaps very catastrophically
boned. And it's their fault, because let's say they were told
up front that it was a gamble.But people are dumb.And you can
put a lot of more responsible community oriented business-
operators out of business before their planning matters,
because people are dumb.Depending on what you do (taco stands?
Hospitals? Building dams and bridges?), your very Silicon
Valley approach could be anything from colorful to criminal. I
will hope the bridges I drive on are 'planned for the worst'.
But there's only the inertia of social expectation, and
whatever exists in terms of enforceable legislation and
policing, to ensure that. And you can always violate
legislation as part of your 'move faster' strategy, and expect
to just deal with it somehow later. So that leaves only the
inertia of social expectation.
FooHentai - 53 minutes ago
DM:SVDidn't matter, still vested.
snovv_crash - 5 hours ago
Only if it is a winner takes all industry, and only if nothing
bad happens before they go out of business.Lots of industries
have clients that actually care about the robustness of their
service providers as well; if in the future, once something
invariably does go wrong, you have effectively just signed your
own death warrant if you can't recover quickly enough.
jacquesm - 5 hours ago
> I'll move twice as fast and put them out of business before
their plan can matter.Yes, maybe. Or you'll be put out of
business because you did not have a plan.You could attempt to
justify your stance by saying that in the growth phase of the
company nothing else matters but the reality is that you can't
really retro-fit the attitude and set-up that your future
company will require once you get there if you didn't do this
from the beginning.See, you can overdo anything, including
security and contingency planning. But that isn't what this is
about. It's about knowing that you are taking a conscious risk,
knowing what the consequences could be and knowing that if
those risks do materialize what you will do.If some of those
paths end with 'game over' then so be it. But there is no
excuse for not knowing and it is exactly those things that you
did not know about that will kill you.So much good luck with
moving twice as fast, headlong into a wall that you could have
seen coming.
[deleted]
hluska - 4 hours ago
Not only are you unnecessarily rude to a very intelligent
community member, but you demonstrate a complete lack of
respect for your customers. Please change your attitude.
jacquesm - 5 hours ago
> I could attempt to justify my stance? Thanks, jacquesm,
you are truly magnanimous.No need for personal attacks,
this is exactly the attitude that you displayed two
comments up thread.> To be clear, I'm saying that no risk
outweighs slow growth in a high growth industry.Very very
few of us will ever be in a position where this is the
reality and for those of us who are the decision still has
to be a conscious one.> Nothing is more likely to kill your
business than standing still.That just isn't true. I've
seen tons of businesses fail, the majority of the causes of
death can be attributed to:- founder conflicts- product /
market fit issues- wishful thinking- bad financial
planning- wrong investors- sloppy execution- existential IT
issuesI've yet to come across a single business that was
killed by 'slow growth' in isolation, usually slow growth
is an indicator something else is amiss.> So I'll enjoy
hurtling towards walls, and shoot for nine nines after
securing funding over my competitor, etc.Why would you
change your stance after securing funding? After all,
growth matters even more after securing funding than it
does before.
MBCook - 4 hours ago
How many businesses are one back-up failure away from
losing all their customers?So many people don't even TEST
them until it's needed, and just hope they've done it
correctly.
jacquesm - 4 hours ago
Enough that I get called out at least once per year to do
a rescue attempt. Good for me: they usually work. Bad for
the users: they don't always work and they're not always
perfect. Of course all of these companies retrospectively
agree that maybe they should have had their back-up setup
tested a bit better. But that's hind sight, and I'm quite
convinced that prior to failure there isn't an argument
in the world that would have swayed them to actually do
it.And so the cycle continues. I probably shouldn't work
too hard to educate people about this since it negatively
affects my bottom line but those very same people
definitely carry insurance on their real estate and their
vehicles.A backup that you haven't tested (and with a
test I mean to rebuild your business from the ground up
using nothing but the backups) is most likely
incomplete.Bonus points if you remember to store the
decryption key outside of your backups ;)
[deleted]
sctb - 4 hours ago
I don't think this is up to the standard of civil
discourse here. Can we please try not to be so rude to
one another?
[deleted]
stephengillie - 5 hours ago
Contingencies are a type of Plan B. And when you put energy
into your Plan B, you rob energy from your Plan A.
ChuckMcM - 5 hours ago
I read an excellent article on Sear's logistics after it was
mentioned here. The parallels between keeping products flowing
and data centers operating was quite striking. Multiple suppliers
for key material, multiple transportation technologies, a series
of plans which would keep you available as you fixed an
underlying problem (defense in depth).It is a pattern or process
that repeats itself in everything from keeping web services
online, stores stocked, casinos from being robbed, and even
individuals from being dumped into poverty by events. I expect if
there was a book "Everyday Prepping" it might actually sell
pretty well :-)My take away from the article however is that it
is better to choose indicators that have a wide dynamic range
:-).
Mz - 2 hours ago
I am a huge fan of Waffle House. I love their hash browns. There
are none on the West Coast. It is one of the few things I miss
about the East Coast.I am incredibly impressed. They tend to look
like glass boxes,* typically with at least two outer walls of large
windows. One back wall is bricked in on the kitchen side and one
end wall where the bathrooms are. The front end may have glass on
three sides. This is where extra seating goes.They typically have a
galley kitchen with bar seating in the middle and booths to either
side that are readily served by cooks and wait staff, plus that
additional seating area towards the front (towards the main road,
usually).They have sort of a bad reputation as a dive restaurant,
probably in part because the building looks so much like a
glorified trailer and the prices are low, but the food is good and
I have always loved the chain. My dad grew up on a farm, so I come
from humble people who like down to earth places like this. Reading
this article makes me feel really good about being a huge fan, in
spite of the classist contempt some people have for the chain.You
would think that a place that (very often) looks like a glass
trailer would not be the first thing open after a hurricane. So, I
am astonished, but happily so.*
https://www.bing.com/images/search?q=waffle+house&go=Search&...
Animats - 5 hours ago
Waffle House has a map of their locations, but it doesn't show
which ones are down. Waffle House #1260, on Galveston Island one
block from the beach, is probably in the worst position.
ryanmarsh - 1 hours ago
This is very interesting and mostly true but I think I speak for
all coastal Texans when I say it's not really a good measure of how
we're doing, especially those of us that have lived through several
hurricanes.
10000100001010 - 6 hours ago
That's my favorite Waffle House[1] (It was reconstructed after
Hurricane Katrina)! I used to eat there all the time and watch the
shrimping boats out on the gulf.This comment doesn't have much
purpose other than one of those small world moments.[1]
https://espnfivethirtyeight.files.wordpress.com/2016/12/prom...
Exuma - 5 hours ago
Im sorry. I bet that is weird to look at the photo then. D: