HN Gopher Feed (2017-07-07) - page 1 of 10 ___________________________________________________________________
Good at programming competitions does not equal good on the job
[video]
218 points by jpn
http://www.catonmat.net/blog/programming-competitions-work-perfo...___________________________________________________________________
rkunnamp - 3 hours ago
Just like 'There is no such person as Good CEO', but only 'A Good
CEO for a particular company, at a particular point of time', there
is no such a person as 'Good Programmer', there is only 'A Good
Programmer for a particular job, at a particular point of
time'.Context matters a lot. One does not use an AK-47 to kill a
mosquito. It is terrible for that job. But that does not make AK-47
a terrible weapon.Good at programming competitions does not equal
good at any programming job, is a more appropriate sentence.
dsfyu404ed - 2 hours ago
Have you tried killing mosquitos on the back of a deer[1]
standing behind a tree at 400yd. Even the AK with it's heavy
cartridge isn't enough. You want a full power rifle, especially
if the tree is a hardwood. /s
kpil - 1 hours ago
I think that there are personal traits that make people perform
good in most situations, but perhaps not all extremes. As
continuity is normally good, you try to find those, and change
only if you really have to.A programming competition evaluates
almost none of the traits thats doing a marathon run as a highly
preforming team.
agounaris - 4 hours ago
Winning the dunk competition does not make your team be an NBA
champion...
rdiddly - 3 hours ago
Another sports metaphor: Being a good sprinter doesn't make you a
good marathoner.
BoiledCabbage - 3 hours ago
But winning the 3pt shooting contest is a pretty good proxy for
shooting ability.
paulcole - 4 hours ago
Also important to remember that good at programming does not
necessarily equal good at job.Excelling as a communicator, being an
empathetic person, and having great interpersonal skills are just
as (perhaps more) important than how well you can code.
potatolicious - 4 hours ago
It continues to surprise and frustrate me that as an industry we
continue to highly prize proxy signals for engineering skill, when
engineering skill is so directly measurable.Why even bother with
measuring things that are N-degrees removed from actual
engineering, when you can just get people to engineer things?I know
others on HN have been hammering this point home for years, but
until something changes it deserves to be repeated ad nauseum: work
samples work samples work samples work samples.Stop with the trivia
questions. Stop with the contrived algorithms questions. Ask people
to design systems, ask people to defend their designs, ask people
to write real runnable code that directly relates to the work they
will be doing at your job, ask people to review a real piece of
code written at your company, ask people to critique real design
produced at your company. Anything but what we're doing right now.
misingnoglic - 2 hours ago
And with a work sample, how do you know:1) how well they work
with others?2) how long it took them to come up with the
solution?3) if they actually came up with the solution themselves
BatFastard - 2 hours ago
Totally agree, and if someone does not have code they can share,
they have no passion for programming. Therefore no job with me.
[deleted]
williamsmj - 3 hours ago
It's not that simple.Many good engineers, especially those who
aren't young men, don't have existing work samples they are able
to share. Perhaps, by choice or otherwise, they have a personal
life that does not allow them to write code for free. And their
employers won't allow them to share work code.And many good
engineers are not willing to spend an unpaid weekend writing code
auditioning for a job they aren't yet sure they want.And, as you
know if you've ever worked with a disruptive colleague, or had
customers, engineering in a team for a product is much more than
just writing code.
angersock - 3 hours ago
> don't have existing work samples they are able to share.This,
at least, should be fixable.Don't work at places that won't let
you talk about your work and demonstrate your competence.
pm90 - 3 hours ago
I don't think that's always possible. What if you worked for
Healthcare or Security or Finance, where there are strict
laws for keeping code/design private?
angersock - 3 hours ago
What are examples of such laws? There are laws allowing
prosecution if you break agreements, sure, but the
agreements themselves are what can be fixed by encouraging
the market that they're incorrect.
DanBC - 2 hours ago
In the UK:
https://en.wikipedia.org/wiki/Official_Secrets_Act_1989
angersock - 1 hours ago
That seems targeted at .gov employees, which is a little
different.
tptacek - 3 hours ago
That's not what a work sample is.A work sample is a (usually,
hopefully) standardized piece of work you request from all
candidates that mirrors the actual work they'd be doing on the
job.There's no reason a work sample needs to be so onerous that
it costs you a weekend. What people forget when this comes up
for discussion is that interviews are work. In fact, they are
themselves onerous work, since they require you to be on-site
and intensely engaged in ways you don't have to be to do the
actual work of a job.A work sample, on the other hand, can be
done from your home, with a beer next to your laptop if that's
how you roll.The notion of audition work has definitely been
abused in our industry. People get work samples that aren't
standardized. They get work samples that are later ignored.
Work samples don't have objective scoring rubrics. Some
annoying companies assign new features for their products as
"work samples".Done carefully, though, with objective and
predictable grading and calibrated to offset in-person
interviewing time, they're superior to any process we have.
ryandrake - 3 hours ago
As an applicant, I'd be all for going the work sample route,
but only if it's paid for and if it were used in lieu of the
typical face-to-face hazing session rather than in addition.
I notice more and more companies are asking for work samples
(or "homework assignments" or whatever they want to call
them), but they just pile it onto their already onerous
process.The major pain point in interviewing (at least for
me) is the massive time sink for a very small chance of
success. I'm spending a few hours on the application, then a
few more doing phone screens, then a weekend doing a work
sample, then I blow a VACATION day (which has monetary value
and of which I have only 10 per year), and at the end of it
all who knows what my odds even are? Multiply that by, lets
say 10 companies per year, and: I've invested 31 days of my
life, have no vacation left, and I still may end up with
nothing.
bdcravens - 2 hours ago
> I still may end up with nothingWhat do you end up with if
you filter out companies that ask for work samples?
ryandrake - 2 hours ago
1 additional free weekend for every one I don't do.
bdcravens - 2 hours ago
I'd definitely say that if a company is handing out work
product assignments that take an entire weekend, it's
somewhere you don't want to work.
tptacek - 3 hours ago
I think paying for interviews is problematic for a bunch of
reasons.On the other hand, I agree wholeheartedly with
people who reject "homework assignments" layered on top of
a standard interview process.The onus is on people who want
to take advantage of work samples to:* Ensure that the time
they're asking of candidates is offset by lowered time
demands elsewhere* Ensure that they're using the work
samples objectively, so that people aren't asked to do
coding work as part of a crap-shoot application process.The
process we used at Matasano and NCC was less demanding than
typical job interview processes. The challenges were simple
and self-contained, and when they were completed we could
tell you with a decent degree of confidence whether you
were likely to be offered a job at your (shortened) on-site
interview.
williamsmj - 3 hours ago
As I said, many good engineers are not willing to spend
unpaid time auditioning for a job they aren't yet sure they
want, doing work they can't reuse, by taking a formal
standardized test.It may in fact be a better hiring practice
in terms of its ability to predict job performance. But you
will _lose good candidates_.And in particular, you will lose
experienced candidates. You are embedding biases toward
younger (and therefore less experienced) male coders in your
hiring practices if you require what is, from their point of
view, free work. (You can see the biases in your "with a beer
next to your laptop" remark, for example.)
bit_logic - 3 hours ago
I'm an experienced engineer (10+ years) and I would favor
this process way more than the current algorithm/DS
interview. It's the algorithm/DS interview that really
favors young graduates. They already have an advantage
because college knowledge is still fresh in their minds.
And they often have the free time to grind leetcode for
months. Experienced engineers often have families or other
priorities and free time is hard to find.
poletopole - 3 hours ago
Usually the work sample is something simple that doesn't
take longer than a day or two. Many interviews that don't
do work samples can go on longer than that and accomplish
nothing. I was hired after doing a work sample that took a
day. I was still paid for my work, irregardless of being
hired. The work sample wasn't "standardized", just a
typical task one of the other developers would have done
otherwise. So I would encourage any employers reading to
try this process out--it's better for both parties.
desiderantes - 3 hours ago
>You are embedding biases toward younger (and therefore
less experienced) male codersYounger maybe, but where does
the male part come from?
williamsmj - 2 hours ago
Less likely to have personal obligations to others, e.g.
family. Less likely to feel social pressure to do time-
consuming emotional labor outside of work.
tptacek - 2 hours ago
How is the standard interview process, which often
requires candidates to submit to multiple rounds of in-
person interviews, sometimes even requiring travel,
better than a process that people can do mostly from
their homes?
tptacek - 3 hours ago
You updated your comment after I replied to it, and so I
updated mine.Source: I ran a work-sample recruiting process
for the largest software security company in the US (after
it acquired my startup, which was one of the top 5 in the
US), and our process most definitely was not biased to
younger workers. Work samples drastically improved
diversity.EditYou've edited your own comment again, which
makes you very hard to respond to. If you want to rebut
anything I've written so far, can I ask you to copy it into
a reply to this comment?
williamsmj - 2 hours ago
Sorry, new commenter, unfamiliar with the etiquette of
the edit button. I'm done.
chucksmash - 3 hours ago
I agree with you somewhat, but couldn't you just as easily
replace "unpaid weekend writing code" with:- unpaid weekend
updating resume- unpaid weekend practicing algos- unpaid
weekend browsing job sitesjust as easily? I don't think
asking job seekers to be interested enough to give a work
sample is an undue burden. If that filters out people
playing the numbers by applying to every opening...is that
such a bad thing?
ng12 - 3 hours ago
It's a problem of scale. Spend half an afternoon skimming
through CtCI (shouldn't need much more as a seasoned
developer) and you're good for N number of job
interviews. I'll take that rather than spend 4*N hours on
take-home problems in addition to time spent on-site.
aNoob7000 - 3 hours ago
It depends on how far along the interview process you've
gotten to with a potential employer and if this is a step
away from being hired.If the potential employer is asking
for code samples up front, I think that's a bad sign.
Let us talk about the work environment, expectations of
the position, and then we can focus on my tech skills.
morgante - 2 hours ago
If I spend a weekend updating my resume, practicing
algos, or browsing job listings, then that time is useful
for all jobs I'm applying to and interested in.If I spend
my weekend on a work sample assignment, it's only
applicable for that one company which might very easily
not give me a compelling offer or be somewhere I'd want
to work.At this point, I'll only do work assignments if
I'm already very familiar with the company and have
reason to suspect they'll give me a compelling offer.
potatolicious - 3 hours ago
I don't mean to propose that people spend their off hours
writing work samples - that's unreasonable for all the reasons
you've given and more.What I mean to propose is that we
dramatically alter the current on-site interview process used
by most companies.Instead of 5-7 interviews, consisting of 5-7
independent but ultimately equally contrived algorithms
questions on a whiteboard, use the several hours you have the
candidate to produce real, working code that reflects the work
that your company does.Which is to say, this requires no more
time commitment than the existing interview processes at
typical tech companies.If the work your company does is deeply
algorithmic, this will be reflected in the work sample produced
via this process. If the work your company does is more heavily
UX-oriented, this will be reflected in the work sample
produced, also.Instead of "over the course of a full-day on-
site round of interviews, candidate produced several short
snippets of hand-scrawled code solving various CS textbook
problems, which we will now use to infer general engineering
ability"...... you can say "over the course of a full-day on-
site round of interviews, candidate designed and produced a
module of code that does [X small thing that company does],
which can be assessed with the same rigor and methodology we
already apply to our existing work and employees"
sidlls - 3 hours ago
While I agree that a focus on "work samples" is inappropriate,
I must emphatically agree with the parent's broader point.The
way companies in this industry conduct interviews and measure
performance is like an aerospace engineering company testing
candidates for satellite engineering on their understanding of
the Standard Model (physics) or group theory or some such.
There's too narrow a focus on what is required of an engineer
in the software industry.
kenning - 3 hours ago
I am currently reading "hackers: heroes of the computer
revolution" and it seems like this culture of optimizing
algorithms ("bumming") comes from the extremely early days of
programming, when people were laying the groundwork for all the
basic functionality of CS.
kafkaesq - 5 minutes ago
Well it's not just that -- algorithms do matter, and a very
small number of programmers do move the needle in this industry
through their ability to shine in that area.But the vast
majority of the time -- that's not what your company needs at
all. You need someone who's smart and reliable, and more to
the point, believes in your mission. And who will go in and do
all that far from algorithmic, highly unglamorous stuff that
keeps your business from going underwater, day in, day out.But
of course - those skills are difficult to evaluate in a short
amount of time. So instead companies go for a skill that can
be "measured".A skill like -- you guessed it -- algorithms.
teen - 3 hours ago
I actually like the current interviewing approach... and I feel
like we are a silent majority. I think whiteboarding / coding
algorithm and design problems is pretty fair and reflective of
engineering skill.
sidlls - 3 hours ago
It is reflective of skills at internalizing and applying
undergrad (and sometimes graduate) level academic trivia in a
familiar context. The interview process you and other "silent
majority" members think is effective at measuring engineering
skill is not, actually.I really wish you and so many of your
colleagues would spend a few weeks with engineers in the
physical science engineering disciplines. What we do in this
industry is a farce, with respect to engineering.
gautamdivgi - 3 hours ago
The current approach has a huge false negative rate. It keeps
out the bad apples but hugely favors marking a lot of
good/great apples bad as well. The time invested to study for
these interviews is quite onerous in itself. So, I'll say I'm
not a fan of the approach.That being said we do use the method
where I currently work to interview. However, the problems are
not some made up situation or a test of data
structures/algorithms but a set we've encountered in real
life.From what I've heard the whiteboard method isn't really
popular with many companies today either but everyone is
sticking to it until some other alternative which avoid false
positives presents itself.
tptacek - 3 hours ago
You're the overwhelming majority, which is frustrating, because
the method you're advocating for empirically and clearly does
not work well.
stupidcar - 3 hours ago
How is whiteboard coding reflective of engineer skill? At what
time, during any job you've had, have you needed to instantly
write code on a whiteboard, in front of peers, in a matter of
minutes. Almost by definition, it's not reflective of
engineering as actually practiced in a job.I'm a far bigger fan
of at-home coding exercises. Yes, I know, some people get
annoyed at these because they think you're asking them to do
free work in their spare time. But what other way is there to
test people's coding under circumstances that most adequately
mirror those of an actual job. E.g. they have a (relatively)
unlimited amount of time, they have access to Google, their
IDE, etc.I'd much rather see what someone can come up with, in
response to some novel problem, based on a couple of days
programming, than what they can hack out on a whiteboard based
on thirty seconds of thinking. The former seems closer to what
coders are actually required to do, day on day.
DavidWoof - 3 hours ago
> At what time, during any job you've had, have you needed to
instantly write code on a whiteboard, in front of peers, in a
matter of minutes.Well, constantly, actually. Although I'm
thinking of designs and snippets rather than actual
functions. I guess it depends on what you're asking people
to whiteboard during the interview. I agree that asking
people to whiteboard qsort is silly, but walking through
design alternatives with occasional code snippets to
illustrate implementation options is a pretty basic skill.>
based on a couple of days programmingEither your company is
very well-known and very attractive to candidates, or this is
going to incredibly restrict your candidate pool.I think
smaller work samples are a great idea, I think code reviews
are a great idea, but asking for two days sounds like a bit
much, especially early in the process.
kafkaesq - 14 minutes ago
Although I'm thinking of designs and snippets rather than
actual functions.But that's the thing -- at whiteboard
interviews, they don't ask you to produce "designs and
snippets". They make you write actual working classes and
functions.
recursive - 3 hours ago
> At what time, during any job you've had, have you needed to
instantly write code on a whiteboard, in front of peers, in a
matter of minutes.Fairly regularly. Like when I'm
explaining to someone how something works after they've asked
me about it.
sidlls - 2 hours ago
You regularly have to instantly write code for a problem
from undergrad CS coursework on a whiteboard, and for which
you haven't necessarily been involved in working on, in a
constrained time with your job hanging in the balance?I'm
skeptical.
thatswrong0 - 3 hours ago
How do you think that? I think myself a decently productive
engineer and I always loathed the fact that I have to study for
most of my interviews. Study things that I have almost never
used in my actual engineering career. In a situation nothing
like actual engineering work.My favorite interview process
(given by the company I currently work at) consisted of one
coding prescreen that wasn't terribly difficult, one session
where I simply talked with two engineers about past projects I
worked on, and then one higher level architecture problem that
involved pseudo-code but not some esoteric algorithms.That
interview process, I think, is way more representative of my
actual day-to-day work than any algorithm interview I've done.
It deemphasizes the ability to figure out / reguritate
previously memorized tricky algorithms on the spot (when do you
ever need to do that during the work day?) while emphasizing
the ability to communicate how and why you make decisions.Which
is an incredibly important part of engineering, way more
important than whatever skill whiteboard algorithm interviews
test.
DavidWoof - 3 hours ago
What's ironic to me is that a lot of people are going to see this
video and just start using programming contests as a negative
proxy signal for engineering skill.
fatjokes - 4 hours ago
I'm also a former ICPC world finalist and I'm willing to believe
this is the case in a BigCo because: 1) Top ICPC competitors tend
to be extremely socially awkward, and may not work well in groups.
2) Vast majority of engineering work is very algorithmically
simple, thus these folks don't get a chance to "shine".However,
I've noticed that they are popular in prop trading firms, where
work tends to be in very small teams or individual. I don't know
how their performance correlates to fund performance.If I were
hiring, I'd still prefer to hire at least some top ICPC performers.
The hard algorithms are rare---but can make or break your product.I
also think the knowledge learned from programming contests is
invaluable. I'd like to be able to discuss bipartite matching or
min-cut with my colleagues without eliciting a blank look.
richardknop - 45 minutes ago
This is probably true. I agree. I would aim for balanced teams,
some top ICPC performers balanced with people who are better at
solid design, creating maintainable codebase and working with
business people to define requirements.
pera - 3 hours ago
It depends on what position are you trying to fill: if your
business have various hard and general algorithmic problems that
you need someone to solve then, hiring an ICPC regional champion
would probably be a good thing. But this is not always the
case...I wouldn't hire an Olympic medalist runner to do pizza
delivery because: 1) the person will probably get really bored
(and may quit or perform bad after some time), and 2) the
delivered pizzas will probably be a mess inside the box.I believe
programming competitions would do itself a favor changing its
currently ambiguous name to algorithmic competitions. Then
engineers, who are also programmers, would be ok with it and we
would stop having this kind of threads every few weeks.That being
said, I personally enjoy competitive programming and I do agree
that the knowledge you acquire is invaluable :) and I also think
that most engineers should practice with online judges now and
then to be better at their jobs.
pizza234 - 2 hours ago
> I wouldn't hire an Olympic medalist runner to do pizza
delivery because: 1) the person will probably get really bored
(and may quit or perform bad after some time)I think it's a bit
more complicated than that. What if Olympic medalists were
conditioned to think that delivering pizza is the coolest job
that exists? What if their scooters have massaging saddles and
5.1 hifis?I definitely agree this market (segment) has, in a
way, a very misled attitude/approach, but definitely, I think
there is an (evil) art in marketing such workplaces.
_0ffh - 2 hours ago
> Vast majority of engineering work is very algorithmically
simpleanecdotal_evidence+=1 : Am engineer, only algorithmically
interesting task I had the last three months was to quickly find
all minimal solutions of a given instance of a certain class of
constraint graph problems. That's about average, at my job. Four
algorithmically interesting things a year. And I guess I might
even be one of the luckier engineers.
kwillets - 1 hours ago
While these problems don't come up frequently, I've found that
in a typical project anything involving a graph has a good
chance of being implemented wrong.
0xfeba - 2 hours ago
I just match APIs up 60% of my time. 20% in 75% useless
meetings. And another 20% yak shaving.
joenot443 - 3 hours ago
I agree for the most part, but I think it really depends on the
product your team is building. I can think of at least a couple
companies I've worked at where the hard algorithms weren't rare,
they were actually non-existent. The product's success relied on
user experience, a solid design, and exhaustively tested and
maintained code.The reality with many companies these days is
that the algorithmically hard problems are solved in the
frameworks and libraries they use, it's simply not necessary for
most engineers to understand their inner workings.I'm curious,
would you prefer an engineer with solid software engineering and
design knowledge, or one with minimal experience building real
software, but very in-depth algorithm and CS theory knowledge?
fatjokes - 3 hours ago
> I agree for the most part, but I think it really depends on
the product your team is building.I completely agree. However,
1) Google---whose CTO is the originator of the claim in
discussion---definitely works on hard algorithms and 2) you
never know how your product could grow. A feature which could
get dismissed as "impossible" might be implementable by the
right talent. Presumably these teams don't need to be full of
top algorithm folks.> I'm curious, would you prefer an engineer
with solid software engineering and design knowledge, or one
with minimal experience building real software, but very in-
depth algorithm and CS theory knowledge?Depends on the size of
my team and what I'm trying to deliver, I suppose. In general
I'd aim for a balance in the team, maybe 3/4 engineering/design
+ 1/4 algorithms. I feel like it's easier to learn design
patterns than how to use algorithms creatively.
sltkr - 3 hours ago
> 1) Google [..] definitely works on hard algorithmsTrue, but
Google also has over 20,000 engineers working on various
things. I think only a tiny minority of those are actually
working on hard algorithms.
jaggederest - 3 hours ago
It's like only hiring NASCAR drivers to drive taxis because
"they might need to go fast".
BatFastard - 3 hours ago
Indeed, for Taxi drivers its better if they go slower and
entertain the guests along the way.I have had top
algorithm people work for me and they can do make magic
happen. But they tend to have the quirky personalities of
Wizards. I once had a amazing code ask me if he could so
some side work so that he could pay off his credit cards,
sure no problem. Next week he came back and told me how
he had bought a new 10k telescope....
fatjokes - 3 hours ago
Exactly. They don't just hire programming contest folk.
Most of the people they hire are those with engineering
experience, etc.
raverbashing - 3 hours ago
But as much as you need a good algorithm, you need it to be
testable, reliable and predictableThis involves some team work
lordnacho - 3 hours ago
Aren't contests normally under huge time pressure? I would think
that a lot of people could solve even quite hard problems given
ordinary conditions like having lots of time, colleagues, and
reference materials available.
jmcgough - 1 hours ago
At the same time, if you've never seen a solution that uses
something like DP, you're not going to see a DP problem and
think "oh, I can use DP for this". Part of it is ensuring that
programmers have awareness of the tools at their disposal, so
they can pull them out when they're under a deadline to get
something done on time that's performant
YZF - 2 hours ago
I used to spend some time on TopCoder and got to be fairly good
but not at the top "red" level... There is some transfer to
improved code quality but I think it's generally not something
that made a huge difference to what I do as a developer.There are
plenty of people who can do really hard algorithms but may need
more time or research. Real life doesn't always present you with
the same canned problems that are used in competition and you're
not operating under the same constraint. A competition problem
typically starts from an algorithm, somewhat like a comp-sci exam
question. As a competitor you need to quickly recognize the
algorithm and then quickly write an implementation. I'm sure a
lot of the best competitive people would draw a blank when
presented with a problem that isn't a well known algorithm and
certainly wouldn't be able to solve it within the time
constraints.Basically it's a game.What I'd say is that
outstanding performance in programming competition probably
correlated to some degree with intelligence. Intelligence
correlates to some degree with being a good software developer.
hrktb - 3 hours ago
To add on point 2), most big companies have a strong bias against
'smart' solutions. If something is too complex to be fully
understood by half the engineers:a) it won't be trustedb) it
won't be maintainable by any random staff, which puts an
additional risk on choosing that solution.The maintainability
argument will usually be enough to can any idea that only few
people are comfortable to make evolve.
collyw - 1 hours ago
The interview process would indicate the opposite.
eastWestMath - 1 hours ago
Companies don't have 2 hour time limits on solving their hard
problems - if you run into hard algorithmic problems in your
problem domain, you should just have someone with a graduate
degree in CS theory. Let them work out a solution that's actually
correct and they can explain why, and then the engineers can
implement said algorithm.
FTA - 4 hours ago
Look at programming competitions from an assessment perspective:
what are they measuring and is what's being measured good or bad
for a job?Ability to work under short time constraints (probably
good), to hack out some solution that will work temporarily (good)
but probably not solid (bad), to forego much time to consider the
implications of design and implementation choices (bad), to develop
without communication if solo competition (bad) or without
communication outside of the small group of core developers (bad),
to build a solution without getting feedback and refinements from
stakeholders (bad), and so on.
0x4d464d48 - 4 hours ago
I haven't done a code competition before but my understanding is
that code competitions reward cowboy coding over good engineering
practice, i.e. implementing features while paying no heed to
maintainability.When you have that mind set and start dealing with
a codebase as monstrous as Google's it sounds like a recipe for
some serious technical debt.
[deleted]
WalterBright - 3 minutes ago
All job interviews rely on proxies for whether someone will be good
on the job or not, because the only way to know is to actually hire
them.All proxies are inaccurate.
pklausler - 4 hours ago
As a long-time interviewer, I've learned that a candidate being
good at programming competitions means that they're probably good
at programming competitions.It's a weak signal either way for
success or failure at interviewing and being able to do the job.
Part of the problem, as we've discussed on HN so often recently, is
that a programming interview has to waste time with FizzBuzz-style
questions just to flag the candidates with great resumes,
transcripts, and phone screenings who still actually can't program
a computer to solve even a trivial problem.
kafkaesq - 17 minutes ago
who still actually can't program a computer to solve even a
trivial problem.Or who get nervous, and freeze up.
purpleidea - 3 hours ago
Completely agree with Peter. I'm shit at those competitions (maybe
not the worst ever) but I think I'm pretty good at my job. But I
think there are also some who are good at both.
pitt1980 - 3 hours ago
I'm not able to watch the video at my current computerbut its
actually really typical that when something becomes an advantage
for being selected to a certain poolthe success of the those in the
pool after the selection will negatively correlate with that thing
---------the really obvious example of this is the hockey birthday
thing from Malcolm Gladwell's Outlierspeople with the earlier
birthdays were more likely to make it past each selection stage in
becoming an NHL playerbut those with the later birthdays who were
able to be selected in spite of their later birthdays, were
typically more successful after the selection---------The authors
contend that the strategy might actually work against a team's
success because they found that players born later in the year and
drafted later actually had more productive hockey careers.Deaner
said the study showed that men drafted in the second half of the
year were about twice as likely to have successful careers in the
NHL ??? reaching benchmarks like 400 games played or 200 points
scored ??? than those born earlier in the year."If the team wasn't
making this mistake, they probably would have been more
successful," he said. "The guys born in the first part of the year
are much more likely to be busts."https://www.nhl.com/news/study-
suggests-nhl-has-bias-in-favo...
muh_gradle - 3 hours ago
I am familiar with Outliers. I read the book. But I fail to see
how this is relevant in any way.
scythe - 3 hours ago
It's sort of like Simpson's paradox. It's relevant because the
statistical "paradox" (contradiction of intuition) described in
Outliers is similar to the one in the video.
kjksf - 3 hours ago
He probably meant that people who do programming competitions
are more likely to pass Google interview (they'll be better at
doing algorithmic questions quickly) but are not necessarily
better suited to do the actual job.Just like hockey players
born earlier in the year were more likely to be drafted.
nilkn - 2 hours ago
To expand on this:* Prior to officially selecting candidates
based on performance in problems derived from programming
competitions, candidates who excelled at programming
competitions were likely to do well on the job.* That
correlation was observed on a wide scale by employers, so
many companies -- Google chief among them -- started
incorporating such questions into the official interviews.*
Candidates now observed the change in employer interviewing
methods on a wide scale and adapted their preparation
methods. This fundamentally changed the pool of people good
at programming competition problems in such a way as to
reduce the correlation between the original signal (good at
algorithmic problems) and the goal (good at the job).*
Overall, widespread acknowledgment -- and all consequent
changes in behavior -- of the original correlation between
the signal and the goal significantly reduced the quality of
the correlation.
warkdarrior - 2 hours ago
In other words, what gets measured gets managed.So if you
measure hiring candidates by their performance in
programming competitions, everyone will manage their own
skills towards doing better in competitions.
notduncansmith - 1 hours ago
Yes but not just that. I think the bigger trend is that
training oneself to excel at those types of problems
meant (in addition to other things) one thing about you
back then (that you were really into programming). Now
that same behavior likely means that you want to get a
nice job at one of the big tech companies, as a result of
them publically selecting for that. These are fuzzy
indicators to begin with, but they're definitely
different fuzz.
robrenaud - 2 hours ago
I think it's simpler than this. I don't think there is
much outright gaming of the signal.Programming interviews
and programming competitions are very similar, much more
similar than programming interviews and real world software
engineering. When you are selecting top programming
competition competitors, you are implicitly selecting
people who will absolutely smash your (non-design)
programming interview questions. This has little to do
with their effectiveness as software engineers.
btilly - 1 hours ago
Your interpretation is too weak. It is not just "not
necessarily better", it is the stronger "on average are
probably worse".The reason is that programming competitions
give more of a boost to your odds of being selected than to
how well you'll do on the job. So people who otherwise
wouldn't have gotten in now will, and will not perform as
well as the people that they displaced.Which is what happened
with hockey players. Being born at the right time of year
put you in a bucket with people who were slightly younger
than you. Which improved your performance on the tests, but
didn't matter once you all grew up. So slightly worse people
at the right time of year displaced slightly better at the
wrong time of year, and the average came out that people who
got through and were born in the latter half of the year were
actually better.
register - 2 hours ago
Definetely. I arrived at the final stage in a Google
interview which I failed and I can confirm that out of the 5
interviews three were based on puzzles that I later
discovered are found in books for coding competions. On two I
did a good job to work out a solution myself but I completely
got the third wrong. Who prepares for this kind of
competitions has a huge advantage in these kind of
interviews.
Cofike - 2 hours ago
That's how I felt trying to find a job in the bay area. If
I wanted to compete with the top talent then I needed to
prepare for the interviews and practice those problems.
balls187 - 2 hours ago
Google's data shows that their interview process produces a
low number of false positives, at the risk of producing a
high number of false negatives.That is, despite passing on
otherwise talented people, those people who successfully pass
Google interviews go on to be successful at Google.edit to
add:Couldn't find that original article, this article goes on
to speak about success predictors:
https://www.wired.com/2015/04/hire-like-google/
dchichkov - 1 hours ago
Was it a proper double-blind study substituting interview
result for rnd() to make a hiring decision? And established
correlation a few years down the line?Because what you've
stated: 'data shows' ... "successfully pass go on to be
successful" - sounds like cargo cult science or
pseudoscience to me.
balls187 - 1 hours ago
Google generally isn't known for pseudoscience.
encoderer - 1 hours ago
Even Google has a PR department dumbing things down.
dchichkov - 59 minutes ago
You've stated 'data shows'. So my question was - was it a
proper double blind study?Because it is definitely
possible to do it properly. Substitute the results (or
partial results) of the interview with rnd(), use it for
hiring decision for a subset of candidates. Keep this
information confidential. Establish if parts of your
interview process don't perform better than randomness a
few years down the line.It's possible to do. Only I don't
think this was done. And if it was not done, and the
method was some 'data shows' with hand-waving - it would
be under definition of pseudoscience.
[deleted]
lordnacho - 3 hours ago
> the success of the those in the pool after the selection will
negatively correlate with that thingCould be relevant:
https://en.wikipedia.org/wiki/Berkson%27s_paradox
pitt1980 - 2 hours ago
Yeah, exactly
bit_logic - 3 hours ago
Here's a list of companies that don't do this
https://github.com/poteto/hiring-without-whiteboards I hope it's
the beginning of an industry wide trend away from these types of
interviews.
bluetwo - 3 hours ago
Overfitting?
AndrewStephens - 3 hours ago
I don't find this surprising. I played around on Top Coder enough
to reach the first division but the problems presented were in no
way related to my day to day work. While the people that did very
well (I never got anywhere once I hit the top grade) might have had
an excellent command of very specific data structures and
techniques, non of the code would have been very useful in the real
world.If I was interviewing someone, a career as a competitive
programmer would not be a detriment but it would not count for very
much overall. We are looking for creative thinkers that can work as
a team.
CalChris - 3 hours ago
The market is pretty good at sorting this out and I'm sure
programming competition winners are fairly valued. I've seen more
Berkeley, Stanford, MIT, CMU and IIT degrees rise to the top than
other schools. I haven't seen any top coders. Maybe they're there
and I haven't noticed. But I haven't seen them on resumes that I've
culled through.It's probably something you might list on a first
job resume but further on down the road, you'd just list your work
and your education.
lr4444lr - 2 hours ago
In the chapter of his interview in Coders at Work[0], Norvig found
that the strongest correlate in the interview process with success
at Google was paradoxically to have been given the lowest possible
score by one of the interviewers. He surmised that this was because
in order for such a person to have even gotten hired at the end of
the process, someone trustworthy must have seen so much potential
in the prospective hire that he strongly advocated for the person
to be offered a position which worked in spite of that other low
rating.Kind of ironic for a company whose product values are so
tightly tied to quantitive
data.[0]http://www.apress.com/us/book/9781430219484
andrewla - 1 hours ago
A qualitative assessment of that factor is hard because Google
doesn't hire the people that it doesn't hire.Another assessment
could be that a divergence of opinion among interviewers is
itself a positive sign -- programmers with strong controversial
opinions who are willing to hold to them even in an interview
setting might be better programmers for that.A less sanguine
assessment is that "success at Google" correlates with people who
generate controversy around themselves, simply because that is
something that creates visibility.
aoeusnth1 - 1 hours ago
This isn't surprising - in fact, it would be surprising if it
wasn't the case. This fact says nothing about the ability of
Google's hiring bar to distinguish between good and bad hires,
except that it is not a perfect signal.
[deleted]
sghiassy - 4 hours ago
Question?Does being good at programming competitions make you good
at interviewing for programming jobs?Obviously there's some irony
in that question. But I also think there's some truth to it as
well.
msteffen - 4 hours ago
As a former ICPC world finalist who later worked at Google, I
would say that practicing for programming contests is almost like
cheating as far as Google interviews are concerned. The Google
interview format is almost identical to the ICPC practice
sessions I did in college.I've seen this video before, and IMO
the extreme similarity between programming contest questions and
Google interview questions could explain the negative
correlation. Specifically, borderline engineers with programming
contest experience are more likely to get hired than borderline
engineers without programming contest experience, and therefore
the set of Google engineers with programming contest experience
includes more borderline people than the set of Google engineers
without programming contest experience. Thus the slight negative
correlation.
opportune - 3 hours ago
I expect this to be the true root cause of the negative
correlation as well. Technical questions are good at rooting
out those not good at data structures and algorithms, and good
at promoting those that do. However, you can be excellent at
solving ICPC questions and not know anything about
documentation, object oriented design, OS, databases, anything
web related, etc. It's a very specific skill and practicing
programming contests optimizes for it.
cbhl - 3 hours ago
When I was in high school, being good at programming contests
meant picking up a bunch of habits that you'd never use at work
-- memorizing the same twenty #include lines that include every
conceivable STL data struct you'd need; using single letter
variable names; no comments whatsoever. If you and the next
person come up with the solution at the same time, you might
lose simply because the other person could type faster. You
have to un-learn these habits for industry.Just about anyone
can get exposure to a set of representative coding questions
(see: the USACO training robot, or Cracking the Coding
Interview), but training for these contests means spending XX
hours a year under a time limit trying to write code from
memory (because you don't have time to look things up in the
manual).
xyzzyz - 4 hours ago
In my opinion, based on my and my friends' experience, being good
at programming competitions clearly makes you very good at
interviewing.I don't think that being good at contests makes you
bad at the job, though. Consider the following scenario: assume
that skill of being good at contests is completely orthogonal to
being good at the job (so the real correlation coefficient is 0,
instead of negative, observed by Norvig). Then, since it's easier
to to get hired if you're good at contests, the observed
correlation coefficient inside a single company will be negative,
due to selection bias. Depending on the effect sizes, the
selection bias might even make observed correlation coefficient
negative even if it's positive in reality (which I think it
is).For a more intuitive explanation, consider a used car market
buyer who has preference for black cars (this corresponds to
algorithm-based interview process of companies like Google or
Facebook). Even though the color of the car should have no
correlation with its quality, the black cars owned by that buyer
will more likely be worse quality than non-black ones.
js8 - 3 hours ago
I would tend to agree, and while I was never very good at
programming competitions, when I saw some of the winning code on
Topcoder, I somehow lost interest. If that's the code I would write
if I learned to be better at competing, then I should rather spend
time learning something else.On the other hand, something like
Project Euler or even Advent of Code is very nice, if you do it at
your own pace, or for learning a new language.
jpn - 2 hours ago
This:https://news.ycombinator.com/item?id=13739329
williamsmj - 4 hours ago
The title of the post understates the claim in the link, which is
that, "being good at programming competitions correlates negatively
with being good on the job".
williamsmj - 4 hours ago
I tweeted this link this morning (apparently the first time it's
been posted to Twitter, so perhaps how it ended up here on HN).I
did so in response to the CTO of Kaggle tweeting "Super confused
why we still use resumes. Get 100x the signal from domain
profiles (GitHub, StackOverflow, Kaggle, etc.) & real work
samples", which ... where to start
[https://twitter.com/benhamner/status/883137638084956160].
rquant - 4 hours ago
no the statement is ""being good at programming competitions
correlates negatively with being good on the job, conditional on
passing Google interviews" which is completely different
statement."being good at programming competitions" hugely
positively correlated with "begin good on the programming job"
unconditionally
sbierwagen - 3 hours ago
Do you have... any kind of evidence to support that claim?
Klockan - 3 hours ago
The overwhelming majority of people who can't code would fail
horribly both at jobs and competitions, this effect will
drown out everything else no matter how the distribution
looks for the few percent who can code.
bjacokes - 2 hours ago
Here are some of the positive qualities I would expect from a
competitive programmer compared to the general CS population early
in their career:- knowledgeable at algorithms and data structures-
good at analyzing correctness and edge cases, even on simple non-
algorithmic problems (e.g. FizzBuzz)- accustomed to working hard
and learning new concepts. this attribute is not specific to
competitive programmer ? for example, I'd expect similarly from an
open source contributor ? but it's higher than for a typical
college student.Some negatives I'd expect, which are fixable over
the course of the course of their career:- over-confidence in code,
under-testing- less skilled in OOP, coding style, version control
systems, as well as web development or systems code (unless they
have specific previous work in these areas)- sometimes looking down
on gruntwork/rote as beneath them (like the view that pure
mathematicians have towards applied math or statistics)I think that
list of positive attributes often outweighs the potential
negatives, especially during an internship or in the first year or
two of someone's career. After that, I would expect many non-
competitive programmers to have picked up on some of those
advantages (code correctness, learning new concepts).I've tried to
steer my own interviews away from algorithms (especially DP) and
focusing more on giving problems that are relatively
straightforward, while still being complicated enough that someone
has to write precise code and identify/fix a few edge cases.
kinkrtyavimoodh - 3 hours ago
Adding to the good points here, I think coding competitions
superstars are also potentially more likely to have a diva-complex,
while others are more likely to have an impostor complex when they
get into a company like Google. The right amount of impostor
complex (where it enables you but does not cripple you) is actually
very helpful as it helps you learn and better yourself.
kazinator - 3 hours ago
People who enter programming competitions are looking for some sort
of glory: to be stars. When they don't get it from the job, they
get bored.I suspect that the people good at programming
competitions could easily perform well on the job, if the
motivation were there. I don't think it necessarily has anything to
do with short-term versus long-term problem-solving focus.There are
plenty of short-term problems that you have to solve on the job to
be effective. You're not doing them in competition against anyone
such that if you procrastinate, you will lose, so the motivation
vaporizes.Also, since job interviews are like programming
competitions, people who are good at programming competitions
figure they can easily get a job anywhere, and to do that
repeatedly. They are not motivated into working hard by job
security concerns.
[deleted]
mcv - 3 hours ago
I once had an online discussion about the difference between
competitive programmers and professional programmers, and which
were better. Someone argued that competitive programmers were
better, because they had to perform in more extreme circumstances.
As he put it: they were sent into the forest with a knife to kill a
lion.Everybody else ran with that metaphor. Someone asked what a
lion was doing in a forest; don't they live on the savannah? I
asked whether he was sure there was a lion; plenty of times I've
been sent to kill a lion and ended up having to kill a goat or an
elephant instead. Are we even sure it needs killing?I think that's
the difference between a competitive programmer and a professional
programmer. The competitive programmer will be much faster with a
solution to the given problem, but the professional programmer will
solve a better problem.
wlesieutre - 2 hours ago
Mountain lions like forests just fine! Unfortunately for them, we
chopped most of the forests down.Assuming you're in the US and
somebody tells you there's a lion around, pretty good odds it's
not one of the big orange ones.EDIT: Bay area published yesterday
http://www.nbcbayarea.com/news/local/Several-Mountain-Lion-S...
mcv - 45 minutes ago
A mountain lion is a very different animal from a lion.
Different subfamily and genus, even. I don't live in the US,
but when someone tells me "lion", I tend to assume they mean a
lion, and not a puma. Though if there's any doubt, that's
absolutely something to ask questions about.