HN Gopher Feed (2017-11-04) - page 1 of 10 ___________________________________________________________________
What is backpropagation and what is it actually doing? [video]
334 points by adamnemecek
https://www.youtube.com/watch?v=Ilg3gGewQ5U___________________________________________________________________
[deleted]
cmatt01 - 3 hours ago
There needs to be some sort of intense advocation for visualization
tools.
oliv__ - 3 hours ago
I dove into this not knowing anything about neural networks, but
the feeling I came out of it with was incredible: I love it when
something blurry and obscure slowly morphs into a sharper picture
in your mind, it's so empowering.
quotemstr - 5 hours ago
The entire YouTube channel is fantastic. 3Blue1Brown's series on
linear algebra is the best I've seen anywhere.
wybiral - 4 hours ago
Agreed. Pretty much every video on that channel is just as good
as this one.
mlamat - 5 hours ago
Thank you very much. I must code a neural network with
backpropagation for my AI class. Can anyone recommend a book?
fnbr - 4 hours ago
If you're looking to understand the underlying theory behind deep
learning, the Deep Learning book by Goodfellow et al. is
awesome.If you're interested in general machine learning, the
Elements of Statistical Learning, by Tibsihirani et. al is great;
a more applied book is Applied Statistical Learning by the same
author. For a more applied view, I'd check out Tensorflow or
PyTorch tutorials; there's no good book, as far as I'm aware,
because the tech changes so quickly.I've done a series of videos
on how to do deep learning that might be useful; if you're
interested, there's a link in my profile.
Yreval - 3 hours ago
This book is a good practical introduction that walks you through
the basic ideas as you develop some basic functionality.
http://neuralnetworksanddeeplearning.com/I'm often pretty
skeptical of e-books and self publications, but the above link is
pretty good (and the video series linked here references it as
well.) The Goodfellow book that another commenter mentioned is a
high-quality survey of the field and a nice, high-level overview
of different research directions in deep learning, but isn't as
pragmatic as an introduction.
afarrell - 59 minutes ago
A bit of a side-note, but I think it is an interesting piece of
marketing that Amplify Partners decided to sponsor[1] the previous
video in this series. I wonder (and hope) we'll see more VCs
sponsoring open educational content relevant to their focus.[1]
https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1205
nouveaux - 6 hours ago
This was very timely for me and for anyone else learning, here are
the first few videos of the series:https://www.youtube.com/watch?v=
aircAruvnKkhttps://www.youtube.com/watch?v=IHZwWFHWa-
whttps://www.youtube.com/watch?v=Ilg3gGewQ5U (Original
video)https://www.youtube.com/watch?v=tIeHLnjs5U8
alexasmyths - 4 hours ago
This video series is amazing and I wish it existed long ago.
samueloph - 5 hours ago
oh my god, as soon as i saw this video was from 3Blue1Brown i
immediately thought "this gonna be good!". I didn't realize he was
posting a Deep Learning series.
perfmode - 5 hours ago
Each one of these videos consists of thousands of lines of code.
The attention to detail is impressive.
kharms - 5 hours ago
In my opinion this author produces the best math videos on
youtube.If you can afford it and enjoyed this video, consider
supporting him on Patreon. https://www.patreon.com/3blue1brown
smortaz - 4 hours ago
If you enjoy his videos (and other creators'), please consider
signing up on Patreon and supporting them.
aidos - 35 minutes ago
Someone on here (I think) recommended his videos on linear
algebra a while back and I've since watched them all, several
times.A couple of hours of watching time built an intuition and
understanding of linear algebra and the broader maths around it
that 4 years of university training didn't give me. That's a
little unfair, because I obviously learnt a lot on the courses
that make these videos easier to understand, but man, they're so
well done.
SonOfLilit - 5 hours ago
If you liked him you will love acko.net. Try
https://acko.net/blog/how-to-fold-a-julia-fractal/
adamnemecek - 5 hours ago
There needs to be some sort of organized push for visualization
tools. I know, I might be bringing the proverbial owls to the
proverbial Athens with saying that here, but I really do feel that
if done right this could impact the course of the world like
nothing else. This could be as important as idk, invention of book
press or smth. Make computer "the visualization machine".I think
that one of the fundamental problems is that to be a visualization
machine, you need to have easy access of the GPU and OpenGL is
provides anything but. I think that shadertoy (shadertoy.com) is
the thing that comes the closest but the learning curve is kinda
steep.I know that people like Alan Kay, Bret Victor or Michael
Nielsen (his post was on the fp the other day
https://news.ycombinator.com/item?id=15616637) share these
sentiments but this is a task bigger than a single people.Idk what
I really mean by "organized push". I'm not sure if the problem is
well defined too
minimaxir - 5 hours ago
In deep learning, TensorBoard
(https://www.tensorflow.org/get_started/summaries_and_tensorb...)
works with TensorFlow and Keras to show what the model is doing.
However, it ends up being more complicated/unintuitive than a
YouTube video, so it's not as useful.
adamnemecek - 4 hours ago
The problem is that this is an ad hoc solution. What I'm
talking about would be some formalization of visualization (I
guess kinda like grammar of graphics without the statistical
aspect) so you can visualize just about anything.
minimaxir - 4 hours ago
?Visualizing just about anything? isn?t helpful if you want
to learn from the visualization, though. (c.f the
/r/dataisbeautiful subreddit nowadays: https://www.reddit.com
/r/dataisbeautiful/top/?sort=top&t=mon...)That?s not to say
that a purely artistic data visualization has no value, but
it?s not academic. (I admit I am guilty of that at times)
adamnemecek - 3 hours ago
Data visualization is only a part of it. I'm talking about
visualizing concepts.
seanmcdirmid - 4 hours ago
There was a big organized push for visualization and more
precisely augmented visualized thinking at HARC. It?s really too
bad HARC didn?t work out, but many of us are still very
interested in this problem.
pas - 54 minutes ago
Uh, details/links please on how/why HARC failed.
alfla - 5 hours ago
I agree. Visualization is often key to understanding and
identifying non-trivial issues.Here's a tool a colleague of mine
made for inline "visual debugging" for e.g. computer vision,
written in c++: https://github.com/lightbits/vdb. I haven't used
it myself, but when he presented it I think it made a lot of
sense to have these sorts of tools for processing data in real
time.
SonOfLilit - 5 hours ago
We need more people teaching math through visual intuition. Life a
friend of mine said, "if you want to do computation fast, phrase it
as a problem for your GPU, er, visual cortex".Here is a tool you
can play with to visualize this: http://playground.tensorflow.orgIf
you liked this video, try this different visual intuition of what a
neural network does that I find even
better:http://colah.github.io/posts/2014-03-NN-Manifolds-
Topology/Also, remember that back propagation is a very general
algorithm, it works not only on linear transformation weights but
on any direct acyclic computation graph that is differentiable in
its weights.
phreeza - 5 hours ago
This can also go wrong, for example visualising probability
distributions in low dimensions leads to very wrong intuitions
about the behavior of high-dimensional dimensional distributions.
vbuwivbiu - 5 hours ago
please elaborate - are you thinking of the curse of
dimensionality ?
phreeza - 4 hours ago
There are many examples, one I came across recently is that
the large majority of the probability mass of a high-
dimensional gaussian distribution is in a shell at a distance
from the mean, the mass at the center is actually quite
low.Also anything related to topology, which is important
when you are looking at decision boundaries, becomes
counterintuitive in high dimensions, because so many things
can be adjacent at the same time.
jstanley - 4 hours ago
Can you please try and explain why that is?If true, you're
very correct that lower-dimensional intuition does not
transfer into higher-dimensional spaces: my intuition tells
me that a Gaussian distribution drops off as you fall away
from the mean, and it's quite easy for me to imagine that
in 2 dimensions, 3 dimensions (e.g. by imagining a mound on
a plane) and 4 dimensions (e.g. a cloud in 3-space with
increased density around the mean).Is my intuition wrong in
any of those cases? If so, why? If not, how many dimensions
do we need before it becomes wrong?
nabla9 - 41 minutes ago
> it's quite easy for me to imagine that in 2
dimensionsIt starts to fail really badly when dimension
grows.Two simple examples:1) Consider 3 dimensional unit
sphere centered at origin and unit cube centered at
origin. Cube is clearly completely inside the sphere.
Now generalize to n-dimensions. Hyperdimensional volume
of hypercube with side length 1 moves almost completely
outside the n-sphere with radius 1 when n-grows.2)
Alternatively almost all volume of n-sphere is close to
the surface.These are all very counterintuitive, yet
simple to check toy examples. When you start to integrate
over more complex multidimensional function, things get
weird really fast and intuition fails.
smallnamespace - 3 hours ago
The center of the distribution always has the highest
density, but the ratio of 'probability mass close to
centroid' / 'total probability mass' drops off as number
of dimensions grows.This is somewhat related to another
'curse of dimensionality' observation, which is that the
volume of a hyperball / volume of hyperspace tends
towards zero as dimensions grow -- there's just a lot
more volume that's in some sense 'far' from the center.
orangecat - 3 hours ago
Because an outlier in any single dimension will put the
point outside the "center" of the distribution, and as
the number of dimensions increases there's more of a
chance of that happening.Say you have an N-dimensional
gaussian where each dimension has mean 0 and standard
deviation 1. Define the center as the N-dimensional cube
whose edges go from -3 to +3 in each dimension. A
normally distributed value is within 3 standard
deviations of the mean with probability 0.9973, so the
probability that an N-dimensional point being in the
center is 0.9973^N. With N=4 that's 0.989 which matches
your intuition, but at N=1000 it's 0.067 and at N=10000
it's 1.81e-12.
shas3 - 3 hours ago
Theoretically, yes. Can you give a more concrete example?
Many hard high dimensional and general topology problems
can be visualized through their 2D special cases.
SonOfLilit - 1 hours ago
Related: Hamming's "The Art of Doing Science and Engineering"
chapter 9, N-Dimensional Spacehttp://worrydream.com/refs
/Hamming-TheArtOfDoingScienceAndEn...(I assume Bret Victor
has permission to host the PDF on his website, he is far from
an anonymous pirate)
adamnemecek - 5 hours ago
I'm not sure I know what you are talking about but let's not
throw away the baby with the bath water.
decisiveness - 4 hours ago
I don't think knowing exactly what parent comment is talking
about is required to see that they weren't suggesting we
should do away with all visualizations just because there are
some cases where they might not be the best tool for
teaching.
kobeya - 4 hours ago
What he means is that we only really have intuition for 1, 2,
and 2.5D visuals, but many areas of mathematics don?t map
into low dimensions very well, or do but lose essential
properties in the process. Building a low dimensional
projection of he problem might prime intuition, but it will
also introduce fundamental biases as well.For example,
learning geography by flat map projections only. No matter
what projection you use there is a trade off, and you end up
instilling both the pro and the con of that trade off as
intuition.
Retric - 1 hours ago
Flat map projections work fine if you provide enough of
them.A video from LEO or a rotating map projection provides
very different intuition than a single static map.
https://www.youtube.com/watch?v=EPyl1LgNtoQ
phreeza - 4 hours ago
Yes, exactly, thanks.
adamnemecek - 3 hours ago
I didn't understand the distributions part.
chestervonwinch - 1 hours ago
It's related to the geometric problems the parent
described because probability distributions roughly
describe geometric regions (of high probability density)
where observations are likely.
nitrogen - 49 minutes ago
Part of it may be that in higher dimensions the bulk of a
volume is concentrated near the surface.I found https://b
logs.msdn.microsoft.com/ericlippert/2005/05/13/high...
with a quick search.
nkurz - 3 minutes ago
The article here (and the links in the comments) might
clarify the connection to machine
learning:http://www.penzba.co.uk/cgi-bin/PvsNP.py?SpikeyS
phereshttps://news.ycombinator.com/item?id=3995615
zardo - 5 hours ago
That means you need to keep track of what properties actually
hold under projection to lower dimensions.
smallnamespace - 3 hours ago
Keeping track of properties is not really something
'visualization' necessarily helps with though, more symbolic
reasoning through proofs.
zardo - 1 hours ago
I don't mean that spatial reasoning helps with that, I mean
that if you do it, you can still apply your spatial
reasoning where it's appropriate.
surrey-fringe - 2 hours ago
What percentage of teachers use visual intuition, and what
percentage should?
jfaucett - 2 hours ago
> We need more people teaching math through visual intuition.I
would modify that slightly and say rather just through
"intuition". Visualization helps a lot, but you can also have
great intuition from situations, stories, feelings, etc (anything
that hits the non-reasoning part of the brain i.e. your "gut
feeling"). IMHO one of the biggest problems in mathematics and
science education is that we spend too much time working on
things which humans are bad at (precise calculations) and far too
little doing the 'rough estimation' and 'intuition' work which we
have been evolutionarily optimized for and which is essential to
us for actually remembering and understanding how things work.
phkahler - 5 hours ago
What tools does a person use to make a video like this? I've been
wanting to do the same on my topic of expertise for a while now.
edanm - 4 hours ago
He uses custom tools.However, he actually recommends against
using his tools. He suggests a better option is to use
traditional animation tools.I'm actually not sure what one would
use for more traditional animations of his style though. I mean,
theoretically you can use blender/etc for most 3d things, but how
easy would it be to make something math-based there? Hopefully
someone with some real animation experience can chime in.
mcintyre1994 - 3 hours ago
On the Manim Github he has some suggestions: "For 9/10 math
animation needs, you'd probably be better off using a more
well-maintained tool, like matplotlib, mathematica or even
going a non-programatic route with something like After
Effects. I also happen to think the program "Grapher" built
into osx is really great, and surprisingly versatile for many
needs."
raverbashing - 1 hours ago
> I also happen to think the program "Grapher" built into osx
is really greatI didn't know about this, and it's a nice find
henrikeh - 5 hours ago
He uses custom, self-developed toolshttp://www.3blue1brown.com/ab
out/https://github.com/3b1b/manim
lelandbatey - 5 hours ago
He creates each animation using a set of Python tools and
libraries he wrote. You can find them published here:
https://github.com/3b1b/manim