Example of Learning

In this video I'm going to showyou an example of machine learning it's a very simple kind of neural net andit's going to be learning to recognize digits and you're going to be able to see how the weights evolve as we run avery simple learning algorithm so we're going to look at a very simple learning algorithm for

A very simple way to recognize handwritten shapes

training a very simple Network to recognize handwritten shapes the network only has two layersof neurons it's got input neurons whose activities represent the intensities ofpixels and output neurons whose activities represent the classclasses what we'd like is that when we show it a particular shape the output neuron for that shape getsactive if a pixel is active what it does is itvotes for particular shapes namely the shapes that contain that pixel each ink pixel can vote forseveral shapes and the votes can have different intensities the shape that gets the mostvote wins so we're assuming there's competition between the output units um that's something I haven't explained yetand we'll explain in a later lecture so first we need to decide how

How to display the weights

to display the weights and it seems natural to writethe weights on the connection between an input unit and an output unit but we'dnever be able to see what was going on if we did that we need a display in which we can see the values of thousandsof Weights so the idea is for each output unit we make a little map and inthat map we show the strength of the connection coming from each input pixelin the location of that input pixel and we show the strength of the connections by using black and whiteblobs whose area represents the magnitude and whose sign represent andwhose color represents the sign so the initial weights that you see there are just small randomweights now what we're going to do is show that Network some data and get itto learn weights that are better than the random weights the way we're going to learn iswhen we show it an image we're going to increment the weights from the activepixels in the image to the correct class if we just did that the weightscould only get bigger and eventually every class will get huge input whenever we showed it an image so we need someway of keeping the weights under control and what we're going to do is we're also going to decrement the weights from theactive pixels to whatever class the network guesses so we're really training it todo the right thing rather than the thing it currently has a tendency to do if of course it does the right thingthen the increments we make in the first step of the learning rule will exactly cancel the decrements and so nothingwill change which is what we want so those are the initial weightsnow we're going to show it a few hundred training examples and then look at the weights againand so now the weights have changed they've become to they've started to form regularpatterns and we show it a few more hundred examples and the weights have changed some more and a few more hundredexamples and a few more 100 examples and a few more hundred and now the weights are pretty

The learned weights

much at their final values um I'll talk more in future lectures about the precise details ofthe learning algorithm but what you can see is the weights now look like littletemplates for the shapes if you look at the weights going into the one unit for example they look like a little templatefor identifying ones they're not quite templates if you look at the weights going into the nine unit they don't haveany positive weights below the halfway line um that's because for telling thedifference between nines and sevens um the weights below the halfway line aren't much use you have to tellthe difference by deciding whether there's a loop at the top top or a horizontal bar at the top and so thoseoutput units are focused on thatdiscrimination one thing about this learning algorithm is because the Network's so simple it's unable to learna very good way of discriminating shapes the what it learns is equivalentto having a little template for each shape and then deciding the winner based on which shape has the template thatoverlaps most with the ink the problem is that the ways in which handwritten digits vary are muchtoo complicated to be captured by simple template matches of whole shapes you have to um model theallowable variations of a digit by first extracting features and then looking atthe arrangements of those features so here's examples we've seenalready and if you look at the those twos in the green box you can seethere's no template that will fit all those well and will fail to fit that three in the red box well so the tasksimply can't be solved by a simple Network like that the network did thebest it could um but it can't solve this problem