Neural Networks

Reasons to study neural computation

can compute things the first is to understand how the brain actuallyworks you might think we could do that just by experiments on the brain but it's very big and complicated and itdies when you poke it around and so we need to use computer simulations to help us understand what we're discovering inempirical studies the second is to understand a style of parallel computation that'sinspired by the fact that the brain can compute with a big parallel network of relatively slow neurons if we canunderstand that style of parallel computation we might be able to make better parallel computers it's verydifferent from the way com computation is done on a conventional serial processor it should be very good forthings that brains are good at like vision and it should also be bad for things that brains are bad at likemultiplying two numbers together a third reason which is the relevant one for this course is to solvepractical Problems by using novel learning algorithms that were inspired by the brain these algorithms can bevery useful even if they're not actually how the brain works so in most of this course we won'ttalk much about how the brain actually works it's just used as a source of inspiration to tell us that big parallelnetor of neurons can compute very complicated things I'm going to talk more in this

A typical cortical neuron

video though about how the brain actually works a typical cortical neuronhas a gross physical structure that consists of a cell body and an axonwhere it sends messages to other neurons and a dendritic tree where it receivesmessages from other neurons where an axon from one neuroncontacts a dendric tree of another neuron there's a structure called a synapse and a spike of activitytraveling along the axon causes charge to be injected into the post synapticneuron at a synapse a neuron generates spikes whenit's received enough charge in its dendritic tree to depolarize a part of the cell bodycalled the axon hilic and when that gets depolarized the neuron sends a spike outalong its axon and the Spike's just a wave of depolarization that travels along theaxon synapses themselves um have interesting structure

Synapses

they contain little vesicles of transmitter chemical and when a spike arrives in the axon it causes thesevesicles to migrate to the surface and be released into the synaptic Cliftthere's several different kinds of transmitter chemical there's ones that Implement positive weights and ones thatImplement negative weights the transmitter molecules diffuse across the synaptic cleff and bind to receptormolecules in the membrane of the postoptic neuron and by binding to these big molecules in the membrane theychange their shape and that creates holes in the membrane these holes allowspecific ions to flow in or out of the postoptic neuron and that changes theirstate of depolarization synapses adapt and that'swhat most of learning is changing the effectiveness of a synapse they can adapt by varying the number of vesiclesthat get released when a spike arrives or by varying the number of receptormolecules that are sensitive to the released transmitter molecules synapses are very slowcompared with computer memory but they have a lot of advantages over the random access memory on a computer they're verysmall and very low power and they can adapt that's the mostimportant property they use locally available signals to change their strengths and that's how we learn toperform complicated computations the issue of course is how do they decide how to change theirstrength what is the what are the rules for how they should adapt so all on one slide this is how

How the brain works on one slide!

the brain works each neuron receives inputs from other neurons a few of theneurons receive inputs from The receptors it's a large number of neurons but only a small fraction of them andthe neurons communicate with each other in the cortex by sending these spikes ofactivity the effect of an input line on a urine is controlled by synaptic weightwhich can be positive or negative and these synaptic weights adapt and byadapting these weights the whole network learns to perform different kinds of computation for example recognizingobjects understanding language making plans controlling the movements of your body um you have about 10 to the 11neurons Each of which has about 10 to the four weights so you probably have 10 to the15 or maybe only 10 to the 14 synaptic weights and a huge number of theseweights quite a large fraction of them can affect the ongoing computation in avery small fraction of a second in a few milliseconds that's much better bandwidth to stored knowledge than evena modern workstation has one final point about the brain is

Modularity and the brain

that the cortex is modular or at least it learns to be modular different bits of the cortex end up doing differentthings genetically the inputs from the senses go to different bits of the cortex and that determines a lot aboutwhat they end up doing if you damage the brain of anadult local d damage to the brain causes specific effects damage to one place might cause you to lose your ability tounderstand language damage to another place might cause you to lose your ability to recognizeobjects we know a lot about how functions are located in the brain because when you use a part of the brainfor doing something it requires energy and so it demands more blood flow andyou can see the blood flow in a brain scanner so that allows you to see which bits of the brain you're using forparticular tasks but the remarkable thing about cortex is it looks pretty much the sameall over and that strongly suggests that it's got a fairly flexibleUniversal learning algorithm in it that's also suggested by the fact that if you damage the brain early onfunctions will relocate to other parts of the brain so it's not genetically predetermined at least not directlywhich part of the brain will perform which function there's convincing experiments on baby ferrets that showthat if you cut off the input to the auditory cortex that comes from the earsand instead rot the visual input to auditory cortex then the auditory cortexthat was destined to deal with sounds will actually learn to deal with visualinput and create neurons that look very like the neurons um in the visualsystem this suggests the cortex is made of general purpose stuff that has the ability to turn into special purposehardware for particular tasks in response to experience and that gives you a nice com combination of Rapidparallel computation once you've learned plus flexibility so you can put you canlearn new functions so you're learning to do the parallel computation it's quite like anfpga where you build some standard parallel hardware and then after it's built you um put in information thattells it what particular parallel computation to do conventional computers get theirflexibility by having a stored sequential program but this requires very fast Central processes to accessthe lines in the sequential program and perform long sequential computations