dl

LECTURE 1:

Like, uh, you know, is being figured out in the background, then you will not notice, or maybe it will again happen, or reach a peak, you know, and not be able to deliver, people who lose interest and then will have to go again through. This I called the I Winter, so, you know, go to a Jennifer, and I Winter, and then to kind of recover from that. Yeah. in 2006? Yeah, one is the landmark Lat seems so because is that. was this people in app So there was this schemes of training V architecture layer-wise, using either IBMs or the encoders. So Jeff likes sort of the idea formulation, you’re sure, like, you’re going for the termination, but, um, There’s this idea that now you could train a deeper architecture by layer-wise pre-training, each layer, and then stacking them together and then learning it by style. Nope. If you talk to, or, you know, you guys end up playing that, you actually figured this out 10 years before and had a paper. But I think NTL sort of the sort of deep belief, there’s why it sort of stage, this wasn’t reproduced on the same scale. Like, you believe, so you have people in that respect of that motors are sort of another thing. I think Yalik Kun was doing something with I was parsity, part for that. Anyway, but like you started getting more and more groups that were able to come up with different recipes that would do sort of this replaying layer-wise new training, then were successful. So it felt like, okay, something was figured out because before maybe there was one group that I knew that I could do it, and not a lot of people introducing it now, like every group came up with their own recipe and all of them work. So it felt like something was happening. Usually when multiple people independently you guys are, you know, managed to make the same thing work, maybe you decide validations. So it’s usually a side that something has happened and now everyone can do it. Like if we get just one group that is able to do something and no one else is able to do the same, there’s usually something weird. But like when it kind of scales up and everyone is able to come up with their own recipe, it means that something has changed in sort of how we approach this problem and, um, where sort of at the point where we’re starting to solve the problem. why did you even need 6 years for Alex? Like we only know that getting like sk. So, um, I mean, okay, so, so, the whole understand is kind of in music. So, I mean, obviously no one knew exactly what it was called this, the community before, like, you know, let’s put it this way. It’s not like in 2006 everyone knew that, oh, if we do I listen that, you know, everyone will buy it and it be fine. If everyone, if they knew that, they would put their forces together and just do it. Um, the way Alex met happened is, um, Unless, uh, Twitchis, if I’m confident, correct, it was the 1st of all, the paper. So he was… Hop anonymizing things up. He was a PG student in the English group. I don’t know think it was a master. It was a P think uh student, Um, his background wasn’t necessarily machine learning. So if he was basically, um, or it doesn’t matter what his background, but sort of the character, he was more of a hacker kind of like, like to write, uh, hair and all, uh, you know, uh, I mean, in the background, okay, so there’s multiple things that are happening in the background. The other thing that happening in the background these sort of GPUs are becoming popular because of games and whatnot. And, um, I think what happened is just took this path and that alleys to the student that told him that I, look, if you manage to improve the numbers or image that by a few percentage, you’re gonna get their PDPs. Like you don’t need to worry about sort of standard research, just get those numbers up. Um, what through Alex Net will happen is really sort of, uh, um, like there’s no novelty in the paper itself. Like, all of the methods that were kind of, you know, a net were kind of around in previous papers. What really took that thing to work was some specializedct kels that were able to use multiple of the same type of training on models. And that was something that was not necessarily easy to do. And if I remember correctly, but that kind of thing where Alex was interacting with engineers and NVDA, because, um, like the, the, the, the, the libraries that the media had weren’t transformation learning, there weren’t that documented, that they weren’t able to do the things he wanted to do. So you got to that kind of low level where, you know, you was talking to this guy, they’re like, I need this pretty team and whatnot. But that’s kind of how Alex that happened. And it actually worked much better than everyone expected. So what Jeff was hoping for is the annual airport is to be on par with previous matters. What the outcome was that it was way better. I forgot the difference, but it was like a significant jump in performance. To the point that, you know, specifically years after Alex met, a lot of computer vision groups tried to get to that kind of performance with traditional methods that they couldn’t. And that’s sort of when they realised that I grew, we’re not, we’re never gonna, we like sift and all of these other tricks that we were doing, we’re not gonna be able to get there. So they had all of this I anyway, so that’s why sort of the change kind of happened, right? So you thought Janice had paper that had a big number, but there was like a bunch of groups trying to beat that number. They couldn’t. they were very far away. Then, um, There was Andrews, these are a man who is like a very big name in computer teaching community that everyone tries to. So his group switched to neural networks and they started working and, you know, like at that point, you know, like it’s one thing to have a really good result coming from, like an outsider, that maybe you don’t try it, like they’re not part of the computer region, they don’t mess with in popular region, but then when they’re like deep names in the field, they start using these techniques and they’re starting getting the same kind of results, then everyone is like, okay, this is the thing we should do with this. you know, so, so I think that’s how it kind of happened. But yeah, it was, I guess sort of, Yeah, I guess it was a mix of Jeff Eaton being eccentric and doing these kind of things and getting out as a student, others being kind of the right person at the right time. So he’s kind of left with the other far as I know. Like, after Alex Knight, he’s not, because he was never into this stuff. Like, he did it because, well, he wanted it, especially. I mean. Yeah, when he’ so I mean I kind the paper really has to stay by another folks around were sort of more like Mus audience but yeah, that’s that’s kind of the the name. I also don’t know what internet came about, but I don’t think in 2016 network there. But I don’t know. Oh, yeah, yeah. Where those music that right? So where goes where you went to consider? Oh, okay, so this came about March later. Okay, so, okay, technically the 1st kind of geometric, the learning thing, is 2009 or earlier, by some Italian guys. There’s paper was largely ignored and no one paid attention. I think it’s like 2016 when the field actually took on. So we had there were like a bunch of papers that the rap competition in our networks. there was uh interaction networks that we did. There was, um, sort of a pizza, very paper called graphical networks, I think, or something like that. But that’s sort of what it kind of took, of. Um, I think people are sort of just a question about the right tiving. So the 2009 paper and it came up, out, it wasn’t the right timing because, well, the learning was still not established yet. So, and and this was uh, so, so dramatically learning, are these sort of the originals, or Gracnio Net, or kind of structure? It’s kind of like a weird mixture of neural networks. So something that really doesn’t look like new e because you have these gr factors so you have all of these spent non-parametric structure of a bit. So, it was weird for folks that were not doing neural networks, but weird for folks that are doing neural networks, because it does mix involved, and, you know, no one both languages. The timing wasn’t right and because what happened in after hugeet, a lot of the f in appealed went on a lot more like uh typical topics, like computer vision and language. Um, and like 2016, 2017, and dramatically learning support. Um, it was more about people wrenching off into other topics. where graph structures were sort of the predinant form of the data. So I think before that, there were just too many problems to be solved in compared reason and language for, and the community wasn’t big enough, which is for all of these domains. So, you know, that’s kind of how our fund. Yeah, from real time, you said, cookies or noodles, and yeah, uh, Uber. Yeah then do you say that 2 survived been give me the like explorgianona oil and Yeah, definitely. I mean, as a community, we tend to focus on whatever is high for the moment and now most of the research is around neural networks. There are certain aspects, there are some of the, right, in the refirmation of, of, um, learning, uh, self. um Okay, so let me just give you an example, but now we’re going into, like, this is all sort of panothetical now. I don’t necessarily have any kind of comfort with this. But, uh, one, you know, we talked about some generalisation hours and distribution. So we talked about this idea of learning the actual task, or, um, this has been, I mean, I might adapt to my mind about it much later. Um, so there is a subfield in domestic developing, but this type is there without branch out where it’s about, um, algorithmic imitation. So the idea is, I have an algorithm that I know, like soty, I know how to write it out, and we understand it. and I have an email effort. And what I want to do is, I want to give some example to that new network, for you to learn how to solve. Uh, but, like, I wanted in such a way that I can actually read out from, you know, like, the goal is to make sure that the other actually learns the source, right? So a point where I can identify, in the neural network, the computation that he does, Um, and I can, you know, guarantee that what he’s doing is some form of certain, right? Um So so this is, you know, this is the space of our in alignments and I going to take decision and so forth. And – and I think the – you know, the – the high level outcome is that – and then there’s papers about the result, is if you can’t do that. Like, unless you cover, you give all possible examples, like, you’ll never, even though in your network has the capacity to represent the working algorithm, it will never be started by learning, using very young child. Um, and that’s Like, one way people would say this is because the, yeah, architecture itself is not sufficiently aligned to the algorithm. Uh, whatever that means, and then, you know, this is a UV sort of handwriting. But there’s like some aspects of it that I do actually personally think are important. for example, an algorithm usually has part access to the data, like you know, as parts operations. Well, neural networks are very dense. Like in the neural network. So – so in an alori that we have variables and you have sparse access to the variable that we you are variable the whole time. When you’re on network, the equivalent of the variable is, like, even say, of the neural network, that is, but you can actually, you know, network. But you, in a, in a neural network, every time you do a stack of computation, you use all of your hidden space, like all your hidden space changing, right? There is no type of spice access where you’re changing your heater activation locally in one region. It’s always everything changes locally. While algorithms are very low from it, how they usually. So this is sort of brought up as one sort of difficult research. Uh, the other thing is, algorithms often want to increase structures. Neural network everything has to be engineers. And some of these things are sort very gang texture. Like maybe the global versus local kind of saving, you can work around it. But everything being continues, it’s sort of a very kind of like, you know, it’s very difficult to the architecture. Like we, there is obviously there is people working on these these variables and neural networks and so forth. But, you know, I can tell you, it doesn’t really work or it works as long as you have, it needs to be variables and and then you very well. Uh, so, so this would be a fundamental limitation, that would mean at some point in the future we might need to change something drastically about your own networks, in order to be able to have the speed structures or in order to be able to have sort of like local updates or like local interaction, which is like, um, And there is a question, yeah, so 1st of all, we’re not exploring that. Because, well, it’s a little bit off topic and people, you know, like this works. So there is this integation of wow, maybe we just need to sc things up more and work. you don’t have to be in the dispute factor like it’s sort of all on your eyes. like you know how to do the feed factor. It doesn’t mean that there’s no solution that everything you continous. We just need to search very private. So you know, exactly what you’re saying. Like people are ignoring these things because trying to do speed structures and kind of going into the non-parametric world, and people believe non-parametric don’t work. So, you know, I mean, there are some folks who are interested in creating sort of a mixture between your Netflix and unparametric, you know, Russian processes or other kind of things. So there are some folks interested in that, but there’s not many statement. People usually don’t like that. you know, they’re like, okay, you know, letters. So itreases respiration the way you said. Um, and we might just sort of just take, um, us skating for a point where we really can’t do certain things, and then go into an NI winter, for the community to be able to say, like, you know what, like, let’s be focus, and let’s, let’s not, not, not everything has to be confused, not everything has to be easier on that, but. Um, but I think it’s a very, like, um, natural sort of way to feel this enrage, right? Like, it all comes even down to the publication process. If you and then sort of a new process in the publication, right? The you are done by PHD students, who are just sort of now learning about all the vision involving stuff that’s excited about, like, you know, it’s like when you’re going outside, you give it for generic viewers, when you have a paper that goes outside of their comfort zone and started doing some weird stuff, they will have a tendency to read that, because they’re easy. They – they will have a hard time to judge it, but, first of all, they don’t have the context of where it is sort of compared ideas coming from, and what is time to do. And so – so it’s all part of that. Like, I mean, you know, that for example, true that I did learn, you know, there’s a name, initially, while basically driving a neural network, particularly able to publish papers. Because if we had a paper that had neural networks in the title or neural networks in the abstract, usually rejected, like, basically years, and not read your paper properly. They would be like, this doesn’t work. at some point in doing it. So, so we ran in things to be learning how, because now you are not doing your letter, you’re doing learning, which is a different thing. So, you know, so it takes like, uh, you part ignore the impact of all of these things, right? Like at least in academia, you can’t survive unless you publish. Uh, and, you know, to publish, you need to follow sort of what what is published. And you can do something that are led target to publish on the side, but like that can’t be your main focus. Unless you’re someone who are quite established and unfortunately whatever you want. you take up this same camera, right? Very senior people that you don’t care about papers. you don’t care about what other people think you just do whatever you you think is moving. And, um, yeah. I need to be, this will be a network. was applied to a problem and it won work. review this t up for ourselves and one. Uh, I’m just looking on this, what is, um, mood, via problems specific? Most of the, yeah, most of the problem. see is because of the problem. Or do we weight on the problems? And, uh, maybe, um, a DVD model application or something like that. I mean, Okay, so like generally the models are meant not to be probablyified the top, right? It is the the big selling point of – of – of of welin or machine is, you know, we have one anchitecture and can works everywhere, are formerly the one – the dead run is going signal all at the moment. Um, I mean, I think, yeah, I don’t know the, the participant, the, the, the, the, the binary, the, one company. Like I’d imagine it worked but it worked at a very small scale because one thing is not there, you have to train sort of new models. So shortly, and a very unique number of ways and stuff like that, so you can do the, um, I mean, and then – and in my mind, that is probably what I mean, is that the most, is sort of the scale. um because for better worse now we’re in a space where if you can’t run things of sufficient scale, basically the one they just use. So until like you can run sort of one of the dist. So, there – there is some – I mean, and – and I’m not sure if I’m answering your question. I’m actually not sure what the question exactly is, But, um, there’s definitely working quantising on it, right? And I really like the LLM we use nowadays, you know, there are ways to be their contact and then four feet or something like that, right? So I mean he doesn’t say does all early kind of. But I don’t think it’s necessary the kind of distive structure and thinking of. Like, these kind of free structures usually come from, uh, relaxation or a continuous thing. Like, this work leads fashion, actually, you have, like, a full precision model, and actually, the more precision you have is better, you train in that way, and then you have a whole process where you’re kind of, you know, this kind of relevation process process where you are, or, physicalizing your model, and you kind of move into the – the – But you rarely learn, like, directly for these. If you try to learn directly for these, like this, they’re very hard. So, usually, you take a great model of them, and, and unfortunately, back to it. Um, but, uh, this case factor I’m thinking of is, I’m not sure if this is, like, I, I, I don’t, I don’t know whether just having discrete weights used, like, what I’m thinking about these, is more of a, um, Yeah, like, it’s more like if you think about how an algorithm would work, where you have, you know, like, 3 or 3 and if you see like that, like, new networks don’t really work that way. They don’t recreate internal data structures and manipulate their new in different ways. And this data structure, they, if they do anything, they’re not sort of disputed given nature, like the way you could let me know, you have sort of numbering, uh, uh, thing. So, uh, yeah, like, you know, like, Yeah, so so I think it’s, I think there’s something about the internal state being always from the US, and like everything being projected to a continuous place, everything becoming recently. Well, you can go, like, later, in the sports, with remote, um, mystery, and the things will try to do this, and I guess, science, and all, like, one of the, sort of, essentially, new aspects that I need for, you know, networks, and, you know, with them, a little based on what they want to mean, all the intricate, and then rejected. And I have, um, like, for example, my, um, iPad, Friday, which I think, uh, Italian on network, is, is a, uh, to potentially generalise all this information, but, uh, I can, you know, if – if there is time, or, yeah, I can go use to my reasoning, though, why that people take stress. Uh, but now it’s a bit early to justice that. OK, so this slide is kind of just saying exactly what the other side of the saying is just sort of fine. I don’t know. maybe I don’t even need to stay on it too much. Uh, but it just started showing the metres and the the boom periods and you have the symbolified in the in the 40s and the 60s. Then you had the expert systems, and now they’re sort of technically in the machine learning group room, and, you know, there’s that question in the air, like, is this it? And over the, uh, this or is there going to become another winter? I mean, to be fair, um, Again, I’m not expert on this, but a lot of people are claiming that I’m making lots of money from there. And I think sort of at the point when you start making money, probably if I get to get into a winter. Like if you can still go in for a metre in terms of innovation, but at least you know, there is some technology that is not going to go away because, well, now this is how we do X. And, you know, X is a kind of service that we do, and people are buying, they can, you know, they need to use it. So, in that sense, it at least got to that stable point where it’s actually used the industry, sort of, on a regular basis, to solve, like maybe the nuts, like, I’m not going to say now take about the childpots that are sort of those, you know, the special things, but they’re sort of like more pragmatic things that are common with that bashi, where even the way the neural networks are used might look very boring and then sort of the best real to the community sectors. But you know, those are established way you’re doing such very specific things and those things are then of that interview. So, so I think there is, there’s still quality of the pressure point, I, we’ve had particularly one, I think, from a research path, people, and and and all of that, but I think it’s at least that is a very point where like, AI, now is a, this machine learning or deep lighting, that’s AI, is very, very tight. But deep learning, which is something very specific, is a technology that’s being used in the industry. So in that sense, you know, it’s not going to disappear from there. There’s no point of replacing it because you’re doing this Japanese doing it well. So, so they were at least were over that month, which is like, at least for this boy kind of stuff like that, it’s maybe not the case, but maybe not the same case, at least not. Okay, so, um, we’re going to start getting into sort of a bit more technical stuff, and then from here on, there’s going to be a lot more technical, though. I’m happy to, so, contextualise sort of information. So, you got point of view of any time when you read them, when it’s done, you know, the same? So, uh, to kind of start putting the roots of the kind of thing we’re going to discuss in the course. we’re going to focus deal it on the network uh planning and a supervisor. but there’s like lots of other things, a lot of them are not in the power. And I sort of wanted to say, like, there’s different ways of categorising machine learning, whether you’re posting, or not, your models, and what you’re talking about. You know, that’s what you’re reading and all, it’s fair on Apple. Well, if you’re looking at learning regions, like, the supervisor learning, LAL, self-supervised, not learning, the teacher learning, not providing, and, and, and, and, and, and other things. So we’re gonna cover some of the learning regimes because I think it’s’s actually kind of useful. I’m not going to focus on in your models because very typically, if you want to make theory about neural networks, you can see you’re talking in your models because you can make your body and all that. And then you hope that you’re equal for your networks. And we’re gonna go through some examples of that. Um, but I’m not gonna talk about here now, but, um, OK, so, yeah, I don’t know how this would go. But, uh, I’ve met you, many of you already, probably know quite as much, no machinery, but anyway, I only decided something very basic. So if someone gives you this data, right? So you have three squares, uh, purple panels, and then you have this circle, and they ask you, is this a square or a fan, right? How do you know about this question? Have anyone had any idea like how would you Andwise. Okay. Yeah. find that this group of the green network the crab. Right, I don’t possibly see that it’s a swing. Yeah. I know. That’s yeah. Anyone else has, or should I just go? I think that’s sort of a very natural way of solving it, right? I would have done the glasses in the same kind of solution, right? Um, so, it’s actually one on my slide. Because it is exactly the kind of sociality. So this is sort of using KBS labour, but the idea is you decide on some distance metric, whatever it is, here, it feels like a KBS experience might make sense. And also have to sign this for any other, different type of views. You compute the distance within any parairide points So you can you have your queries, you have all the other points, your computer distnces, you starting by the instces, then you look at it okay. So the ones that are closest to use for the 5 plus f and the data test. and you can look to see what pl they have. Are they square? Are they treadels? then you can be the majority class. because most of them are squ of my time is just Okay, so what is the card what is yeah. What is problematic, if anything, about this kind of approach? this is a this is a machine in the first, right? This is your, you’re describing this is a meta algorithm, right? Is that may discuss? So you’re not, you’re not saying, well, like, you didn’t have like 4 sides and whatnot and it’s a square and it has beats a pack or that. that would be the prescribed version of solving the problem. This is really like from day. It’s like, okay, I don’t know what makes the fire weather in the on p, What I’m gonna do is I gonna just look at my training set, look at these neighbours and make a decision based on that. So this is, um, and this is sort of the, this is the, the, the metal algorithm, um, and, um, yeah, solve this problem, but, Can anyone think of like things that could go wrong or like, what would happen if you apply this sort of, you know, much more, but a larger scale circling, you say, I don’t know, the tax phases in my English, or anything like that? And I have thoughts of what, yeah, how would you do exactly this week? What would be the 1st thing you just travel with? Like you get images of faces. Like, what would be the 1st thing to travel economically be South Africa? I think when it time scale of larger data sets you seem to have like Just computation issues. Yeah. Yeah, so that’s that’s all the big issue. And then I’m actually gonna just you in a couple of flight in the future. after this I’m gonna’m going to use exactly the issue to motivation the gene part matter. So one issue this is a non-parametric matterod. I’m one way to, okay, one way, you didn’t get there, but I wanted to appreciate parametics for non-parametics is that here every time I’m making the query, so I’m trying to find up another point. I have to use the entire 800 iPad, right? So I have to do this for those data, sorry, data set, do something to every data point to kind of come up with an answer. And this is a problematic, right? If I have like a video data point, so I need to do a phone for a video every time to make a decision, that’s not going to work well, right? That’s not what you want. You want to be able to do like computational one, like, you know, you want to have, like, a fixed number of patients that you need to get to make your condition. You don’t need to do this So that’s definitely one problem. That is, um, yeah. Yeah, definitely. Any, any other, I mean, there’s not any trauma, but yeah. Yes. It’s very sensitive to noise. Right. So, So, one point, what do you mean by noise? I mean, like, if one point is off, it will really affect the prediction, the decision that we get to. I mean, if you if you increase K, you become a bit more robust, right? So that’s the point of K. But yeah, I mean, if you do top K with K equals one, I imagine that you have an outlier and you get posted, you’re doomed, and that’s it. I agree that’s a problem. I guess sort of technically by increasing the value of K, you’re making the system a bit more robust because, you know, you’re not just… It’s not invariant. It’s changing a unit. If I measure one axis on kilometres and then I then I measure it the same thing in metres, then the Euclidean investors should be… Yeah, we will break the analysis. We need that an answer, just by measuring distances in kilometres from metres. Yeah. Yeah, definitely. I mean, I would go, like I said, further than, yeah, I mean, that was sort of the top of my mind, which is defining in the distance on itself, it’s crazy. It’s hard. You’re like solving how the problem just when you repeat your distance. Maybe more than 50% of the problem. So if I would go to these data set of cases, like what does it need to do? Like, obedient distance in pixel space into images? Not just a dimension, it’s just the round, the wrong business match. Like you, you’ll find. I’ll be telling you many teachers that are not faces, that are closer to an image of a face, than other images that have faces, because it’s the wrong distance, is the, yeah, basically the distance is really problematic. Like this – this method guys as thishetic. If you don’t decide, this is our, what, what, we are saying as well, if you if you somehow don’t have the right discipline, it’s not going to work. And then, you know, every time you change some property of the data, like you need to reevaluate what is the right distance metric, and you might need to change it because active change and so forth. Um, I wanted to present another idea, just so that they kind of cover a bit more things. And this is more like, okay, this is trying to make things a lot more probability. Like another issue that you have here is in the way that you frame, but this is on the next idea’ all it easy is you don’t actually the way every I was young algorithm, know they have a senseral uncertain. don’t it’s it’s harder to talk about like, well, you know, it’s about uncertainty. So one way to add the idea of uncertainty, I mean, there’s many ways to do that. The simplest way on this people algorithm is you look at your K name, that you speak, and you look how many of them are, one kind coming out, the other, and, you know, from that, you can get some kind of upright, right? If you say, there it actually seems to be, then it means, well, you’re uncertain. it could be either a sp signel. And comparing how many are squared, then the more certain they become, right? But that only so, we know it’s a different aspect, which is, we know it, that this is. I’m just looking at, you know, like maybe I have majority are squares, but they’re the furthest away, and the closest neighbours are actually all of the triangles, even though they’re just 2 of them. So, you know, so you might want to say, well, in that case, I’m not so sure I should say this is a square because the distance itself matters, right? And I think sort of fire and need those kind of going that direction, at least is sort of how I remember learning that long, long time ago, where really, like, this is now you’re not directly trying to do the classification. First of all, you’re just sort of normaling the data. So this is just a best you uh um uh kind of a way of – yeah,ing the best data. Where what you do is, basically, initially a training point, you put some kind of cattle, and just for a terminal. And then that kind of gives you like the sense of the kind of creative distribution, right? So you can you can see it here as well, right? So because this carry out overlap, the news in that area, you’re a lot more likely that, you know, the points in that area are going to be of that particular class, right? So you can, um, you can con- in fact, Um, and and obviously the – the issue we – we have in this kind of square kernels, uh, simpler kind of windows, is that you get areas where you have like 0 probability, which maybe is not what you want, and it can soften that by using lotions, which is sort of what people are doing practice, and that can be cautions, you know, they have, like, even support, so you have information about any point in the space, even though the, the, the quality, the kind of becomes vanishing, uh, picney. And and with this, you know, you can you can compute sort of your probability, your lives to be in a sweat, given the data set, by just, you know, um, from building this distribution, all your data points that are spares, and then you compute the probability of between the triangle, and then you can have this thing, then you can reason about that. And you can have a condition. What are we trying to Vietnam? Is it to get like a measure of the confidence of the art? Yeah, yeah yeah. It just gives you an uncertainty and gives you like the better way of reasoning a lot of the data. So that that’s why Johnny involved. So I was trying to compare this in a more proisticistic setup. Um, and this is sort of kind of the basis on globe building this and GGP and all this other stuff that I’m not improved. 100 is 5 So what’s going on here with this? So you wrong or this one. Yeah. Yeah, yeah. so this is like your about making your kal your point, your position minus is send because it depends on how you find pie to this is it could be a distance. Yeah, in in practice, this will become kind of the visual actors in this stuff, right? So you’re basically looking at I mean, this is just a fancier way of really just looking at the distances where that point and incorporating people, like a, like a wind sum, according to the distances, but it’s a, it’s a, sort of, it’s a, So what this is in trying to ensure by making sure of each of these files being eruptions, is that you actually get the results of it. So, you know, you got to take like everything comes to one and and all of that maybe I presume. Because I just, I have certain points. I just sort of ran and try. So you’re saying, oh, for this, I have 7 points. Each point has its own Gaussian, so I want to get a green card. So basically what I’m doing is I’m summing those things. and then renormalizing them. So I have to renormalize them so that this is actually tougher, uh, uh, just, uh, you know, that makes sense. Just sort of like a, I mean, this is not, you know, Netflix, but I just wanted to give you kind of a flavour of other ways of solving these problems. And to be more sort of potentially intriguing, but at least to me, um, uh, this kind of techniques are a bit more intuitive. Like you can kind of reason about what they’re doing and they’re trying to kind of delay it out. So I guess as you most in your life, right, students can become a little bit less intuitive. It starts becoming more and more of a black box. You have the black box, you refer it, you have the optimisation, you deploy, don’t know how to black box or where the data, you get something out of it with the inference and it’s harder to understand what’s like that. So I thought like, I’ll start you at 3 examples, the thing that what everyone did, and then maybe we’re gonna be able to kind of, uh, do something singer from your letters as well, kind of try to get to that. Obviously, this is problematic the same way, these are the thing of problematic. So here is the part of the window, saying, what’s important is the size of the carnal, right? So if I’m making this sort of red, so she’s trying making them too narrow, then I’m going to do very well on the training set, because, well, I haven’t one of those very straight in point. But then I’m going to have a lot of areas of very, lot of community where it’s going to be very hard for me to reason. So this picture is trying to to say, so this is in deep. So, so now imagine you have, like, little rectangle alert, like, I know that you should visit a point. Those are 2 small, you’re not gonna, you’re gonna do very well on your training, but, you know, you are, technically, you’re an overteacher to go into training data. When you’re have to make predictions, you’re not going to be able to make very good predictions because you know, you’re gonna have many areas where youre be very uncertain about everything. Well, if you make these 2 lives, then you start having issues even with your training data because you have these windows coming from neighbouring or holding data. So I was just basically trying to say that there’s like everything, there is no there’s hypper parameters, which is basically selecting the caral and depending on how you select that you it might or might not work on whatever task be. So so you know, there’s a to that. Okay. Hopefully that will make sense. So this method that I describing is a sort of part of the non-parametic family and it’s non-parametric because you always have to work with the data, right? So you have this data side that is describing your problem, and you always have to go through it in order to make decisions. Um, um, and in interest, uh, at least to me, interesting, the sort of non-hometric uh, scenarios feels like a search. So you always have to do this search in your data to make a solution. There these methods can be quite integive. which is a part that I like. But I usually seen as there is sort of not as much in favour. I mean, there’s a lot, you know, there’s places that really like this stuff, uh, like right up from the Cambridge, that it starts to GPUs and stuff. But, um, more or less they’re not seen as a driving corps commission learning. And the issue for that is because they don’t f up it and they don’t say up by the point they was brought up, which is that you have to go through all of your data that’s not going. So you need something else. Um, and alternative to this kind of scenarios is to use biometric, not certain. That’s when you’re at our side, and these are the ones that are dominating sort of the field at the moment. And the relationship between that, it’s complicated in the sense that in the, you know, there’s lots of work that, for example, show that even we may or scale, you know, naturally become an automatic method. There is a lot of work in trying to combine them. and and it’s Let me put it this way. They have very different properties. So it’s not like… It’s not like primarily methods are supersetical, non-primatic methods, or like parametic methods can do non-parametic methods. you know, tell you as well. And, you know, there’s no point in having both. Like, you know, you can, and replace, whatever you are doing in the primary group. There are certain aspects of like non-primetic matters like on the death, right? Because there’s no explicit, like, there’s no great dissent in environmenting matters, that you’re doing this idea. There are different properties of this that pharmacy methods just don’t have. So, you know, just basically I added this time just to say that you shouldn’t, like, they have very complementary properties. Therefore, even though that is happening in a very limited way at the moment, I do think that there is a point of merging this to kind of having a plan. because you know, they can do you want everything that they are recommand. And also, in my view, one area of research is probably going to take off more and more in the coming years, sort of mixing between this. I mean, what kind of problems do you need to do? I mean, the typical way people are doing right now is using the pharmatic method to compute some kind of embedding, and then put non-parametric on about it. than words. So basically use your primatic method to project from your input space to some other space, but it’s better behaved, where it’s easier to define distance metrics and whatnot, you know? And then from parametic, uh, non-paramedic, and of other space. Um, that is what being done more often than that. Um, I think more, uh, more can be done. The propagation is more interesting in things that can be done. And okay, let me give you another example. And then this is a bit more debate about, you know, how you, and again, like, I don’t know how near it one is, but like if you think about the Transformer, some people do argue that Transformers, eyes, sometimes a bit sophonetic and unfamily, because the attention acts almost like a non-parametric component. Because the attention, whenever you have a query, basically it does attach to the content by computing distances to every point in the subject. But it’s not really, you know, and you can say like, okay, we take this perspective, right? And we can push it a bit further, and say like, okay, let’s let’s make attention layer, you know, proper non-satimentary, kind of. So you could go that way. I don’t know if it’s that useful because uh the hardness is you have to propagate atients to that search process. So, you have to do it in such a way that it can still have a signal for the layers, you know. But there’s only an example where you’re combining these things a little bit more, they’re just putting one on top of the other, which is sort of the thing, but in our way of doing this. Um, but I think that’s kind of the place where I think people. You just, sorry, just make some understanding, this is the problem, because most people define the boundary between parametic and normalisation. I mean a week in book programme helping attention people. These weeks are? Yeah, I mean, sorry, where did that drive? Well, it’s this is parametric. Okay, so non-parametic one property that you would have is that if you increase the size of your contents or whatever, what set, whatever you are trying it. And they still apply it, right? Like if you have a weight for everything that you are, well, if I have an newvirment what would be way for that, you need to lie. Is it the same to the transformant that it will not take a time to develop that, right? int layer itself is Okay, so it depends on why you draw the boundary, why the essential size enough. But the, the, the, the, the key query is, there is the same key choreometrics that apply to every embedding, right? To the same transaction that you do, if you are. In sometimes you can think that the non-parametic component is really just doing like a sign products everywhere. So you’re just looking at this. Like, forget about the projection into the key space and various spaces. Those are just pitch protection. You can fold them in the layer below if they’re just part of the non-primetric component. And maybe the mechanism of the attention is computing, because science discusses everywhere, using softmax and normalising numbers of maximum-parametric, and then sort of averaging based on the software. Instead of going to do a top change in the top matches, you say, you know, I’m hoping this listen to the 3rd pay largest and then do something with an happy majority of whatnot. But you need a soft max because you want to be about the proper gate variants, you know. Um, it’s – it’s not that much of a distinction. I’m sort of trying to, I mean, it’s like, you know, I don’t care whether it’s easy to probably as look, everything is hard to grow, sleep, boundary, away from it. You cannot have a. I, I kind of described it this way because I feel like this is sort of maybe a more traditional way of doing the field. There’ definitely in the in the field there having the view that you know you have met and there’s like books upon non priarily approaches to be machine lin and then you have groups about primetic methods. had usually seen a distinct thing. And at the moment, usually the intuition is a primetic matter to work much better. And then basically everyone focusses on automatic matters, specifically neural networks, rather than, and you look at, you know, where I am. especially let me on networks. And, you know, so that’s why I went with this kind of distinction. In practice. I do think, okay, so maybe just sort of a high level. I know fine in practice, but if you want to do research and you want to type with this matter. The, the, you know, most research ideas, they really come from someone who is familiar with 3 different art fields and basically just taking an idea from a subfield and applying it to the other. best by far the most, you know, uh, and then, you know, there is a lot of creativity and a lot of, uh, it’s not that you know to do, but, you know, in that sense, like, in our mind, it’s, like I was trying to create some parallels or I’ll try to play more parallels, if you have parallels, but, because the idea is don’t try to box them into different things and say, well, you know, this is just a parametic, this is just for non-parametic, and they have, you know, usually like when you want to be invented, it’s really just about looking at both. I say, well, like I do, do this trick but there, I could do it here as well. We find we design things one way or another, and that’s sort of how how you play new ideas and your methods. But yeah, I mean, the ball is not a key time to say. I mean, you can try to draw a boundaries, but it’s matter the point is not 50 I think you just be aware are, don’t know, boundaries and so forth. But, yes, well, in India, I would have, you know, a, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, the uh structure of the is that you non was for a B forometric or non part model is more than hopefully uh It’s hard to say they all met by the way. Like, for example, if you have Have a single layer model and there’s a lot of things people not be able to do no matter what future. So, um I would argue they they might be on the same level or what be like they they they – I mean, it depends – depending a lot on domain, like, I don’t know, if you have a very specific question, and, like, a very specific scenario, maybe it’s easier to judge. They say, well, this – this thing, the – the driving force behind the performance, you see this or that, but, you know, they – they can all matter, by the way. I mean, Basically, okay, yeah, yeah. I Like, if you, if you have a good people’s practice, for non-familitable, then then they can be on, but I, I, I think, so. They can look around, say, well. But, yeah, I, I, uh, I would say they all matter the same. I would say that typical scenario people with 1st speaks where they are. So in most cases it would be like, okay, I’m using world networks all Max. But then you’re not even asking this kind of question because you’re already kind of set of animals using this part on your matters. Basically, you look like this, it’s a layer, architecture, that’s combined, I don’t know, MRPs, and then attention, or if it’s a performance style, or you just combine some revolution networks, or what, none of conditional areas, or something else, you’re not even asking this kind of money, kind of, because usually you’re in a much kind of narrower regime, and then you’re kind of just cleaning there, and then you can use that. But, uh, um, there was one impact they can have. I think it’s done the same same world. Okay, so this is still part of the intro part. Okay, so okay, so what are these part about these methods? But then this is what we’re doing, we’re going to read the basic stuff, and the whole point is that if we, if we go through the basic stuff, then we advise stuff, there are more code is gonna be there to, to, to understand, and, and, and then we’re gonna have much more information about that. you manage to do on the same thing. So, maybe, maybe, like, taking a step back, like, what are we doing here? So we’re trying to find this new good predictor. So what is this good predictor? This is a function where if I give it an input, so this is supervised learning, right? I use a useful X, I’m expecting to get an out of the Y. Um, whether that why is at last this is classification soput an it just part of timeangle. Um, Or, or it could be a regression where, you know, I’m pretty surprised of an object and there’s sort of like a real number. So that’s kind of the difference between, you say, modification and regression, everyone is good predictor. And you have some data. You have some examples which are vari of inputs and outputs that you expect. and like what you’re really looking for is sort of this metod, algorithm the new data give me a function age. That’s sort of what you want to do. And in a non-parametric scenario, the way I would playing this, it doesn’t matter, algorithm, it’s defined on the power set, or what possible data sets, and an observation, and easier labels. And this particular formulation I is trying to say is that the method that you get out of non-primatic architecture. So this power set thing. So maybe I shouldn’t have used age. So this is the machine learning algorithm. This is a myal version. The what you get out of each other, the inference that you have to do, go from your data set. But the whole point here is whenever you want to make a prediction, you need to have your data set, but examples, and your prediction or your imperence is really some form of certain procedure that goes over that entire set and makes a decision. And whether it’s looking for the toasted neighbours or doing something slightly different or some process of that, it’s fine. But it’s that sort of in reasonable how it works. And then the parametics scenario what you’re doing is you’re replacing your data set some. So really what you’re eating, what you can, what you can think of it, is that really you’re trying to compress your data set in the Disney data. That’s basically the process of going on here. So instead of just doing this search for the entire data sets, you’re having your parameters and you’re compressing your data set, those parameters, and then the advantage now is that you don’t, you know, this better, okay, we are very big. This data is space. is a big size, so the cost is fixed. So you know exactly the thoughts you have to pay when you’re doing a prediction. And an example of a paratic model could be a linea function, right? So, you know, you take X, you multiply it with some WN and some of your output. Yes. How is is it all known part of achievemental style, is the entire average activity? He was a. Yeah, the… Most of the crazy ones do, right? Imagine you can do all kinds of tricks of not necessarily, um, I mean, there are different varies of answer in your price. If your, your, your, um, You don’t necessarily have to go through the entire day cut to make your search, right? There, you can, you can, you know, if you have a place, make a structure for your native ID, um, you know, that made your search to be way more efficient. If you like login or whatnot, because you’re not absolutely looking at all examples, you have this data structure that allows you to navigate the space in the right way. So, do you know that some data points are gonna be very far from what you’re looking for? There’s no point in checking and puttinging any businesses without. So you can definitely do all kinds of streets to make this very fast. It doesn’t have to be this extremely slow, uh, end that I’m trying to sell to you. Um, and you can also decide to throw data out. you can also decide which there some data for in I form of another develop that you have. So there’s no point in keeping them around to kind of make your memory, or – or, you know, how expensive your forward is. Uh, so, obviously, in all of these things exist, and and then they can make things by – by the efficient, and There are, there are sufficiently many parts of the community that would argue that, you know, non-parametric are really parametric. So it’s not like I’m trying to build this sort of picture here. So by the way, most of the time when I’m going to try to build a picture for with accelgerated. So, don’t believe everything that I say. But I’m trying to say this picture that we have, this are very sluggish, not parametric method, and then a much better parametic method than everyone is on the parametic, uh, model, uh, uh, uh, side, and, you know, they think this is the best. There’s not really the case, you know, the non-privatic art, there are particular scenarios and places where they’re much better. They’re very useful in low data in gymms as well. Um, Because, um, it’s much harder sometimes to do this process of extracting an optimal data when you have, say, a 100 examples when you’re trying to classify something. So then those like kind of scenarios where, like, very well thought out non-primatical methods actually were methodic. So there is like many pointed like places and places where non-primatic things are, are still pretty appreciated and there’s lots of, there is huge literature. I mean, as I said, they’re like groups now actually working a non-partnative met. It’s not the same as, um, I don’t know, there’s some other topics that are maybe a little bit more niche that are kind of really hard to do. Like, I don’t really want to do symbolically, or something like that. That’s probably gonna be highly identified groups that are very active in that space. N-p ofametic machine lining you would probably easily Like I was mentioning, like a temporary place, and to be obsessed with someone, but it’s okay. so you’ll find probably other people doing all kind of things and stuff. So, so there is, um, that now there’s as unpopular, as I make them sound, and there actually is to make them more easy. But, uh, it is a biometric method that are sort of state of the art in most domains, but even when you are in a live data region, like when data is you, when you have sufficient data, and that’s not usually the bottom-up of what you’re doing, um, more or less parametric matters like the one, um, that are performing the back. But I think methods can perform very well in building regions as well. If you have auxiliary data that you can, like, we train your model on, and then you just have a couple of examples on your target tasks, but it’s sufficient because it’s retrained. and then you have all of these clean but you find experiments of project and log it so. Um, Anyway, I was just trying to to set up like a variety that makes sense and and – and it’s easy to kind of do. Uh, okay, so this is the paramedic model, so the idea here, uh, this is sort of the basics and from this onwards, we’re probably just going to jump, you know, on that rest, cuz that’s one of my questions going to be about. But the idea is that we’re looking for a meta algorithm that will help us find destruction age. The draws from Pita, my parameters, and an observation adds to some tradition life. H itself, the structure of disfunction, defines the family of the Marvel’s family, the family of okay, that sounds proper English. So the model family in architecture and an example is a linear model. Another example is a neural network, like a transformer or a DMLP or whatnot. And in machine learning third, when the time we lied, this model is H, what we mean is we find some data star. Besides that when I evaluate HDPAS star and some data point, it gives me an asset that is correct for as close to being correct responsible. So this is the line in staff of the models. The 1st thought is the architecture, recognition, and the 2nd study that you do there, you know, which is basically just this search process, which does start, which is sort of your optimal parameters. And in the field, usually when you’re trying to do these things, you can have 2 different concerns. One, is a lot of the speciality. So when we’re talking about the specificity, you are worried about speaking an architecture, and this usually comes by different sort of where this Tita lives, like how many, if you grades you have, how they’re arranging architecture, like what is the structure of, you know, network and so forth. In such a way that the model is that is expensive enough to express whatever you’re trying to do, right? So creativity is basically the diversity of functions you can get by changing the back. And you want to have sufficient diversity so that the function you’re looking for is within that set. So this would mean proximity, and then the other, uh, side of things is liability. So we talked about inability, is sort of how Ava, are we, given this architecture, and doing the search processes that we have in British, which will be great in the sun, to find this Kita’s card. That’s always the problem, right? Um, so usually we use sarcastic radio for that, but then the question is, you know, how likely is for grade and design to be able to discover this deep as part from the data that you have? Um, and then these are 2 2 sides of of the same point, but they start to find the GVT worry about when you set up your model. A lot of times, you know, each other, they treat separately, so you’ll find a lot of work that are just focussing on a facility and they’re asking questions of, you know, can an island do X? So you have the fashion of 10, 11, X, and usually you approach this on exclusivity for interview, and what you’re trying to do is to show that there is no configuration of parameters for the other land that you’re considering, such that you can include the tasks, right? So yeah, this is sort of a typical avenue of approving things. And usually if you’re just focussing on a specificity that the sort of different things, a different reason, and makes it easy. Uh, the other, the other side of things is on ability. Um, Usually there’s not that much work, like, group of 12 papers if you learn an energy, but they interact with each other. Like you can have a very expensive fam your models. They can express anything that you want, but we’re learning is really hard. And because of that, you will never be able to discover any new resolutions. In the same way, you can have models that are very easy to learn, but they’re not expressive. So you cannot be something. like that. The idea moving forward is we’re 1st gonna ask of the question of what is the different regulations? What I mean by that is basically what is a good architecture, what is a good choice of model family? And then we’re going to ask the question of how do we find the start? It’s a bit, um, and the answer to both of these questions is going to be, you know, on that right, and then you do that, and that’s that’s the way that you said, uh, good purpose. Like, you know, are, are biased, are structured, right? Well, we’re gonna start with a linear model. because it’s a little bit easier to understand. gonna play a little bit of your novel to understand sort of how this would work. And then from that all way we’re gonna start talking on neural networks and why would you want to join for dinner your models in your network sort of mod out and how they so for. Um, so, your modelX basically align. Um, And earlier model, uh, so the size slide that I borrowed from Antonio, I get those course, um, on authorisation, but, uh, you know, New York Mall is actually quite, quite easy to reason about. So like how would this work? Can you have your data ads, you the matrics or big data point, uh has Ddimensions as the features, and you have some fact that why? So some real number that one for food for any of these data points. and then when you want to build these models, I’m kind of repeating what I said. soart already, is you’re looking for this FWOX which is really just X multiply. I think it’s a sort of set up a notation, and and therefore, the output that you are, which is this Y hat, and that you are outputting, it really doesn’t weight in some of the features on that side, right? So this is sort of a outlook. So the question I’ve been trying to answer now is how would you find the W for it this? That’s what you know. I mean, that’s kind of, uh, so, um, if why is a real number, if your data, SMI, we’re gonna go back to classification, because classification is sort of where we started, and we’re gonna talk about having the classification with this kind of thing. But for now we’re gonna do the pressure. So let’s assume that why the real numbers? So you really just your data point and this, this thought, you know, really what you’re trying to do, uh, is a relational password, and I’m trying to discover it, and like. By the way, how many of you are quite familiar with this stuff and this is going to be some important for you? How many do you know being education and all that Yeah. right, so I’m. Okay, so and then you have high dimensional or become the hypperame. So this is we can come back if there is like any surprises, but I imagine this is not an surprise here. So Um Okay. So, um, there is this question of biases. This is 2.00s as well, but when you want the hearing. Otherwise you applies advices. When I try one theory, it’s very annoying to have that button. So usually fold it into the weights, which is by augmenting your features with a one and then, you know, you buy it trans away. And that is just more notation than anything else. But like by keeping everything to be to WX, like when you’re doing it, it simplify your equations and everything a lot. So it’s just sort of, you always can say, without lots of generality, I can drop the bias and I can just go forward. So, okay, so this is sort of what you want to do. So how do you how do you getire there’s a one way is to minimise the distance and here we use min start error to tell terms the most natural decisions when you have your number. Um, and you’re trying to look, for example, your side, that the loss is as possible. Okay, so… How do you do this? Well, assume you have some XW start, because why? You can multiply both sides, the X, and then you get this X sample, X on this type, with extra sports Y. So now this this metrics, you can work, you can move forward, then you get sort of your, um, your solution, which is, uh, just X, transpose, X, the minus one, X, transpose, Y, and this is sort of the, um, the inverse. Or another way of being 2 ways of doing this. So this is why we’re doing this. I don’t know how interesting it is, but it just sort of as deriving manipulation or you multiply things and everything. But the other thing is, you are just observing that you’re lost in the photographic, and you know how to solve the battery, the back, et cetera, the room, and the gravity is zero. So you just got a idea close to zero. And then when you said, you know, your friend beside this, you said this thing goes to 0 and then remove the So you open the brackets and you get the same thing, you know, the soil on the other side and you get back sort of the same the same formula that that’s where we have. All right. So I’ll go to the next episode. So so far we’re talking about in your aggression and I showed you sort of how solving irregation in performance, which you already knew. very much like that. Uh, so the next was the most the probability in that attention. So the reason for this is that we want to move to our specification, which, again, I probably, and then it’s just something that, um, this is a high level. Like you can do more of deep learning without necessarily caring about mobilities, but if you different things are probabilistic framework, it’s very happily in many scenarios and you can help you answer all kinds of interesting questions. There are much harder to play, more understand, if you are not on music, uh, scenario. I mean, there’s also like for lots of people actually framing things in a probability, by statistical manner, it’s more useful because they write it, some people. I’m one of those people about like it. I don’t realise the statistics are probabilities, but I don’t know how many of you are very, you know, statistics are preparing things in us, just because very more. so that’ not. So at least uh I wants like them. But it’s something that you can’t leave your voice. And when it comes to the method that it process things, you know, it’s’s called statistical things are a big part of it. But, okay, so you have art in your models, like, I assume many of you already know this, but is there anything since, like, uh, non- non-what you would expect than than raise your head? So I need your way to take this near model and compare it to a classifier, you simply, you know, you use the line and the decision boundaries, so you’re basically classifying it, you start to start with A or B based on – on the value of whether the values like are smaller than zero, right? So this will be one way of playing it, and then, you know, this will be, okay, this is all, every time publications, but you can you can make this in a more publicistic, um, uh, play much. I assume, I guess how many of you know likeistic integration? I assume I can kind of relatively go faster in this part as well, and I guess I should have maybe junker activity in an effort. But, um, but there is a different way of doing this, and that is, if you think about it from a more probability place. I, again, I borrowed some slides from a transport course. I got to go through, and the idea here is to introduce, Sigma, it, and what you’re going to end up getting out of it with some form of regression, right? Which is sort of the basic. So disappoint, the generalised continuing amount of time on each other side of kind of the models you get on top of the linear, architecture, and there’s sort of the bases. And a newer networks are an expression of this, where you move from the linear to the nonlinear by adding some this narrow-wise structure that has no immunities. And that’s where most of the magic happens and sort of most of the research is in defining these types of layers, how they, what they view to the space and, you know, stuff like that. So we’re gonna get that inst lot. But if you are still sort of in the disregulation in your case, or you have these observations, or this is going sort of very step-by-step, it’s very kind of no levels. So, be boring, but, you know, you have exercise, a teacher actor, right? all of these X, X1 XN where they represent different features. For example, I think maybe it’s on the next line. The example they’re doing here is something like movie classification and you have this sort of detector, it’ll say, well, you know, the work, worse, what do you, what do you, so run out? inside sort of the teacher, that you’re going to use to predict, predict the sentiment, whether it’s supposed to be a negative or completely a movie. So your output is 0 one for like positive or negative, and then your features are some properties of the of the of the review that you’re doing. So exactly what I’m saying. So you know, you have the video containing the word awesome or at least one or one. These are sort of your your thrillers, your your teachers, and then you wanted to say everything is possible energy based on that. And the way you do that is you assign some weight. Um, I’m running away with different words, did they appear in the, in the text and based on that, you can do some kind of condition, right? Um, so, so this is sort of the problem statement, and then we also can have multiple classes. It doesn’t have to be your normalal or long it’s usually U. So the way you can do this is you can, you have to wait in some of the features that are telling you, for the features are zero or one, if some particular word happens, you do your way at some, you can have a bias at the end, and then you get sort of these value Z. And if G is high, so you use negative weights for tinder, negative, but you wait in a positive. Like I said, well, this is positive and high and musical. It’s good positive uh sentiment, United, zero. But you’d like to make this into the classifications, or from releasing frameworks, and a reasonable authority, and you want to do other kinds of interesting things. But. So what you’re doing is you are taking your senior, push it to a function. that they normalise things between and one. And then once you’ve done that, if you do it properly, you get out of it is a distribution. And then the functional use is a cigarette, so it’s one of our one class, right, that’s minus zero, 600 functions, and the function looks like this. So, I can see if something is very positive, because to one, something is very negative, those minus on, has this particular shape and you can, you can, uh, So why now is the probability? Because between your N one and has a very nice, uh, See? So the idea is you do what we did before, but we pass it through this. And there is a nice thing here, it turns out that, well, POY plus one, equals one minus POY plus zero. um which is because of the form. So sorry. So, one of what is match is fine, show you, is to say, to show you that, well, this 2 are one man and the other. which you talk about the fact but they need some to want. Do you have only 2 positive values and some. they kind of respect this property which you’re looking for. So now, you know, if you, you can, you can train this into a, if you just need to make a hard prediction, you can use the same thing, you can compare it to 0.5. And in fact, what you get out is also, um, and secondly, other things, right? So when you’re looking at why, you get of the probability of the sentiment of the review to be positive or be negative, right? more minus what you have. So these are basically the comparison from, principle, in a model to something that has a producing interpretation in the classification scenarios and the binary classifications. So not you by using logistic function that you sort of re normalising your output in such a way that you can now use them as probabilities. And then your decision boundary is going to be as .5 for dollars I can. So, man, yeah, I mean, there’s a lot of nice properties of this particular population and this particular graduation functions. So like this, this point here is also the unstable point. So, you know, the amount of learns, it prefers to go to the extremes rather than staying there, which is also something that’s quite restored, because you want more of the half. Yes. I’ve seen a few times that in order to group some models, predictions, we can do this decision boundly. Is it? Is that the problem and isn’t that it does produce better results, or is it not just about in the problem that the model is about you just probably friendly? Sorry, can you, I maybe missed the question so the 1st one. tuning the the decision back. Um and also the classified. Um not this mask the fact that the – the class very by class? even though it does produce the results. Is it just that you for performance? You would mean, like, maximum project kind of things to make, not even necessarily, like, yes, yeah, it depends on the solution of my data. maybe, like, moving the zoom value from .5, so there’s no one percent of the other, and, you know, some relevances of my first time. Oh, Oh, I see. like people playing with where, where do you see your boundaries instead of using 0.5. Um, I mean, I, I, yeah, like generally, yeah, okay. So generally this happens a lot when you have imbalanced data, I think, and that’s sort of a 50 scenario where you play with this. Um, I, I mean, I haven’t played as much in that scenario as I should have tried. My, my, my, my knowledge is a bit shagy, but I, I, I, I imagine that by just um, subcentering the data or or doing other kind of tricks to the data to correct for the imbalance, would, would work equally well, leaving the decision boundary where it is. But it might also be that these things are actually quite equivalent. Uh, I haven’t quite it out in my mind at the moment, but I feel like whether you’re playing over it in decision boundaries, or you’re playing a bit of the data, to make the data be more balanced. But th- they they kind of did the same thing. I would trust more, uh, not playing with the city boundary, but actually trying to figure the data, particularly if you’re using data augmentation, which adds something extracted, instead of just upsetting the very popular class, and then reducing your own data set, which is not ideal always because it makes learning harder. But if you have a way of absenting by adding their augmentations of all time, I would. Yeah, I would believe that would be the optimal way of dealing with this. Playing the decision boundary, I agree with you, because of the dodgy, but I think that’s exactly the same as the substance, you know, in the technical sense, particularly for the logistics regression, right? When you have, uh, at least, you know, an attraction, it’s become, maybe, really well-matched, which is good for, yeah. Uh. So, I mean, this is the point. I always, I mean, for, you from, like, uh, uniform, like, this. Um So I’m not sure, I… So, what do you mean by me from my computer? I have 05. Yeah that’ like beautiful So show me the data some are distributed for me thatat one quite likely. Yeah, yeah, yeah, yeah. yeah, yeah. that’s usually just somethingion that debating it in all of this. So you assume that your data set is balanced, where it means that the likelihood of, uh, a priori, without any information on the data for yourself, you assume that it’s equal probably different. there are point to be of any of the classes. This is the standard assumption that done and uh in information. So you have the imbalance kind of scenario, which is usually seen as a niche, and then you have sort of all kinds of techniques. But most of the techniques in the imbalance case, when there is some, like, prior on the distribution of the classes, they are usually just about making the data set balance. Most of the techniques are like, okay, more things in such a way that you have legal number for all buses, so that’s exactly what you meant. And then – and then the playeration. I expect that my application number, we should be in for the bus, we can reprove. Um, So you’d expect that like, uh, If I implement on the balance, how… Yeah, yeah. I mean, you expect that like, okay, if you don’t know anything about it, without looking at the data point, the property of the data point to be of any of the class, in the VTD, yes, that’s that you’d likely expect. I mean, there is other things that people do like when, but a lot of, okay, so maybe this relates to the 1st question. So, like, one thing that you can do is you can pick the model, and then maybe you can play with the decision boundary, and then kind of move it around. The other thing is, you leave the decision boundary where it is. And you just learn the weight to push the distribution to be another one. So, like, for example, one thing that people like to do is to have this max marg- margin, sort of lining, where basically what you’re doing is you’re pushing the weights when you’re learning in such a way that the decision boundary that is present, this decision boundary, ends up being sort of furthers apart from both classes. Like you see, if you imagine the classes as being coming in some some areas, like you want to cut exactly in the needle between those areas, right? Um, So you could – you could do the same by displaying with where you put on the same boundaries, but usually people don’t do that. In machine lining, usually, actually, well, at least the part machine I think I’m aware of usually don’t touch this. You only touch W and B. So, you’re, you’re just like tweaking your learning, um, you’re adding regularisation terms WNB. such that you push out of the learning to discover. So I said that this decision boundary is where, like, where you want it to be, without you, like, playing with this number. A lot of times you don’t even need to make decisions. For example, in another level, right? Like what you want to do is you want to stand for. Like, basically, there is not even, like, what, the – the desired output is just the probabilities. It’s not even really a decision. It’s up to the probabilities that – that you – you need to be more welcome to something else. Okay, I don’t know. Hopefully this is clear enough. I mean, yeah. Okay, so you can you can do this street to do that. So now the interesting part is how do you do the distant traffic? Now like this so you can apply news by er as well. Yeah, that is the wrong wrong thing to do in some sense. because it doesn’t take into account the time that you have this probability. So it just feels like its around loss. And then there’s like senses, you know, changes around lots, right? You can see that by seem to be playing with it. Like, say, the right loss would be negative, not black people. But you can see like what happens if you take a system like a SC one, the software, and you have to layer, and you try to try to make better. So it’s going to try. train? they are a bit dog but performance well. I mean there’s like I biggest reason for this I hopefully will get into some of that or you can anation. But, um, in in, in machine learning, usually there is this concept of a matched loss function to get relation function. So usually the way people hear these things is I guess, this is from the pro going to you. Spending what your choice about, too, like whether it’s a linear model, or the output, or whether it’s a signal or whether it’s a softmax, or it’s some other type of activation, usually those leads to some probability interpretation. So if it’s a linear model, you assume that you output on your model is like a dalshan, discussion, distributed, it’s, you know, it’s a signal itself. I norm. sometimes they want you know. So um So because there’s a provision interpretation of it, then there is a loss function that is matched that probability interpretation. And hopefully we’re going to get there, but the whole concept here is that the, uh, matched, uh, loss, uh, for for that particular process, interpretation, just comes from, um, basically using tile divergence, as a decent machine, too much, Ontario. So you can show that if you use kayal, as a business, or maybe right away, and then you work right, you basically get back out of it. Maybe you can go back. this that so important so. So it’s it’s basically just because you’ve made this probability of reputation, you have to now respect the policy of reputation and come up with a, um, with a distance between, you know, the taste of the contribution that you use. So you have the distance and internal distance, the further, but you have the distance, the tile, and given that distance, you know, and given the proposing assumption that you made, whether it’s adoption or a binomial, if you work out what that means, you get sort of a simplified loss, which is sort of a match loss to that. So, in a much more probabilistic kind of framing, that would be the way you would approach the problem. The way I learned this stuff, it’s usually you have a set of occupations that is always linear to the regression, it’s always software, it’s really presentation, multifacetation, or person-wide, if you’re going binary presentation, and then you have the losses, and you just learn in my disk, everything. If you use see my use business US of nice. At least that’s how I line them. I, I, I learn or I figure it out by leaving books much later, but there is a reason of why being this lost for that activation function rather than just so I can do whatever you want. But usually if you just learn sort of group that way and I’m sort of how this means sort of. Um, Okay, so let me let me done. So the way, you know, you usually kind of entire decided sp rice is from the port I haven’t written the sl myself. But the way you get to this, if you want to get to medicine, go back, if you say, well, like, what really am I doing here? Do I have this holistic interpretation? So I know that the zimo is taking a mobility. of why viving on a certain class even your obsation act. So what I want to do is I’m looking at the likelihood of the 2 labels. So you can write this product, you can easily see because Y 0 one, then either be one or the other, is that this product simplifies to something that would make sense, right? If a true label is one, then the quality of this is just white hat, which is sort of what the mouse is, you know, the problem people in this week, they were, what, I don’t know if you guys remember, but that was sort of the convention we had, right? The model, the kinds of probability, and in the probability of the past being one. The prodigy of the class being 0 is one minus that thing. So, you know, if why is one, it’s a flip, and all you get is quite happy, probably of the time being one. I mean, the true level by zero, then this time disappears, you just get the 2nd term, and then 2nd time in just one minus by half. which is over here over the past be zero. So this product is really the probability, I guess here it should have been POY had equals Y U the X. So this is sort of what we want in Mexico. And we want to maximise this because, you know, you want, um, You want the, the training points in your data set to have hydrogen UP under your model, right? Your model very competident, he says we’re training point part of the correct class. So this kind of makes sense. Hopefully it does. So you are a maximum. is this? Uh, where you do it, each again, it’s convention and uh, makes things easier, uh, on my table, and you take a lot of this, because, you know, you, you just kind of usually move into a lot of place. So you know, it’s a lot of space to just the alternative formula here. And then you will also additionially have the sound this overall your trading fund that you have. And then again, this is an organimization problem, so you need to maximise the load. But uh I guess what people prepared machine money again, this is convention. I think I haven the opposite. want to maximise results. I mean, machine learning, you always want to be a model. here is friend around relation. So, you know, we have this, but here, the ball, uh, we should maximise it because we want to maximise the building of the direct class, even your observation. So if you want to go back to some organisation problem, we just add the minus on the front. now now I have to nameim I it and then this turn out to be just for profession and then the sort of the – the reasoning of how you get a person. So why is person to be the right objective is because, well, my person that he’s trying to do is maximise the probability of the correct label or only in the training set. So this are kind of a individual step-by- self definition of that. And this is sort of the kind of correct of the difference that you need to have. And this is present, I mean, this is trying to go back into the results. I just I don’t know you know, terms if we go travel around, but it notget has a minus loans like people white. So that’s – that’s a very tough. So there is a, I wanted to also mention that you can get something, um, a lot of different way of kind of, there is multiple ways. Once you move in the, I think that’s why people really like to paint things from closer point of view, because lots of type things are worse in point of view, there’s a lot of communication that I do and guess, overall time, it’s, it’s a kind of, very flexible, pretty much, which is what you want to do. So I I hear some flights there where you can kind of read write the same thing, but start from the main r. Um, where, instead of training, so the way we train the problem before was we wanted to maximise the probability of the trill label over my training itself. Here, you can say a generic way to kind of get your objective is to say, I want to, um, find the probability of the correct ways, given my data, and then I’m using the base role, aside from this, to get back to the trust centre pilos. So you can say I have some data, and what I’m looking for is, what is the probability of, of uh, you know, what is, yeah, probably a few times you have this, this data, but, uh, so I wanted to get some of these, but it’s key, because I don’t mind, but here, and there’s more of this must here, we might get a start, so this is the most likely, uh, other distribution, is sort of the pattern, it’s talking about, characters that I’m looking for. So basically, okay, so the returning here is that’ll be more derived. kind of assuming there exists a keep us stars, that’s solve my problem. but we assume what we assume that there is a institution of Aita and the distribution tells me how likely is that te have to be sort of the optimal parameters even the observation that I have, right? And the assumption here is that, well, I mean, this is sort of different in practice, there might be multiple people stars that would be equally likely to give the optimal things, even in the data that I have. So what I want to do is I want to learn sort of this stuff here. I want to find my distribution of our TTA given the data that mean how likely is or just up to be the of mobile test. And the way I find this is that they need the page rule and I provared things factwards. So, like the base rule, you know, it is, uh, you know, able to be, but still be the name, but it is, like, you know, modify with, uh, the weight by the recording, by being messed and most around.. So I can use the base role to invert things because I can, you know, I do not know POP type even data, but I know PO data given a deduct. because that I can just evaluate the health. And my model, I have my here so I can do that like what is the pub date I can push um uh the inputs from my model like in the outputs and I look to see how likely the 2 labFi data, right? So this is something I can confuse. You tell me, I do not know. this is what I want to do. obviously the. So, the whole point here is that another way of being driving this whole thing is to start from the base road. Sorry bit but we start on the base road and what you get in the end is the same objective as before, right? So you end up for the same kind of object. Um, So, you know, you do the reversal and then you just sort of, um, you know, again, because you want to minimise, you put the minus the drone, then you take the load, and you get a negative of light code, which is what we had before as well. So there is a difference, I don’t if I left it here, which is if you start from the Beijian perspective, because of the paved rule, you get sort of this additional term, which is um, okay, so because of the base, you also have this, uh, in the from the page room, you also have the probability compared over your parameters. So this is just ignoring it so you, you know, because you’re being sort of propertyation. Besides, or just, you know, you have probability of your parameters, even the data, because the probability of the data between the parameters, times the probability of the parameters, without, you know, in knowing anything about the data, dividing up the probability of the data. So this like a parameter, this is your prior. This is sort of your belief over like what I’m doing for parameters, parameter values, many, which right working, without any data, right? So you can have a learn from the prior, so you can say, you know, everything goes. There’s no assumption, or you can have any foreignology prior. You can say, like, I believe my diameter should be small in order. This is otherwise a difficultical thing that people would do. And then that’s just because we like small numbers. So say, you know, and but this is like it has nothing to do with the data. It sort of just your general belief of what I mean for value for your parameters. So you have anything like this and a proper Asian person with art, you should always have a prior. You know, you should always have a belief, you know, even if you, before you’ve seen any data, you always believe what is likely and what’s unlikely. But the, the, the thing is, if you start from the base role, and you’re trying to derive your objective this way, which is sort of the more proper way, so to say, when you apply the log, you get plus a term, which is sort of log of the probability of your parameters, right? So by jump ahead, the thing here in being is comes from the base rule, is that thing Yeah. it’s that thing that you have the minus because you put the minus in front of the whole thing because you instead of maximising our minimising. and this turns out you know are going to be your regulararizer? And the role of the regularizer is to say that you can have multiple task that seem to work for your data. You have some preference. Which of those people you would rather have? So say if you have multiple values of data, then you use your error. If you have the small non-prior, you basically say, well, give me the one that has the smallest magnitude among all of these possible data that means your error. This is a way of ranking sort of possible solution that you have. Obviously it’s that doesn’t necessarily work like that in practice, but that’s sort of kind of diseat it. And then you have this next turn which is just the pro of your data. But this usually disappears because, you know, this does not depend on your parameters. This is just something about the data. So once you have this thirdd and you tell the very endength of it, you have very any sense, this time is suggesting but the variant of this respect data is zero. So this time is usually ignored but itgn know because you don’t have any control over our data. Data is the data. When you’re 5 or 8 is 5, but, you know, data is given to you and you’re not controlling the data or anything like that. So this is not a fact. The only thing you’re controlling is the moment. The only thing you can do, the imaginative is not by because, and it can also look at your prior, and it’s a prior to, so it’s not a data basic solution. So let me see what the next slide, please. Okay. so it’s prearily and I say this is sort of this andction. So we can take half hour, right? and then we can continue the next dollars. I’ll try to, so manage the name, okay. let me let me offer me. So mine is sending guys like quite everybody simulation and this kind of stuff. So I might kind of keep my hand and need the ne network person next session because I feel like that would be more interesting and better of our time and I added this section because I didn’t know exactly what we said, but I I see that they nice view point of it. Uh I might mentioned the uh I might might keep the point about the cayak because it’s kind of useful. Maybe a slider to it end up into the new network stuff afterwards. and then hopefully sometimes I’t always gonna speak up and it’s gonna become a lot more like want do that

LECTURE 2:

Um, and, um, these are, like, 2 standard scenario, that people, they are, for example, what, you know, what, some do. One is they cannot separate the 2 moon. Which is obvious, like there is no way I can rotate this line and move it around. The boundary is not union, so there’s no way of hitting a line of humility. Explorer. And maybe the very… I don’t know, I can explain the high baseball, thinking of the… kind problem which is the same reason Things are not very acceptable, right? This is You can do that however. You need to go protection your day, right? Like, you know, like, here, you have uh, sometimes with circles, there’s no line item here, and stuff like that, like, means, versus with the rent, and it doesn’t exist. But if I project all my data points to just Arabias, or, you know, how far the data points from the centre, using the centre and zero here somewhere. Then, like, and then I look at the, the day after this protection, you know, things are even really simple. And everything is fine. because was called Uh, so you have to try… I, I, I mean, I just think this is very intensive. I wouldn’t be surprised. I took this from an example from another that they were trying to give an example of like, I wanted a projection that I can easily write down mathematically and it’s very. So I went through the radius and said you want to know about the latestency. But I wouldn’t be surprised if there is a particular name, but basically, they just need to put it differently. I think in the 90s, computer vision in the 90s, maybe early 2000, it was all about this. You know, they would come up with all kinds of projections, much more interesting marijuana, I wrote. But all kinds of mathematical function protection will call them features. these are feature engineering, and you’ll get an image, you compute this sort of battery of features, and then you put your linear model on top of it. And that’s sort of what computer vision used to be. And, you know, you have shift features, another kind of feature that are exactly something like this, right? That you continue. Um, And then region here is that, um, if you take your input, you push it through projection, some non- some nonlinear subtraction, that’s important because why is it important? What happens in the product in un? doesn6 the ring I think. Exactly. So in the projection of the lunar, you can modify these to linear things and, you know, linear to a linear linear. So that means they should have been a linear solution to start field. If, like, you’re defending your projecting your coffee. So exactly. it doesn’t change the type of thing that. So it has to be nonlinear, but you know, if you have a good, non-biniar places, then, you know, then you can put sort of your generalising your normal jobs. It be intergation, your logistityation and whatnot and things like, well, and then this has been what people have been doing for a while. But the realisation or the fact is that picking this correctly in a transformation is hard. And sometimes it’s basically solving 90% of your task. Just coming up with the protection is basically solved the thought by hand. So there’s not much to learn afterwards. in some sense. So a big a big question in the field, and this is sort of working around at the time, G is, um, can we learn? Can we discover the projection itself of the data? Because we don’t know how we already want the right convention is. Um, and then this is all about your methods are trying to do, or, um, and, and, and, you know, to do this, you need to pick a, a way of formalising what structure in the protection so that it’s not above, right? So you need to have a structure for it. And the structure is sort of this layered, architecture of murals. And this is inspired from biological models. So the way the field has started was in sort of the neuroscience, more or less, where people are trying to understand how the brain works, or it doesn’t, you know, the field hasn’t started by people asking the question, here we learn a projection to make things really inseparable. They were trying to understand biological neurons. So biological neons, I’m not a biologist or neuroscientist, but you know, I think everyone knows this. They have this kind of shape, they have an accent, and they have the Zendra, and they’re sort of connected to each other. And so the other people have done is sort of the death track are collecting inputs from neighbouring neals, so there’s sort of this rated sum. And the input, this input is being sort of connected together into the sub operation from here there is and I’m going to go back to it. There is something more, which is there is some no any transformation that is happening within the Europe. And then this is sent forward to other un know. So that’s the other. So this is kind of the structure of, you know, so you have a normal idea that is, I’m gonna talk about it. And furthermore, the break is very… I think this is a very sort of traditional view of the rain, the particular part of the visual pathway. Um, I think there’s questions about it because the brain is is layered, but it’s also highly decurred. So you have sort of feedback connections throughout the brain, that we try to be more in this sort of way. So there used to be, I don’t know if they still do it, but there used to be sort of this stoyish view of the of the of the visual pathway where you’d say, you know, the information goes from from the eye, it goes to the UR, or V one, it goes to be two, from 23, it goes 24, from before it goes to the IT, and, you know, things happen from then onwise. So it used to be that like you have this year ID or stages in the how vision was processed, and this used to be sort of what people would argue that would happen. I mean, do you want, you do edges, a knife detection? And then in detail, you start looking for shapes, and then in B4, oh, there’s no D3. So, as I said, so they skip these And before you start doing objects and in I you have faces, right? So there was… A yerarchy, a laird yerarchy of how the pathway works, and there were some of the semantical differences between these layers. They were composing, you know, there were like building up more and more complex features. And this is sort of what the neural networks are trying to mimic. So this is this are the intuition from the I mean, like a lot of it, I don’t know, 60s, 70s, before that, 50s, uh, in how the brain works, and this is sort of what, what, uh, the neural networks like in the day were trying to mutate. So you have like the… What was it called? Neuro, neuro, commuter, new. Exactly. Thank you. That was basically inspired by this, right? So, um, okay, so this is the architecture and it’s, it must not be here. Why must not be linear? we already talked about it. If you don’t have the nomeniality, when you layer, this architecture together, like it all collapses, because media composalenia is linear, so there’s nothing there going on. So, a key point is to have sort of this on inality that you have after you put in the sequence, right? So, so this is the inspiration and sort of this is what you’re in order of are trying to do. They alert a inspector by doing these steps. But of course, you have some bunch of choices here, how you do this linear, what you’re doing these weights, and how do this in your projection, and how do you do the long, yeah, one choice of them, you have key, either. Um, and um, So we ext these choices are hard to try. The way people have started was we used for the symbol linear test in your production, which is what we do nowadays as well, we say in an MLB. And then as activation functions, they used to use sign modes a lot. And the main reason why I signal it was kind of the starting point in playing with this is because the signal it has a probistic interpretation. So the way they would see it is sort of the same way, but you get the probability whether this neuron should fire or not. And the whole point was, instead of, so biological systems, negation of service, you know, they work in impulses, they don’t, they’re not continuous, right? They have these trains and the pulses, the days before. like in the artificial one of thegin one that because you don’t know how to do this kind of speak me so you want contin your values. So we replace all of that with like the probability is higher. then very touch interesting might especially more kind of American point. But obviously these choices join out the idea. So then we’re gonna take a bit more math perspective to kind of get a sense. And in particular, um, Not necessarily, this is how it felt right, but But in particular, I think one activation that is widely used right now is real, instead of simoid. And we’re going to talk about why people ended up switching to level protein weight and so maybe what was the, what was the, what forever before? and sort of what I did to the using weight. But a simple function is the Ralu. Um, and the reason I was going to start with Brau, and then after Delco, Resimo, it is because Brau is a bit easier to understand, mathematically speaking. So, you know, that allows us to be. So, okay, so if we have, like, a single layer of a neural network, following this sort of 5 under this, right? So, as I said, the visual… The visual, you know, had this, like, 3 or 4 strages, or like, few or 4 layers. But for now we’re going to just start with a single layer because it’s the be sort oft to understand. So if you have a single layer, like what happens is you have your input, you do some in your projection, so here’s frame as a protection, but we are, and that plus B, you apply rather than is applied individually to each neuron, because that’s sort of how the neuroscience, the neuroscope as well. So then the non-reality happens within the axle, within the neural sub. And then you have your linear model on the top. So what is nice about this architecture? Well, 1st of all, you can you can now solve problems that work before unsolvable. Um, In particular, the random model, it has a piece-wise linear boundary, and it appears like leaner boundary increases because you could draw this kind of season boundary, that will separate the 2 modes, right? Um, So, uh, maybe I’ll stop on this like… Is it intuitive? Why it is a besides linear product? And For this particular problem, like integratively, like, would you even know how to set up the weights of your model to get a decision boundary that looks like this? All right, then you’re having regions about how these things actually work. So you can. the coming flight we’re gonna get to this. But I just wanted to make sure that I, that I think, I mean, maybe this is obvious. Like to kind of get an intition of how all of this things work. So, um, You can imagine, again, like we were typically kind of talking to this perspective where we have sort of indomain and out of domain. Like we really don’t care about necessarily the behaviour of the model. I’m entire real accesses. We care about the behaviour of the modelels here where we have the, right? This is how we start thinking about. And then when you’ll start thinking about the key that it one at a time. So you can say like, okay, a starting point for the decision boundary is I have sort of this straight line, right? Which is sort of maybe where the linear model starts and, you know, up to this point. So I can take my 1st human unit. and say, well, it’s 0 for negative numbers and decide because I don’t care what they’re not doing. But then from this point onward, it does, sorry. I should not have the same. Um, From, from, from 0 onward, we have to be straight by it, right? That is fine. Um, and then I can do and ask what is the 2nd li does. And I can say, well, if even the 2nd heat in the match, I can I can use my bias in the linear protection for my 2nd season unit, and make it such that the 2nd hit community is 0 up to this point where I want to sparkle in the 1st few sides in the net, right? So I can say, OK, up to this point. Actually I will ask all the to be 0 off to that point at least. And say, for the 2nd unit, after this point, it becomes active. And then once you become directly with a linear function, and you just need to be a linear function site that added with this previous one, you’re going in the slope to that. And that’s something you can compute. I mean, I here, I’m just doing the high level infl. But like you, you know, you know how if you combine to linear things, you get a linear thing and you know what it would be, right? So here you just say, after the 2nd after his point, all of it, the 2nd view of it is basically as the delta that he needed to change the slope to that part. And then at the next junction where you have the next, please, we make the 3rd feeling and then become active. And again, we can change the linear as well to anything bit more. And then in the next step, you get your foot and so on and so forth. So if I have like a one B problem and I one by hand to construct a particular solution, I could do it this way, right? I can order my heathen units, get them to activate the different points on the real line. that’s where I’m going to have sort of the the joint between 2 in your pieces. You know, I besides linear function. And then every time I have, like, energy, the internet becoming active, I can change my linear slope anyway I want because I can do whatever I want. And I can’t go backwards because when it’s a function, but I can I can change it to anything that I want. At least sorry, I can construct sort of any kind of piece-wise, at least in one week. I can do this in high dimensional, but yeah, you can. So do you mean that every new one partitions our space into 2 regions? That’s what you mean? Yes. So that’s piece-wise linear. That what it means. Yeah, PCI is in here, it means that locally the function is in here, it’s sort of a composition of, of, of cleaner pieces. That’s some of the functional exercise, you know. This, but the way this happens in each neural, it’s the space into two, and it changes the linear behaviour in the moment, just means. Is that why we say we can represent any, uh, content with a single there? Yes. Yes, yes. Yes, exactly. Sorry, I have that. So, um, this is not a proof. There is there is proper proof that you can read and that technical, no, that. But this is kind of the way to the universal opportunities. here, which is exactly way to ask, which is that travel network, I mean the universal approximation from 189 from they’re all 1 monings. But the newer ones are for, uh, well, because we just got back in the day where we want, I think. Uh, but basically what they’re saying is that, um, If you have an infinite number of units, if you get approximate temporary as well, any service. And in the rail case, this is a one D case, you can see how that can happen, right? You can can start if you have an infinity number of min pieces, you can ad them any you want and then you can follow the other, right? So this is, this is the, the universal approximation of theorem. So these are the big things. Um, Sorry, they were, they started some document, I think, it’s a big thing about exciting. really got a lot of people working on normal letters because of it because people were like, oh, you know, this is the right, right? It actually says a lot less than you would think. It’s actually not informative at all. as a theorem. Because it is, um, It’s a, it’s a, it’s an argument about expressivity. It’s in the limit. So it’s no way in our means you can never have it practice. And it doesn’t say anything about their ability. Like just because I can express those things. I mean I can learn them with any kind of many hours well. Any particular, that other things that are different for personal interests, that we don’t use, like polynomials, you have, you know, you can do whatever you want to call it. We don’t like polyonians, you know, we’re waiting for non uses a nice data, you know, that’s right, because they don’t work well. So, so, uh, It is useful, but at the same time, it’s very unusual. It’s not a reason why we neural network. unless I said other thing. But it came on the back of the previous AI winter, where you had a sword problem, things like this, where people are trying to show that, well, they think you can’t express. Well, this year, people were saying, you’re not going to be able to find another example like that, I can express everything. but Yeah, the question, the PDS, right? Yeah, previous, sir. Given that, as I said, we have one, one, I think between the number of things, more boundaries, or drawings, and the number is in units, within the same day. How does it translate if attends that 2 layers? Exactly. That is the next sl about. Can you just understand itly? Yes. Yes. I think also? or? So you get exponentially more linear pieces to prepare you make the model. should more profession. Should be more efficient, yes. So I’m gonna go on all the details. So this is exactly what the size. So you have this stuff, right? So this is all about the female people like I said. But in did like in look at the music church, you talk about that, right? If you look at any of the paperapers even now, it’s come about having very demarble or 20 class layers or 100 there so forth. And actually, you know, when I said there was this rebranding from your own efforts to be learning, this was basically the boundary of the dread, prevent, you know, like a deep lighting, and a deep new method that had at least 2 layers or more. while neurural e used to get similar. So that was sort of how they rebranded and how they pushed the field. But anyway, that was very important. So then one big question is, why, you know, what do you get? Like if you have universal opportunity with a single layer, what is the importance of that? What does it provide? And the answer is basically that it becomes in sometimes more efficient, but then it will turn out to be allowing for motivation as well. So we can. But maybe before that, um, How do you do this? Like how do you how do you argue? So the way you typically argue, and I’m already kind of in start the position, is you find a metric that measures sort of the expressivity of your model, the flexibility of your model, right? And imagine that you typically live in a red architecture is the number of linear pieces. So the number of trainp points or number names are whatever. I don’t know what the write image is. So feel free to correct me if I’m using character the the sound of where. So if we started a single layer, you can actually have the maximum number of of linear pieces that you have. And this turns out to be a very known result in math. So maybe some of you provided know. This is for the, the last is here from the 17th. Does anyone know how it works or what is that? Do you even like Okay. So in the match, guys, well, the last team was trying to visit, I don’t know if you’re on the third, but that, especially is not German matter. So this is not a missionary. So the last 3 was trying to do was trying to understand if you have N hyperplanes in the space, into how many regions can you speak the space? This is the question when the last is trying to answer.. So he is more about hyperane arrangements or line arrangements and plane or anything like that. But then it turns out, as someone pointed out in the session before, that you can think of a hidden unit, a splitting a space into two, right? In the region where the hidden is 0, where is the region where it’s active. So it is called active and active. So you can think of it as being a line or a hyperplane, if you’re in a high dimension space, right? So then the question is Rashi is asking is exactly the same, right, right? each unit unit is the hyperlane, then sort of how many regions are you split in the space? Well, how many visions you get by have, uh, an hyperbase? So I’m just going to bother it because I keep, sorry, I think it’s kind of interesting and it’s kind of related to what’s going to happen with the people. So it’s good people don’t know it. The proof is actually quite simple. Now, I mean, I’m not going go through all it. about just going give it the large levelation. So the way the proof works through by induction, and it relies on the simple facts. So, like, let’s do it into the, I don’t want to do it anymore. right I miss spaces where gets messy. But you can look it up afterwards if you’re really into this one. But in a play, it really gets into account, the fact that 2 lines cannot intersect in one person. That’s all that you need now. It’s 2 lies, the money is likely one point, and if they buy that change. So if you do you start releasing a. you can only stream the space too. We have 30000 business, that’s it. So now you have to eat the minutes. So you add a new line. So you had only one line before, if you had a new line, that line can only get exactly the line that you had in one place is. So then you can count how many new region it creates. And it creates 2 new regions. And then you go to the 3rd unit and you say, okay, I have 2 lines, I have a new line. This line can only intersect the other, so you only have 3 intersection points because then you have 3 points. And then at each intersection, you can count how many regions get added. And you always assume so, you know, this is sort of you always assume that you intersect with many points. you can to maximise the number of regions you create, every time you add the new line in the in the plane. So that’s kind of like how the construction works. And actually, doing the proof, it’s quite easy for me to know the outside. And you just need to work up. You just see how you, I think design is, is that does that make sense? Why you want?one okay with that? construction? Okay, so, so this is very nice because in, well, 1st of all, it gives you a way to achieve this sort of maximum expectivity, maximum partition of the space, then it gives you an exact number, you know? So this will be the limit of what you can do with a single different layer, or a peak size model. So now the question is, what happens if you go deep? So what is What is the crucial limiting factor in the number of regions that we get when we do the last 60 year? So the limiting factor is with the fact that 2 lives cannot intersect with one point. The number of intersection points is limited, and you get a piece number of new regions per intersection point. So the growth of this thing depends on how many places you can get in place at. So what happens when you move to a divide detector? Yeah, okay, sorry. How many how many of this makes sense? So, it’s it’s a little bit, like, if you just look at the architecture, you know, it looks at a bit scale. But one thing you can do is you can look at a hidden unit on the 2nd layer. So I’m counting layer from the bottom up. The 2nd layer is after you had one layer. So what is the function that a human unit on the 2nd layer designs? What is it like mathematically? What does it look about? I think it’s a function of the 1st layer. Yeah. So, but what is, yeah, compound function? No, it’s at, I mean, all of you arrival, like, the word I was doing for, is, they have a pizza. Oh. So if I look at this before the activation function, maybe even if that close. If I look at this unit, and I write it down, it’s one in an ARMFP. But you just le your direction I’ clean your attention. That’s that’s’s a human image on secondary. So then if I’m if I’m looking at the name of the space, how is it going to apply? Like if I if I ask the question, when is this unit going to be zero? If I want to draw, yeah, the function, you know, H to Y equals zero, how is that gonna? It’s just an MRP. What does it mean? Why are people sometimes that? I use Europe, but like people sometimes what is that? I never mean. So, like, what do you, like, okay, if you go backwards, then you have, like, what do, what do you call, like, when you have an output of an RP and you say, well, you also do for 25, and we draw that, what is that? I mean, this binary classic. That is the decision already.. Yeah, like if you say, well, what is this your name? that will look like a decision about me, right? And it just looks at the decision boundary repor a single AM P here. And I said that this good side of preides. So if I take a 2nd unit or uniform 2nd leg, there is just a one layer in my feet. And if I ask the question, when does this music give us a const, It doesn’t matter the constant. That’s just sort of something that we can. I can tell you one people 0 when people prep five, it doesn’t matter. What it is, it is, that’s almost like a decision boundary. and then is the T function in the eospace. right? Does that make sense or am I lost many of you? Can you say it again? Can you elaborate? Okay. Okay, so, if we have a single, then we have an MLP. or a single, right? One layer, I’m going to give you the single opt, right? And we want to draw the decision boundary of this MRP. So the decision boundary, you know, it’s sort of a very overloaded term, but we use it in terms of prediction or whatever you do. I mean, a decision boundary is asking the question, I mean, if I write them on the board. if I have maybe if I write on the word, I make write the but you’ll find out. Um, So I have H1 equals like WR. I’m gonna skip by seat because of we devices are not important. And I’m going to use this one for it, I will, because it’s easier than right in well all the time. So it’s, But instead of WL, I’m called WW, you… Um, 2 W one X, right? So this is this is a one layer of health here. without biases, there specific hurting. So as I said, we can have viruses that help, but the Indian amount looks simple. So, decision boundary of this model usually is asking the question, where is age one equal to some transcript? So this is the problem you’re trying to solve when you want to draw the decision language, right? So, for a linear novel, when I ask, when is HR, we focus on constant, this would be a line. And it’ll be the decision bound. I mean, that line will be like .5 and then you brother run. Well, one there, I’ll be. You can, you know, we can do the constructions we were talking before where you can look at the difference even when it’s that are coming here. So basically we put all this stuff Z and, you know, you have, I don’t know, you have K units from D one to Z K. Each, the kind of place the space into a line, because each, each unit, ZI, is a linear model. because it doesn’t have any vibes. The linear model GI is sitting a novel. It’s a linear version. So if I’m looking at one, GI equals to zero, because that’s when relative kind of switches behaviour, you’re gonna get a line. So each of these units is alive and it splits the space into into parts, each line, right? So now when I’m nearly combining these things, I get my piece sizing a function, by doing that trick that I would say, you know, you have all from Z2 to ZK, all of them are zero. It’s a Z one that’s active and that’s alive, then at some point G2 becomes active as well and you can change the slope and it keeps. so when I’m asking this, but I get is I get a precisely here comes. I need that moneyor. So that is the, the functions you can expect from a single layer of LPU. Now, if I go to a 2nd layer, so I have Yeah, so let’s say, I got age 2, equals W3, or Sigma, W2, Sigma, W1. my name for about. Thanks. Hop my my name. So this is just repeating the same thing over an idea. This idea is just sort of age one, I guess I called it, right? Because the thing come about. This is H, right? So, um, I mean, you said Chunk here about the state. Yeah, I get a bunch of H ones. I will leave exam with an index up word. So I get H one one, K one two. I get 10 of them, right? So I have ad minutes on my 2ndary yard. Each of those units is a different function that looks like each one. So each of these H ones here are a piece of a linear function by because they are just this guy, right? Does that make sense? Okay, so then what is the process that we’re doing here with W3? So, well, with W3 and the Red. So the real, what he does, it splits the space again whenever H one is negative versus H one is positive. So, but, but, uh, But the way it does that, it basically leads the space by a PC linear function. So, for example, this blue line here. I, my subnotation, so what I call a drama, the board, the college to hear. Sorry for that. But this blue line. This blue line here. is the besides linear function that corresponds to one unit on the 2nd layer, and it’s placed the space into 2 regions. The one on the left and the one on the right. So if we go back and try to answer the same question we’re answering before, it’s a less key to them. We have a very similar missionary. We can bring back we can do reduction, we can start by just assuming we have a signal unit on the 2nd area and then I keep adding them one by one. The only difference is that now instead of having lines, we have piece-wise lines. So what is how does that change? What are the properties that we’re expecting with with the theory before? Is, understanding correctly, we are spreading the only related space, so between a square number of these index resources, right? Like if I have been the best day, like 8 for 5 and the 2nd they have K to point and between 25 different tool. Your… Not Italian, but very correct. You are almost on point. that’s sort of what the project would be. So like just to some of the more fantanticic way, like just working into it, you know. Like what we care about in the past is was, in how many points and to life can intersect, then it was only one? Now, the question is, you know, how many points do piece-izing or functions can be designed? go. And the answer is many more. Like as many as you want. Like if you can, if you have the choice of defining the value you want, you get a lot more intersections. And whether the truth of it is the same, the time you intercept, control so many regions you get to add. So the trick now is you want to get this piece of cycling, a punctuous intersect with as many points as you can. And the more the intersect, the more I mean your pieces will get out of it. But at least number of intersections isn’t founded by the number of between units in the second day, in the first. Yeah. I know how you found that. Yes it is. So the question now, so this is the sort of the machinery that’s being used. So the question now, by knowing by tracing the number of units in the layer below, is like, can you build a construction that leads to a more intersection that you can get from a shallow model? So the, you know, like there is a lot of question about what is fine and what is up there. So like one simple approach to make things easy for us is to say what you want to control is the total number of units. I have the same number of units in the channel model. We’re just saying the keep. So if the shallow one has plenty even in it, the deeper will have ten and ten. And the question is, if I have 10, because otherwise I said, I did, one keeps hiding more time, because I’m not complaining. So if I have that and that, the question is, can I build a construction that 10 and 10, that leads to more injured that a longest 20 can have? And I know the number for the one is the last ever I can plug in instead of BM 20 and I get the p number. And now the question is, how do I construct something? What I’ve exploiting, that this, this, this engineer function, this is set in multiple places, to get more regions. I mean construct something close to what you said, like get sort of thisonentialial growth. But this also the machinery I’m trying to kind of get you guys to think about it. This is the construction. So this is a paper. So it looks a bit likely, but because we have the time, I want to try to get a trade because I think it’s actually pretty well. And then we can we can take a break after the construction, because probably people will be tired, with this kind of condition. But but I really like this kind of stuff, so hopefully I don’t know how many of you like geography and this kind of stuff? as many? Okay, so hopefully it is a really, really bit along, but I think it’s a really nice part of the, of, of this, because when this kind of exhaust started coming out, it really made a big splash, because people were very unsure what’s going on. I will doubt at the end why this action is non-informative. It’s an interesting construction. Everything is nice, but it turns out that the reason we’re using our networks is not because they have more linear pieces. It’s actually that’s not important at all. And that’s not the reason why I didn’t have perfected by the international one. But for a long time, people believe that’s the reason, and then that’s not what we talk about. Okay, so how do we do this? So, I’ll give you an example if it kind of becomes. Um, So, I will not use Remus because Remus are a bit too hard. So I’m going to use the absolute value. I’m going to have an activation function, you have to have value. So after that, you know, if remove the sign, and I argue any other mother can mimic the author value by using 2 levels. So if by decide and get the absolute. So it’s gonna be. So then what is the construction impact? So, Say I have the claim, so I have Exxon and I so, and I apply up to the value of the both of them. What am I doing is, you know, the, um, boundaries where you have zero, splits the plane into padrans, so you have 4 padrans. So when you apply to that, you’re actually folding the ps on top of each other. So the point is, if I have, uh, sorry, if I know, I mean, just read this and write this on the bottom. I mean it helps. If I know that the, um, on the 2nd layer, our college age to, is I get the values 37, I don’t know if in the input I had 37 or I had -37 or I had 3 -7 or I got -3 -7. All of these points get mapped to the same thing. I cannot, once I applied after value, I don’t know which one was it. And this is what I mean by folding. It means that any point, in any of the other 3 quadrons, get mapped to a particular point in the 1st class. And when I’m looking at this, I don’t know which one can, which, which, which one was. So then if I have this function that now leaves only in the 1st background, so I have this new function now. And on top of it, I put another linear projection and sort of a logistic regression. I tried to do the decision boundary. And I do a decision boundary that looks like this. So, the sun island chucks. If I take each of these points and I want to trace it back to the input space, this can be any of these 4 points. So therefore, basically when you trace it back, you end up copying this line symmetrically into this world further. I don’t know if that is potentially 9 So the whole construction here is you take the space, you hold it into a 1st programme, you use the biases to shift things back to the negative values. And then you pull it again, and you shift, and you pull it again, and then when you draw the cision boundary, the number of linear pieces is going to get multiplied by 4, you know, every time you go backward, and it’s going to become highly symmetrical in some kind of weird kind of ways. We’ll get a decision one way that it was very weird. Um, You can actually, you don’t have to, maybe they do, um, um, wait after the value, you can just start from 10 years. I am worried that this is becoming very technical, but you can spl it into many segments by holding segment wire and this is sort of the constion for that. Um, So, so the way, I mean, this is the actual value thing. The way you could work is your techie. So I mentioned independently, and then you start doing your sweets, and then you connect. So this is a very kind of structure. So this is a proof by construction, right? So what you’re doing here is where setting up the weights by hand in such a way to maximise the number of holes and the number of spreading, and it does exactly this thing that I was saying, right? It treats the space like this and you keep doing me like this. Uh, And sorry, I was looking for the number. No, I don’t have a number. Oh, because here. This is early numbered at the bathroom. This is the maximum number regions you get. So it has an exponent and there is a product to the number of layers. So I, this is for the paper, it’s not for the class, but you can, you know, you can buy, you can look at this now. I don’t say I but we can look at this and you can see that as you increase the number of layers this is becoming a condition larger than what the travel model can do. And the whole point of this theorem is to show that the upper bumper, the model, is a lot more linear regions. Another question? Yep. Can I group this from the previous theory that you mentioned? Yes, yes. I mean, this is exactly the proof. So the proof is that you are exactly our least. So the absolute value is a way of constructing this, bicizing your functions to intersecond multiple times. It’s not the only way. There’s actually people constructures. You know, everyone comes up with their own different attractions. I find the absolute value the most interesting way because it’s sort of highly it’s easy to visualise mathematically. But all you’re trying to do is to come up with a scheme of how to define the exercise in your function. So as they forcefully intercept in those many places that you can. And depending on the on the scheme that you find, you get different bounds. Some bounds are better than others. But this used to be a treat for a while. People were playing coming up with schemes of how to transcribe these things and give the number. And this is the more magic side of machinery. Um, So you get this number and the number is bigger. That is not the most interesting thing. The thing that I think, sir, this XY is… The thing that I think it’s interesting and maybe maybe I’ll leave you with this. Maybe this will help, is it offers you a way to understand what a neural network does. And it’s not exactly correct. I can go in why it’s not the exactly correct? But you can think of the layers of you thinking of the deep lighting is doing some kind of very Italian. You can think of the deep layers, which is folding the space on top of each other. So when you’re trying to decide a very complicated decision boundary, what you do is you find a way to follow the space. So that when I do a linear, because of the output layer, it’s always going to be a linear decision company, right? The output player is linear. So when I draw a line at the top layer, right, when I go backwards through these holes, deadline is converted into the highly shaped kind of decision boundary that you want. So this is how Mechanically the deep lining works, right? So like all the layers are folding in the space on top of each other in weird ways. So that when you draw a line, when you go through the folds and you unfold it and then repeat it sort of a mathematical way, you get the decision boundary that you want them. This is sort of just a really bad problem from by time. Does that make sense because that’s kind of interesting. yeah. So can we say that instead of learning just the classifier, where actually the deep neural network is actually learning a representation of the data that makes it easier for the classifier to split, right? Yeah, that is what’s happening and the way you land, the presentation is to this definitely process of, um, yeah, I mean, that’s why data keep. I mean, that’s the role of the layers. But in the end, yeah, it’s a different representation by that senior thing becomes a deep sort of. So you have a question or is similar? All right. So this f but we don’t want to far is the one that goes for construction I waslining and I’m not to do that, but it’s – it’s not that sort of important. So I had a question on, okay, I have a question on, maybe we’re going to take the big battery session. But the question is, yeah, I don’t know if anyone can answer this. It’s actually easier than you think. But a question of very okay. So this is supposed to be a feature of how the space is being spl as by sounding your own effort. Um, or point outside, so that’s usually important because we’re only talking about chasing your own efforts. So the question that I’m trying to ask is, if I go on any of these dimensions for infinity, so I’m going on this axis order for infinity. From some point onwards, how is the behaviour of being on a natural and on the life? there’s something that can be described. There’s something that can be easily set. Because anyone has implication. Can you repeat the question? Yeah, so I have I have annual that, right? There know about the c. right? But they acup one dimension case to make things easier. So I’m only by one of the dimensions of the input. And one time, you said the behaviour, one defined on the network on the line, it could be easier, so you don’t know what I miss. I’ this is the plane so that’s why I was still. So it’s in the final one, it’s on the line. What is the behaviour of that function as I go towards infinity? How is that function in another back? from some point one? Is there like is there any pathology that you think is going to happen to the new network? Yes, Iasticron that work’s going to be a straight there. Yeah. So they because the number of the regions is fine, right? It has to be that from some point homeward, you’re gonna stay on a linear piece.. So the way the way the space is going to be split is into many linear regions of finite volume, and with some regions on the edges that have even the volume, they set infinity. There’s no way around it how to have a fixed number of near users in an infinite space, except that some of these people need to have, I think, importantly, something. So the beh network has to be here at some point on. Oh, very normally, yeah, it’s just, but you do it from the previousization, not that this is a final number, so you should have, uh, it’s not that I see. But, like, assuming we got this answer, what does that mean in terms of expression? Like what does it mean? What can we not express? Or what will be impossible to express in whatever neural net? Any general outside is confusion. let me tell for moving system distribution and do know anything to have you can’t more anything of you. Yeah, yeah. Just to be a bit more specific, give an example, you cannot, you cannot model periodic functions. Because because you have to be liar from one point over You can’t necessarily come. So if you’re talking about a team that cannot be expressed. Like you an address cannot express their functions. another will never be able to press asideise. Like you can do it for any like interval if it’s in domain, that’s what you’re value for. But you would not be able to do new failures. No matter if my contractions have not been yet. They cannot be fair, right? This is one of those kind of dry results that, you know, people say, like, oh, you know, maybe develop myself. But it points towards, you know, there are some invitations. This is sort of not kind of mathematically move it like you want. And this feels like a very simple limitation, you know, some kind of poor thing that you can’t do any different. stuff. But yeah. Okay, let’s take a break for 10 minutes or something and then we’ll go back and. Probably moved to 5 time probably. Nice, daddy. No, no, don’t. Instagram I mean this is the thing about generally like when it comes from generalisation You’re like a structure. And you hard for you hard for part of that structure, but you don’t know how to distract one piece. You can use learn PDBS and as the reputation of the period you have a little bitment., exactly. Exactly. So there’s a way…. yeah So attention can be taught about the particular kind of that you buy. Whether… Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, is still the right induct device for anything. It’s like my… And that the price of the election is the correct one for… I’m not sure. But definitely like dimensions Yeah, yeah, yeah. So, so, I can get to this kind of assign me. So, what was these? Yeah. At some point, all these lines. Yeah, you can… Yeah, yeah, yeah. Yeah, yeah I mean, there are But yeah there are So Perance perform not still the high down but eacha they call it different they try to find it they completely different, but. LA work The reason why people are not joking on them is because they don’t necessarily work better. So it feels like there’s no point in changing sort of my flight that being as different. Exactly. So like the hardness for itself so this is like a something that a technology so far that hasn’t happened. And I believe that, actually, there’s a way of showing that there’s a room in the open, there, not a creation. Yeah, the reason why this is man, exactly, exactly. Because one of the reasons that I want to do is called an entire area, and most of our point. And inami, you can’t do that. So like, the technical point of least polards the ex math as well can just coll Yeah, yeah. but like in of so there’s actually a mask where people that doing I might support the people thing if it are getting or not. At this fact, it’s like an area is not just a point, it’s a big normal. So apparentlyly because that half this is not another econ. But as’s the only kind of health that. can also collapse point basically. Yeah when you mentioned he But otherwise is so right now. yeah. I’ll volume my infinite. Into the red quesimi, they more bad, outranges, infinite. So had a yogit, you know. I classify the cool linear locally. Who was that? A lot of… It doesn’t… You have to be sort of a little… So nice. Okay. Okay. right. So um Yeah, I think definitely to kind of go back over all these construction. So the film also, yeah, I know my art of another reason then I mention. I think it’s kind of questions very extremely important around the time it’s separately more. so it was the problem. And, uh, no, no, 13, sorry. OK. So maybe 14, fifteen, I don’t remember exactly. Um, and the reason for it is because, um, We started getting things like this relation, where I was going around, and maybe you tried. The whole idea was you can make a deep model and compress it into a shallow model. So there was a big, sort of, question in the air, whether the, the beat in the deep architectures was useful because it helped learning somehow. But it wasn’t needed because you could always do what you could do in the shower, you can do for it with a demodel. And I think these kind of constructions, I’m trying to argue your position, right? They’re trying to argue that if you fix the model, the size of the model, like in the limit, none of these questions make sense because in the limit that between these universal personal equipment. But anything can appx toimate anything if you the number of whether they shall or deep, whatever. But when you have real network that you actually use in practice, it’s a real question of whether, you know, Am I better off to make the model shuttle work? Or should I make it deeper? Or like, should I do Disney or 10 layer? So I think the, I remember participating a paper. So I’m actually a poker newspaper and I say that. Anyway, so like when we were presenting the papers, it was like a newspaper, I think, because it was funny, there was another tape I in parallel that was around insulation. I was trying to argue that it can take any remodel and converting into a shovel once. And it was sort of like one of those points when you were looking inside a poster, the one in that part of it. They’re like, newspapers are saying your opposite. Like, where is the truth if you’re talking to do this for the same time with this stuff? So it does, um, it was kind of like a, uh, like an interesting theory in time. But I think it is kind of a… Like it might feel technical and obviously the proof itself, the construction is probably not useful for you going forward. Like I, I struggle to find a point in time when someone would ask you to build a construction to show that this model can represent, so like that. But I think the intuitions behind it, or the machinery of how these things work, can be quite useful when it comes to questions of expressivity where it comes to understanding what are the limits of these systems. Like, for example, like this question that I in justice in terms of our friend, it kind of gives you one of doing how this works out. like, you know, this question of periodicity, right? That we are very, like, meaningful question to ask where you can say, you know, I have some, like, biological signal or some, I don’t know, some signal that I’m modelling. The signal itself is prioritic. And, um, you know, you’re trying to continue the transformer. Somehow it doesn’t generalise out of domain the way you want it. And there’s a fundamental reason for it. Like, you don’t even need to run the experiments to see that that happens. Like if you have the right information that you understand is the system, you’ll know beforehand that there’s no point of even trying to do that, right? Because if you tongue with that, like there is a lot of this kind of things that are happening, like there’s going to be a matter of popular example that we’re going to go over at some point. But it’s another soft match, where like anyone who looks at the formula of softmax, carefully enough, should know, maybe I can ask this question, but I think you guys know about softmax, exactly, speak about it sometimes in general, most of them. So, what happens if you apply soft match to an unfinishedly long factor of values? If I have a ve, I keep increasing his life, you know, keep added load this way. And you know, towards the limits then those infinity have add know. And I try to normalise the result max. What what happens? Oh, that’s amazing. Yeah, I’ll be distribution, of course, a normal situation. I will approach sorry, what? Normal distribution.. nice some of components will be used to see. So it’ll actually go to uniform, not normal one. You are right, but you are never behind it. So what happens is you this this, the – the weights are gonna become important. So your distribution is gonna, uh, personally, use uniform. Why is this important? It’s important it came about that people were discussing about LLF and increasing the size of the contract. You train another alarm on some context, we want to have a 100 times bigger complex. So what happens is you increase the contact? 3,000 the attention weights of bec become uniform and then the attention breaks down because with the attention right are uniform, then there’s no point of the attention there anymore. And the whole model breaks down. But it’s like one of those things that like, you know, I mean, maybe I’m making these sounds too, too chill. It’s not trivial. It’s like, there was a paper about it, so it’s, and that’s the new processor that I sent you. So it’s not like, you know, people laughing the terms of the girls around the paper. But it’s one of those things that it’s really down to just sort of looking at how the formula reacts as you change some variables. And in some nights what happens? Okay, so there some details about why this is happening. Sorry, I’m being very I was trying to make a po I being very, very fishy and probably. So the issue that is not just that you have a little number of budgets. That’s not the only issue that has for anything that needs to happen. working important because theologies are fine. They’re not allowed to have viol. And this is true for, you know, on that part, because in your efforts, we have finite weight. The way adverts work all the intermediate values in the your network are not not by necess. Because once we got minus infinity, you Viet non instead of form lot of bread stuff. The whole point is everything is fine, right? So because all the doges are fine, right? none of the probabilities are going to be zero. Because for a soft max to get a 0 probability on one of the entry, you need to have -580, that’s a logic. That’s sort of how it works in a normalisation area. So because of that, as you add more and more elements, you have a realty mask that needs to go to them. So as you distribute property masks, like the peak is going to become short and short and like everything has to go to uniform. There’s no way around it because, you know, you have an infinity number of pieces where you have to give the laptop probability, like you’ll end up everything being sort of unipoint. So that’s sort of the machinery that is going on there. So this is our sort of, I mean, you will revisit this much later when you talk about transformers and stuff like that. But I just sort of wanted to say that like, this kind of mathy things, it might look scary and boring, but I think there is a value to that. So that’s why I wanted to go through the whole yes. you should like before ingredients to college experiments to our inst. Yeah. So, I’m going, what kind is you about? All this on ideal magnet videos? Like, many very similar, but, I mean, the authentics are easy. Yeah, like, where, where, where, where, where, there, there, there, there, there? I Yeah, so it’s a very, like, during a very strange time. So I think I mean, these are changing systems who work surprisingly well for some things, and then they felt catastrophically for others. And I think understanding where the boundary is, is hard, but it probably was the most important aspect right now. So, you know, a lot of these agentic behaviours, the systems are not really operating out of the main. And maybe this is sort of the key component of this. So what happens, uh, with neural efforts at least, is that, um, in terms of expressivity and sort of their their ability to to do things, as long as they stay in the domain, where you have sufficient data, you tends to be fine. Like this whole periodicity thing or even the sort of soft maxim, both are scenarios where you push the system out of domain. So the question is the agent, the agent scenario is like a multi-agent scenario as well if you want. Maybe, at least in terms of formalism, I feel like a multi-agent intitu is a lot better defined than you tend to become type from, like, the term. Um The question is, when you have the sort of interactions between multiple agents, like would these drive any of the component out of distribution? And I think technically it can easily do that. In practice, at least how they’ve been used, it seems it’s not as big of a problem as we think, at least for the scenarios. I haven’t, I mean, I should know more about this stuff, but I mostly use that sort of importance scheme. Well, like generating food and stuff like this. So manage the right both and U I play a little bit with this. As far as I’ve seen it, where it’s okay, it’s like there wasn’t a new kind of game of. But it’s really about sort of the rules of interaction between the agents and whether this can end up having some kind of dynamics that appears out of the domain and pushes the system in a language that they don’t know. So I think that’s where the keys and it’s, yeah, like I don’t know that’s something that’s easy to figure out or not. But that’s how I see that whole life space of what people are doing. um It’s also for LLRs I mean, maybe what, you know, people, it’s hard to understand what’s up to domain and what’s in the domain. Like and then for even for very pragmatic reasons, like the data is always secret, but we don’t know what any of this models has been trained on or what was nothing on. And and all of this kind of element, it makes the whole thing even worse because it’s really hard to even start building sort of testing scenarios unless you know the secret source of how they revealed and no big company would ever tell me what they did in training. that’s sort of hard is. I don’t know what Gemini is trained on because I’m not part of the Gemini team and also within Gemini teams, like teams and teams and like at the end of the day, you know about this little piece you’re working on, but you have no idea of the whole picture and sort of where the meet is coming from and all of that. It’s it’s a bit of a like a big perform many. But anyway, I just wanted to have that kind of trying to be justify this. Um, I mean, I, uh, will probably have one of the homework serve around this kind of math games, probably more than one, so. So hopefully it’s not as scary. I mean, I don’t know. But um But going back to the point that I was telling you, there was Facebook papers to work on predicting each other, right? So there was this guys who they could compress keep models into shuttle on and then it does us are they theles and the 1 thing the shadow models head out. And the question is, how can this both be true? And the answer is, is the facility even a reason why we do this learning? Is it the exc? And we shall a model underperform because of lack of the rust. And obviously, I can’t give you, no, this is not a kind of answer, it can be sort of mathematical. answer to it, but you can give a veryical evidence towards this. And and the particularical evidence I have a paper about your vehicle events, but there’s going be sl later around it was never an issue of capacity. So the emprical evidence is, for example, there’s papers that would try to see if we can memorise image met, right? So you take image that that’s a 1000000 images, considered be it as a person. you know, you say yourself like that. And you replace all labels with random numbers. So now there’s no more any structure that you can use to learn a label. And you try to see whether you can feed that. And you can fit it perfectly with your. So usually people take this as a sign of like the model has enough capacity to memorise the entire set. It does not need to learn some kind of generalising function. But then when you kind that language that you still learn something in generalises, you don’t like to memorise, right? So, so, the, um, and then there’s other items like this where you start trying to see sort of how much people information you can, like, random use of information, control, you know, all of them. You’ll find that the number is actually surprisingly high. Even for show models and you can see this, I mean, it’s hard to tell, but you can kind of see this from the formula as well. Like if you try to plug in those portulas, some realistic numbers for the number of human minutes and and whatnot, because some really astronomical numbers are there. Obviously, it’s hard for you to kind of reason about like how many India regions did I need to represent CPR? I don’t know what they listen. But another marriage is like ridiculously large. Is it like numbers that make no sense, you know, uh, more than the number button, you mean, speak like that, I mean, I don’t know, hear things like that. So it’s not a capacity reason. There is something else, and this is all the speakers are meant to show in a very non, right? But there’s something that comes out of the deep lab, which is because of how the machinery works. So you have this sort of falling of the space and then you draw a line and then your line gets replicated, you know, all the regions that are falling on top of each other, you get a decision bother, you have highly symmetric. So what I was trying to do here is to say, I have this, and I draw some lines in it, and I unfolded, assuming that I folded sort of in a square version I was saying before, I get sort of ashamed and looks like this, which is very highly snatched up. So the intuition and this is even in the original paper that we did. We know this in the discussion at the end. The English is that it’s actually not the number of regions that matter. What matters in a deep architecture is that the slope of these inner regions, they’re like symmetrical. They tied to each other, and it comes from the structure of the layers. Yes. by the scope of the link. So if I’m doing a one B, I mean, you know, if I have, if this line goes like this, this one has to go like this, it’s because they’re like a replication of each other along. It’s so there is there is sort of this very strong symmetries between all of these linear regions. which comes from how you fold in the space. Obviously, it’s not going to be like this, this assumes a very like chess board kind of drawing of the space. It’s a nice square. That’s not what you Netflix like. But you get a very highly symmetrical region and But in addition that I think evolved over time, and this is not not that it’s being solved, because people are not looking at this anymore, as I used to look like 10 years ago in terms of papers. But any reason that we kind of emerged from all of this was that is this symmetry that matter? It’s not a number, but the issue is, we don’t know how to mathematically describe the symmetries. We know how to come. That’s easy. But how to say that like, oh, we had this and this structure, it’s something we don’t know how to we don’t have the mathematical appro story. and that’s why it feel kind of right?. I understand why it’s like like patterns in but I understand why is that I mean, why is this so much? Does it make complication more efficient for this? general. you look at So surprisingly, again, like hopefully I understand why. This is high dimension of data. It feels like the natural data we care about, the decision boundaries in natural data, actually, and whatnot, is highly symmetrical somehow as well. So this access is inactive bias. Basically, it is for you to learn particular style of decision model, which matches the ones that are found in real-day band. There was a point, I don’t know how truth there is anymore, but it’s definitely true if you try to look more at generalisation than anything else, where if you try to learn your networks, on synthetic pay pal that is overly generated, you find that you have a hard time to do that. Like it used to be, well, like people have kind of figured some tricks out and make things easier, but it used to be, I don’t have a feature of it. I- if you try to train image, uh, neural network, some very low dimensional data, two dimensional. Usually have a hard time writing them. And you know, not to extend to work better and better, the higher people ever started. Compared to other mentors, they’ll do better when the input is lower dimension. because, you know, in higher dimension is in fact you have the first of the nationality and all of these states between in and all that, ne letrics are almost the opposite. They prefer high dimensional data. The higher dimensional data the better, right? And that’s because probably, I don’t know why, so probably because we had a national data, there is some property of the decision bomb rate that they get explored. You know, like how, for years, as you increase the number of dimensions, all the volume goes on the surface. And there’s probably something like that happening with decision boundaries as well, right? We go inside a national, something happens in the decision boundaries that somehow the neural network, you know how to exploit. Like, I don’t I don’t have any theory about any papers, but there is a immediately absurd fact, a high dimensional, very, um, and for the wherever it is. And it’s also like, you know, efforts were very well on natural data. So this is why people think that there is, an inactive bias in the architecture, that we don’t know how to explain exactly the words, but that impactive bias is much more natural, natural, data, depending on networks. Um, and um, another, I think people already know about this stuff. We’ll talk about everything, map and filters, particularly to try to make this point, to say how people at the high level describe this in that bias. So, uh Do people know I say this maps? Some some of your, you know. So there is enough look like this. If you put the typers, um, there are time to interfere, while she is being made. So what do the sales maps are about? So it’s really a sensiv analysis. So really what you’re asking is, if I have this image, and I change this piece of here, would my classifier say something? And you just want to look at which are the pixels that are more sensitive to your prediction? And you colour the pixels according to their sensitivity, and you get something like this. And then you use this to say, well, it says it’s a laptop because it looks at an apple side and it says it’s a cat because it’s looking at the face and the pail or whatever. they lose it, right? Um, This is quite popular. I mean, the defined version of this, like, produced in explaining ability, with all kinds of tricks and whatnot. But as a co-hort, it is a sensitive analysis matter, but that’s what it is. And Um looking at the back of information of the sc the way it works. So. Okay. So it’s really like the way I think of it, is literally you take a function and you do a data expansion. I mean, that’s sort of what it is in my car, right? So you say that my function plus the perturbation, particular perturbation is very small, is approximately the same or my function, plus the noise, whatever. But it, uh, the reality for that. So if you want to compute this mask, you just compute a variability, mobile to respect the input and that’s what it sends in. And you re normalalize the lab that it new things than like. So the drink and magnitude leads the manager of importance. Is there, like, any sense in which this is wrong? Like, what are the people so just relying on radius and trying to make a kind of decision? Do you have anyone has an intuition on why this has to have articulary route? Everything comes from, right? It’s scary about you. So, yeah, so let me rephrase that. I think I think this is what you mean. But, um, So 1st of all, here you’re dropping some turns, and then and then usually you grab those turns because your CMRs are in very small. But, you know, you drop in higher order times. But depending on on those I order terms, um, even if App Store is very small, like they might still be 5 meaningful. You have very high order, um, very high norm, higher order, tires, like curvature or, like, or, or, or so forth. There is a way in which this is sort of kind of drastic in our sense. So, um, that thing about a see point, I guess, or you guys know, policy point is, right? So, um, then you have this very, very simple problem function where you have a cook in it that whole signite. And one of the hidden unit is, like, there’s some like teacher engineering happening under the hood, and one of the healing unit is one, it was the word awesome, in whatever types of mule, right? But it is saturated. So like the weights that you’re set by hand or whoever learned the system, learn a weight of a 100 or whatnot. So the word asum is there, that unit goes to a very saturated state of life. Now, your decision might still be based on that unit, and it will depend on the fact that the word is, the word awesome isn’t there. Well, when you computer silency maps, because the, the, um, the CY is saturated, there’s not going to be any greater point through it. So if you write individual to CY, you’ll see that the greater will vanish. re vanish in the saturation. So when you’re gonna do this kind of colouring of the different words in the view, the word awesome is not gonna be read. It’s not going to be lighted out because the model assume that you do not depend on that word. And that’s just because the documentations comes in the site. So, but it differently, some things can act as a bias in your decision. In a way that they don’t honestly have a pregnant. So, like changing them a little bit might not sort of change or tradition because, you know, there is no such idea. But they are important for your decision. Like the fact that they are part of the reason to make decision doesn’t necessarily need this kind of possibility. So yeah, I don’t know if that example is here to anyone. But there is a sense, you know, yeah, it’s like.. Right, so I’m basically trying to argue that a model can rely on a future strongly in terms of making the prediction. So, okay, so just to give you a example. So you have the signal that is saturated. So because the C might is saturated, the needs of rate in close rate. So the gradient, uh, the tendency map would be 0, but they’re a teacher. But the prediction is being used by the model. Um, so so the the output of the model is really, I guess, it’s easier if I give you an example. So, um, Like the model, the model, the model could only depend on that particular feature. If you if you want. uh you can h- how do I this is the for example that I can give you some values for the weights so you can see how it makes sense. Maybe I can make that as a as a point and and send them to tell everyone later today. But you can imagine, yeah, I don’t know, let me try. let’s go a single feedated unit, right? So you have H equals C modes of 10 times future, whatever the teacher is. And then output is sigmoid of it, say, or like, I don’t know, one or one time it, right? So, and this feature V 0 or one, a feature is 0 with if there is no word, no awesome In text, and one, if, right? So now you have a text that says, ask some, some, right? So, because the text says the word, this is a word. This will be 10 same moment of time will be won, but in a very side rate is speed. So now if I’m trying to compute the silency maps, I have my output, I’m trying to compute the great end of my output, which you said, this is positive. They don’t, uh… So the output is part 10.5. It’s going to be point of onely close to us, right? I can make this moment, it can be like, for example, it doesn’t matter. So the output is positive. You try to compute a gradient through this stuff, but later here will flow to age. But then the gradient, so, paid by the radius of the output, with respect to age, then one of these 3 is greater than 0 because I can do my weight such that is – is nice actually. It’s – But the problem is that the million of age may respect to X, so in my case, in respect to that, the magnitude of this guy is zero. Because my heat unit is extremely saturated. And this is just a property of the signal. So, the computer variance through the CIA, the – the – the more confident the CIA is, the lower the grade yard. Because the, the, the gradient of the signal 8 is one minus 3 minus square, I think. So something like that. So, if you have, if you have that your, your, your seat more, your pen is gonna be basically one, then one minus one is gonna be, uh, zero. So this grad and here is gonna be zero. So when I applied the chain rule, the computer grader of the outputs, you describe to F to see if that was a factor in my decision. I will get that the stailency map is zero. But that’s the only reason you’ve made the decision, right? If you can look at your network, it’s a very simple network. Like if you remove that app, if you make up 0, it’s not going to work. But this is basically, I’m basically saying, okay, let me differently. Maybe this will be. Maybe this is the way I should have said it. Because of saturation states and all this other stuff, the um, the function that you’re modelling can be discontinuous at points. Or act almost like discret. And at those kind of points, the gradients don’t matter anymore. They don’t make sense. And this is kind of what’s happening here. I’m forcing this thing in a saturating state to almost be sort of, uh, th- So, so I kind of break the continuous behaviour of it. So, so I think it’s, I think it’s the same thing because drafting is actual opportunity history. It’s the same argument as having very high, known, higher wider charge that would be equivalent to that. So in the extreme, you know, that has very, but it’s just a lot of like a, I wanted to give this as a more like chromatic thing that actually can happen. So you can have these kind of systems that have like similar, they can have other kind of saturating activation function, right? Soft Max is a good example, potential ASL this issue, right? When you’re attending a particular topic very confidently, so your attention when you don’t lost all of that one, then the radiants don’t go with the attention. So then if you’re asking by using a sentency map, while attending this case, you will not be able to tell. You have to look at the attention where you can look at the derivative detention, right? That is the difference. So that like if you look at the attention w, you can see, okay, the weight was one on this particular top. So yeah, that’s the one I was attending. But if you look at the liberty of the output with respect to the top, you’ll get ready zero. So you can be secondly say, I wasn’t attending it. So this is kind of the failure. Is that clear or does it make any sense though? With me but I can understand it in context of more time age or whatever. But for example, I know it’s very question because if it is radio, it could work perfectly even go un sec any? Um For example, I mean, you can have, I’m trying on my mind to see how to build an exam after this. But forever, you can have a similar situation where the decision relies on the unit being zero. And that’s sort of the saturating state and kind of have the same issue. But you have to work with the bias and stuff like that to make that work. But I’m pretty sure you can make size that the model actually looks at a hidden layer and makes a decision based on the fact that nothing is positive, that everything is zero, somehow, by multiplying negative numbers and doing something with a bias and things like that. And when you have a situation like that, again, like if you look at the videos, because there’s zero, there’s nothing flowing, so you say, I do not depend on noise below, but actually that is the only thing you’re depending on. This is a problem with that not only near level. Yeah, it’s because of the noninea because of the infection of non-in and so forth. Obviously, you can’t use it around united to the models on the word. But, uh, it’s, um, I think it’s just sort of, I mean, I’m making a bigger thing than it needs to be out of this. I’m just kind of pointing it out because I know in the interfretability world, people do silency maps left and right. And um, I mean, there’s other places where people would think like this and because you get used to them so much, you start interpreting them with ground truth. So you assume that it can never go wrong. But then sometimes they can go wrong for simple reasons like this. And, you know, it’s almost like one of those instances where something goes wrong and you can’t really figure it out because there is something simple going on behind. So that’s why I’m trying to bring this up because I think in general, like you, you should always have sort of like, You should always be suspicious about anything when it comes to new America. Like any kind of story, any kind of theory, whatever, there’s always an angle where like the assumptions unfold or something breaks down. And this is just for sentency mark, this is sort of one example that I think can be pathological, and I think there are, I don’t have any papers I think here, but I’m pretty sure can find papers that kind of can exploit this and show how you can make sort of the wrong decision, uh, for the picture. There is a less popular nowadays, but a different, I made up this time. I don’t know if they make any sense. So I was calling that the other one backward propagation view of whataintance. This is kind of like a forward propagation view, which is how people start it actually when you are doing this. And the way it’s exactly how we build or who I trying to build that example of the experi just regression. The way people are thinking about things, right? you think of eaching and it that some kind of feature defector. that’s not the usual perspective that you dont have. Additionally, in the IBD, 2009, when I was doing my DB, we weren’t doing a SMS, where we’re having this perspective that every unit has a semantics. It’s a teacher detector. It takes some particular features, and the way she works in the computer, some kind of simularity, and then some pattern, which is probably W, and X. And, you know, the higher the distance, there are more access to much that pattern, the lower, the less similar. And then the activation function is some kind of normalisation, and it’s on a distigulation, or if it’s revel with some kind of thresholding, but anyway, it’s basically, but it preserves the dynamic of the distance. is good, small is bad, you know. And units fire when it’s 90 is maximised. So it’s kind of vacation. And then if you have multiple of these things stacked together, you can kind of use sort of a reconstruction kind of pipeline to try to figure out what it means in the input space. If youre Basically inverting, even though these things are not be veryible, you’re kind of assuming some kind of inverse of the previous things to figure out what is the field fact that the in is going to. And, um, I’m just bringing this up because this is another kind of observation. Maybe you’ve seen this maybe that kind of other the fashion, but this is like visualising the filters in a content right the manual network. So what you do here is you visualise the weights rather than than than the noise. And then when you go to deeper layers, you basically multiply the weights together and make some assumptions about the activation functions, which I didn’t collect. And, uh, these are used, again, what similar user standards and what the model is doing. Maybe this is the most difficult thing, the layer one, this is the most reliable thing as well. And here you can say like, oh, this looks like a board field 1st, right? It’s like a filter for the colour green. so I have this unit responsibility the colour green in the image. I have this, this, uh, goodness that it found, there is a lie like this. I had the swimming that responded, it’s a line like that. And then I keep composing them and get sort of these shape of like this phone too weird to wheels or this kind of weird pattern and then soft white. Um, Okay, so this was a long story, a long way about, but what I really wanted to say was about this impact device. So we both of this kind of isization schemes. People have started trying to look at what the different changes in other deep model. And the usual feature, so this is a really early stuff in 2013. And obviously there is a bias here as well. Like people knew what they wanted to see. So when they were doing this paper, they knew exactly what they needed they wanted to get out of it. But what they see is that the more in the 1st layer, they have like part of objects like eye, the noses, and then part of faces and so forth. Um, and then there’s this kind of, uh, vision of words, so this is word from Yoshua, where what they do is they take 2 images, like the 9 and 3 that are being classified as 9 and 3, and they try to interpolate in input space and in the. And stay away from the screen. Then in the in the area, in the space, or perfume, layer, and in the space of the 2nd human line and so forth. And the argument is the role of that, um, in your networks, even directly biased. And what it does is we have giratical representations. So your article is the standard neuroscience called side kind of training, you know, 1st layer, double clutter, the part about it and more complex object which was direct tired from there. So this – this is the mission that this moulding of the space that we described, is just basically makes the space ammantical. So any direction that you move is connected to some kind of semantical change of your data. And that’s kind of the intrusion. And then there is this own pressure on obstruction and anarchy. So maybe I even have a sl on this. Because then, like, not as much machine learning, but it’s more, like, general philosophy. But there is, if you look at the basis of people learning. So I’m thinking of like review papers from the IV 2010 and so forth from the film just starting. Um, There was this strong assumption that is coming from coxide, that you cannot do reasoning, or you cannot do any kind of, like, interesting processing of the data unless you have attraction. So this is sort of what the core of how people think that the biological system is doing as well, where you play the rose and relate that in beyond more and more abstract representations of it. And that is what allows you to pre the daytime interesting thing. So there is number of abstractions, there’s special obstructions, there’s sort of ironical structure that is being built through this. And if you look at the early theses of what we learn should be, these things were very great, right? They were saying that deep system, you know, for something to be called in lightning, it needs to be deep because it needs to have your actual representations that build on the representation to know it and they need to have this sort of abstraction being built up of more and more interesting things will go up. And this has been used for many years. I was back nowadays, if you ask people, they would argue, even today, that’s why the network works, because there’s sort of abstractions being happening as it goes to the model. To make excited is happening. but these are the news as a big reason for this. I’m going to mention here I’m not going to mention it later as well. There are reasons why I don’t know if this is a true activity. So I’m just going to throw it out there, but I will go back to this later. So there’s a few things that have changed through the years that happens in modern architectures. They kind of break this perspective. I don’t know if anyone has any. The most basic thing, that is crucial for any modern architecture, is key connections. and we’re gonna go as keep connections are about, but keep connection basically allow you to sleep. Instead of going from the answer, you can open the inpput directly to different legets. So because of the sleep connections, the representations are not Earlier, I’m sure what they used to be, because you can, the data flow is differently now. The data doesn’t flow sort of only one way. The other common example for this, but that’s a lot different, and I will get to that, like, it all, is, um, if you think about it in the model. If future model is not clear that they build obtrractions anymore. They basically go through a noisy version of the data and somehow from noise to more noise, more noise, suddenly something comes out of it. But it doesn’t look like there is any like… I mean, this is a bit hand lady and this is on an all right. But I just wanted to put it out there in case anyone is kind of interesting on this perspective. I think, like, this was the basic theme of representation mining, and iClear, which would become friendly. International conference and representation line, and is provided by Yosha, and American and others. And it was really because of this kind of thinking. It’s not clear to me, leave the representations we’re learning nowadays. It’s the same with the representations we are learning 10 years ago, and whether some of the things that they put in these things is about what a good representation is, it’s actually valid. On the box side side, if you look in the literature, it’s not that established either. The majority of people will say that technicians cannot exist without obtrractions and probablyactateations. But you have books and papers that argue that’s not true. And I think the octopus is usually given as an example that might not have sort of the right kind of actual abstraction in the nose for the least in intelligence and whatnot. But what security is stipulated? So anyway, it’s kind of like an interesting quality philosophical thing. and I realising we’re gonna run this amount of time. The other thing, I mean, this was something I mentioned today before, that I think I wanted to say, with the kind of semantical assignments of units. One has to be careful, and the usual rate, you can show that there is a problem with them, is by going to other serial examples. This is sort of one certain examples are. You can take a picture of a panda, mother is somewhat confident it’s a panda. What’s some noise that uses this picture, the look on how to say, and the model classified is something different. And these are someone’s modern examples where you take these scandal and now it says it the bathtub, that throw is a bird and then 4 is a 6, right? And these are, like, the way you fool the marvel is this very more perturbacious that you can’t really see. So you can potentially go back up to other examples. Um, And, and, and, and, and then start them, but, um, I don’t know if you fully understand adversarial examples. mathematically all the world. There’s been a easy where the other example is there because the system is still linear. This is a map, but you can enjoy the map. kind of intition is we need to have a very high dimensional infrosp price. and you can these your noise over all the events dimensions. Because the way the model works, it goes from a very high dimensional thing to low-dimensional, like 10 classes or whatnot, very low dimensional output, then you can potentially distribute that noise size that it makes a big difference. Once it is concentrated into that goes much smaller dimensional case. But you can distribute it over that, I don’t know, 10,000 or 100,000 dimensions that you have in an input, in a way that is not visible to be out. So and and the way this works out is because the system is pre here. So like whatever small noise you distribute over your input, somehow gets carried on through the layers of the model of the otherput. So it will… Might go back to this later on once we establish a lot more of the rest about neural networks, but, um, This is not the only reason you have a personal examples. There are papers out there that plly crack up, which is they come because the system is strongly available than engine here. Um, But, but, but yeah, this is, um, in some sense, this is one of the established way of thinking of that. But like the reason I brought them lot now was But using adversarial examples, um, okay, so there are 2 ways, you have to set examples, are sort of a discussion, it has so far. One is, by using my personal examples, you can show that units are meant to respond to some semantic or pain, like say, uh, this is a unit that takes an eye. You can make that unit respond to something like very different. I mean, that’s just a simple attack, right? You take the output of the unit of the classifiers, they’re classifying the real arrival, not an eye. And now you have an image that’s not of an eye, and now you’re using your unneserial attack by that. We haven’t started this. Well, my find of how you find this, of how you find this noise by – by either doing radio descent or something else. But you can, you can use your attack, you change the input in such a way that it’s still not an eye, but the model detected an eye. And the truth, I mean, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the works is that there’s always… Basically I’m trying to make the activations of the model like I’m assuming the whole thing discrete, I think as long as it’s continuous, slowly, people protect it. Maybe the fact factor norm is to be higher or some changes like this, but it will always be able to attract it. And, and while this is important is because, um, usually I want to use this as a wording of like this kind of people assigning semantics to units and kind of being very drawn to that. And then just like reframing the whole model in terms of, oh, I have this eye detector, this noise detector, and this multitector, and, you know, once I have these little things affected, the next layer will say this is the face and it’s done, and this is how it works, and this is my system. That’s not really how this system works. Those are not, I think there’s another normal detector and it’s not about completing them, right? And the other attack that is here is this interesting paper that they were trying to do, a different kind of attack, where they take around it, like is lava. And they’re trying to leave them out of the classify the images llama, but they want to change as to what the model is looking when it makes the classification. So the normal initially is looking at the face of the llama and classifies the llama because of this. And after the attack, the model is looking at this region of the infrospace, where there is no llama, it’s just background, and it detects the llama because of that. In here, I think this is the butterfly and I started looking at this corner of the image. So again, these are the chats that just say that just because a model makes a new difference based on something that you can interpret in one case, it doesn’t mean that for another similar input, it’s gonna make a similar kind of judgement. So this is basically just a worry about generalising how models work. I think in terms of the ability, for a given input and to give an output, you can do all this kind of stuff, security, and obviously, the curricular things and whatnot, and come up with maybe an explanation why for this particular input, there is difficulties or output. But generalising from that to saying that it will do the same thing for other similar inputs and outputs is wrong. Like they can be like this images. you can’t tell them apart, right? Then look exactly the same. The phys is the same continent, so the reasoning is completely different. uh the same for high Islamas again. But like, you know, so, so, so yeah, so it just meant to kind of, again, this is just meant to kind of give you that sense that, um, there’s a lot of people kind of say about me on that, right? So you shouldn’t trust that. kind of sure. Yeah. All right, look, maybe for the mission, is it possible? is it the you know the original made identified? Yeah, yeah, so this is the original. I’m a good regime, right, from the 45 year. Time did it, maybe, about. you can go backwards then you attack it backwards. Yeah, I think so. So I think, I mean, I don’t know, I don’t know exactly how that works. I think it’s a great and-based kind of thing, but I think you can always attractively change it any way you want. Um, And then usually these attacks work better, the higher dimension of the data is in the in the data. So, what? Well, for the New York argument, is like you have a super dimensionality that is much, much higher. It can distribute your noise over those dimensions in much smaller quantities and still kind of come together to form an attack vector that is large enough automatically. So it’s, uh, so if you’re in a very low dimensional space, it’s very hard, much harder to make attacks whether humans cannot observe by art. ones will start kind attack where like if I’m rout as a person and just so you, I would say they’re the same,. Like I can’t even tell what the noise is. Obviously, I mean, for breaking the semantics of a unit, it doesn’t need to be that you cannot notice the attack. I mean, there are most maybe scary thing about the neural network. I don’t have a flight on it. You can give an input that looks like this and the model can be confident that that’s a fan about or something, right? So the the new network would be very confident on images that are just Gaussian runs. Um, so that – that is – this is, you know, so that – that’s kind of telling you, like, okay, there’s no santis for that. It confused just on, like, random pixels, with the object that he was supposed to semantically detect, that that’s the media work. Uh, we need to think that That’s not what he’s doing. That’s sad. Like, obviously, um, In some sense, we’re sort of in that weird space where you can’t really reason about improve or research or play with architectures unless you make a simplified model of them in your mind that allows you to make hypothesis and allows you to do things. So I’m not saying you shouldn’t assign semantics, it’s a components of a UNI question. Assume that they work. I’m basically saying that as a researcher, you should do that in order to be able to come up with hypothesis, the experiments, and proposals, for me, I take it and so forth. But you should also be aware that all of them are wrong. And you’re inside of pressing the fine line of like knowing that your assumptions will never be true, but at the same time, there’s nothing else you can do, but assume that they’re true and work with them to come up with something new. I think the danger is when people become used enough of the assumption that they’re making, that they don’t realise they can be wrong. And then, you know, they hear sort of the talk to that it’s beyond what it should be. That’s basically what I’m trying to learn here. more or less. Oh, this place is not that easy. I’m gonna put it out there anyway. just not near. which one is this on? Okay. Yeah, it’s a bit big. So the question is, can you have? okay, so let ignore the details this is going to be a very hard problem. Let’s make it the people of more generic. So assuming there is some kind of control over the model side, the question basically is asking whether a shallow model can represent whether a new model can. And about the diverse, and the new model represents anything that each other models has. So are they sort of, this is one of the soccer class or the other. or like how do they compare to each other? In terms of the set of functions they really like kind of goes back to the media regions. There’s no ther. When there is a limitation. It’s that they, they’re controlled for for a number of weights to be sort of, because otherwise it becomes the right question. But otherwise it’ll be simple. But I just thought of high level, like, let’s ignoring how you control for the weight and like, if anyone has an intuition, if not, we can pick it up next time. I can, I can wait, a more deter. I’t have any thought. or like any guesses. What a lady, can A represent whatever we can represent? Can we do we have infinite neurons? No, no, no, there’s shakes. So in this case, the number of neurons sounds are the same name for this game. So like I guess, get A these. I maybe radio there’s no wrong answer. I mean, there’s a wrong answer, but I So, so, so then I, I, I believe I, believe that I cannot realise what we can. So that is the correct answer. Yeah, that. And that comes because we showed that a new model can have a commercial modin area. So because of that ANR represented, right? So, the diplomata have you not placed? But is the rever true is not represent any function that any can represent. Yes. that. It’s natural. And it’s not true because The way it gets more in a region is through those symmetries that are tied. Well, here there is no symmetries between these things. And if you ignore, if you want to check the number, maybe this is an interesting question. I say it hasn’t thought for myself, but it’ll be an interesting decent question. If you want to count the number of unrestricted linear regions that a D model can do, it’s gonna be weight smaller than the Shadow model. because basically you can mostly do that in the 1st layer. and the less layer the space is don’t do much. Or something like that. Sorry? Sometimes you need to leave your mode. Yeah, yeah, yeah, yeah. So you basically, the way you do this, so that you don’t introduce symmetries, you basically make this units to always be positive, so they kind of disappear. They all become linear. So then you end up with a shallow model that has fewer human units than the other one. So therefore you can’t refuse that. So I think this is kind of the reason I brought this out. It’s kind of interesting because people usually have this mental features that you want to sol it better as the shallows. So, um, you know, demod can do whatever shuttle on can do. But it’s not actually true. They’re just, you know, if you control for size, in the limit, they all do the same thing, right? You know, for size, there’s things a shadow model can do that a 2 point cannot. And I think that’s kind of interesting observation But yeah. They also can do more can view that. Exactly. They’re not overlapping. they’re different. I mean, they have some intersections, but they also have things that can that they can they want that. So, so I think that is kind of an interesting observation that, uh, um, I think it’s kind of useful to have the, I don’t know, for me, it feels like the food happens in the, in the model, never mind. Well, you’re making decisions about these things. with that I’m gonna stop today. Exactly on time.

LECTURE 3:

Picture itself, but also the optimizer. Will reshape the the search space in which you're looking for a solution. So, let me try to make this a bit more. Um, expresses slightly differently to make a bit more sense. Um, I mean, I'm gonna say something that's not true, but just to give you like an idea.

So, for example, the models like Spire Solutions, uh, just because of the structure. I mean, it's not true, but just use spicy as an example, right? So, the idea the inductive bias would be more if I train a D model? I usually get a spar solution if I dance if I train a shallow one.

I'll get a dense solution, so this is the inductive bias. This is the distance between them. In, in what is the exactly bias of the model is? You will get solutions. Okay if you believe the whole cognitive science stuff that I was presenting at some point, which is partially true, but not completely true.

It is a deep battle with fine solutions that are more compositional because they have this theoretical structure. So, the way they learn say to classify your cat is a first line Gabor filters. Then they learn sort of composing those four filters into part of objects, and then keep going so far, so it will only converge to Solutions.

That respect this kind of compositional structure. The solution itself decomposes things, while a shallow model doesn't do that. The the shallow model will try to find a template of a cat on the first layer and just match your image to that template because it does not have the depth to compose things.

There's no decomposition going in there. So, so the kind of solution you find are fundamentally different, like they might give you the same numbers, but they're different and ones that the D model finds because they rely on this compositional structure, they generalize better. So when you get a new picture, it works a bit better.

That is kind of the the argument and this sort of compositional structure of the solution. This is, you could argue this is the inductive bias of of a deep architecture compared to a shallow one.

I'll come close that to make sure right here. Oh my gosh. Uh, so usually, right? That's a good question, so I mean, okay. So, this election is a big field, and people have tried all kinds of things. The standard standard things that people do is they put Kyle. Between the output of the teacher and the student, so it looks at the whole like, no, just basically right.

Um. It matches the distributions like the teacher outputs a distribution over the classes, and you match the distribution in the student that is sort of the standard. Um, distillation process, but you can do all kinds of things. I mean, there's all kinds of variants you can play with the temperature of that distribution.

You can do lots of tricks, some help, some don't. Um. But typically, you are not. You know you're not using the teacher to classify an image, and then you use that as a label for the student, because then it's? Well, you're still. It's still not the same as learning on the data, because at least you make the same mistakes that the the teacher does.

Which is actually quite informative? Um, because? Why is it informative? Okay. Even that will help you and I can explain it a little bit in the sense that neural networks they have a tendency to learn smooth functions. So they don't like fast transitions. It's just sort of by the construction of the neural network by the regularization that we typically use.

We converge to, like, low Norm Solutions. It's a low magnitude, and therefore they like functions that change very slowly. Uh, so there, so if you have very nearby images that have completely different labels, this would be high frequency, right? This would be this will look weird. Um, a teacher would not learn that with mislabel.

One of the two images? And that would actually make the life easier for the student. Because because? In some sense. The teacher is even more constrained than the student because of the article structure and and and the student might have the ability to easier learning loud noise, like overfeed to the noise.

But then, I mean, this might not be noise. This might actually be the the actual true function, but in some sense, is like when you use the teacher, you know, the labels coming from the teacher? Those labels make more sense for a smooth function, so it makes it sort of life easier for the students.

So, you still gain something, but that's not why people do. In practice, people really minimize the Kyle between the distribution, the output distribution of the teacher and the student. And then you have the launches that you have all the other information in there. Yeah, does it mean if I have human entities and this human entities can make soft labels for the data set.

Yeah, like, for example, the famous image of cookies and dogs. Yeah, can I make it intentionally more confusing to get this soft flavors? Yeah, so you can definitely. And there are techniques to do this, uh, that help. Not as much as distillation, but, and then, some of them are super silly.

So, one thing you can do is. Instead of having a one pot that says this is the class you put like 0.9 for the class and then just uniform for the rest, and this already helps in learning. Um, you can do more fancier stuff. Like, if you, for example, for image net, you know that there is a hierarchy of the classes of the labels.

There is a structure so you can use that to decide how to put uncertainty, right? You can say everything that's within the same kind of thing. They, they get a lot more, and the rest is a lot zero, so you can do this kind of tricks. They help a little bit.

I don't have them in the slides, but there are tricks that people do Envision, um.

It's not solving all problems, but it does help, you know, people were using them quite a bit. I don't think that you used as much anymore. So right now, people use that data augmentation that helps a lot more. Um, and, and this just this label smoothing feels like extra work that people don't do it.

Well, that's that's pretty good, um? I had some questions as well. I'll try to go a little bit through them because I wasn't sure whether people have questions might not be more technical so, but let's see if we can. So I, I, yeah, I just wanna like, I think I've said this a few times, but like the first question that I had, uh, for people was?

In deep learning. Why is this emphasis over time? Why is the? You know what? Why? Why do people emphasize the sort of deep architectures like the multiple layers versus shallow? I think we just talked about it. Does anyone want to summarize basically? Interactive bit, introduce an inductive bias. Yeah, exactly because that adds this inductive bias, and it's sort of a very empirical things.

Like, if you train deep networks like the well, it's there's a point where it doesn't help anymore, but actually making the initially making the model deeper helps you in terms of performance at some point. If it's too deep, it doesn't. Really help anymore. And um. We're going to talk maybe today about this, but like the the issue is.

As you make the model deeper, you make optimization harder. So, at some point, if you make it too deep, you just can't lie in hand and nothing works anymore. The second one started to see to make sure that everyone understand what the universal function approximator is, so we know that neural networks are deep.

Networks are Universal function approximators, so I wrote here a little piece of code. Can a neural network? Learn this to imitate this code. Exactly, yes. Yes, so, so they're just approximating functions. This is not a function because it has an internal state, so every time you call it like the output is different, even if you call it the same input.

So, so there are architectures that can do this. Networks and so forth. And this kind of moves towards turing machine, and this kind of. Yeah, sorry, yeah, for RNN. And I don't know how specific number of steps right? Uh. So RNs in. In principle, you can run them for an Infinite number of steps, so you can think of them as a dynamical system.

In practice, we usually have a fixed length, which is given by the sequence line. Was that a question or something else? Yes, that's my question. I can't get it like. This function can run theoretically for infinite number of steps. Yes, but RNs can can reach this because of the training limitation, so I can't approximately.

No, the influence the one stops you to to train them. Oh, okay, okay. So, so your question is, um? In order to learn this function, you need to have an infinite sequence Center to see that to make sure. Yes, that is true, like whether, um? I mean, okay. So, for this particular example, I would.

My path would be that if you give sequences of, say, length 10, oh yeah, let's already figure out what the underlying function is, because it's kind of simple and the inductive bias of the RNN, which kind of looks like this. So, you just have some weights to these two things.

When making that, so this is very aligned to an RNN, so it will discover. But in general, if you have any kind of dynamical system. Um, unless You observe the behavior of dynamical system of over an infinite long sequence, there is always some uncertainty like the dynamic system can always do well.

If step is larger than three trillions, then do something completely different and then the other, and then obviously never observes that. So it doesn't know that it has to have this switching Behavior at lecture. Okay. So? This is another question that someone asked me yesterday, so I said yesterday when we were talking about linear regions that this is a bit like origami, but it's not exactly origami.

Do you know, yeah? Why is that the case? Yes, is it nicer to me? Yeah, because origami doesn't collapse it to afflict set point rather than just Paul. Yes, so you know that I said this pathology that they can collapse an entire part of the space, the signal point when they fall, and this is bad.

Uh, the second question is, what can go wrong with the salesimaps? I try to explain this yesterday. I don't know if I've managed, but I want to know if anyone can. See if people understood why, what was like how, how saliency Maps could go wrong? Uh, or is kind of sensitivity analysis gradient based.

Yes.

In front.

Yes, so it is. So, the issue is that, yeah, the sincere analysis is just a first order approximation. So, somehow, the other because of the other higher order terms, this might be misleading, so you could have. Like I can do, I can draw different people than the ones I was drawing yesterday.

Can you imagine? Can imagine a function that looks. A bit like this. Right, then you are here. So you if you compute the sensitivity with respect to this X because here it looks quite flat. You'd say, okay, it doesn't matter. X. It doesn't matter, I can move as much as I want.

The function is not going to change, but then you know as soon as you move a little bit. Things start changing, right? And this is because there's like, third, order fourth order, what? Not so, the curvature changes immediately as you start moving like, you can make it even worse.

You can make it like a b. So, in that scenario, in a scenario like this? You would believe that. Okay, it does not depend on X, but it actually depends quite heavily on X when you start perturbing the model. Um, so another trick that people do sometimes when they do this kind of analysis to to try to understand.

Uh, or to interpret the decision by the model is, you can use perturbations rather than than doing another instead of computing gradients. The gradient is the analytical form. And it breaks down in the scenarios like this. If instead, I look at. F of. And f of x plus Epsilon.

Where I sample a bunch of epsilons? This is a bit more robust. You know, this is like, you know, you will see, for example, here, because what this does is basically approximates this this function with something a bit smoother. And, and therefore, you're gonna capture some of the the change.

So, so there's like, one trick that you can do if you want to be more confident in your predictions, um of whether that variable is important or not? Um. Yeah, I think we talked about yesterday. Can we just delete this? This is my last question is, we talked about visualizing filters.

Um. I didn't talk that much about this. I, I don't know if people actually follow through, but. You know when you visualize filter? Like, let's just look at the first layer, because on on other layers becomes more complicated, but I'm on the first layer. I'm assuming that each unit is?

Feature detector, so what it does? It takes the input X the compos computer cosine distance with, like a column in a weight Matrix. So that is your template. And then, if it's high, then the activation function. Um, fires. If it's low, it means it's not a match. And if you have a value, for example, it goes to zero.

Uh, so therefore looking at these Columns of the weights the templates are comparing to is a way of understanding what the unit does right to visualize this template, and it looks like some object and say, okay, I'm detecting that object, so similar to the sensory analysis. Can you construct, or like, how can this go wrong?

Like, when can you look at this filter and say, oh, this is supposed to detect the picture of a cat, and it actually does not do that. Does anyone have an idea of how they you can break this out? I mean, the whole point here is that you can break any kind of analysis you do.

If this is not going to happen in practice, the CNC Maps is not going to happen in practice that much either, but you know that technically, you can do it mathematically. You can do it, but how can you mathematically break this one? Does anyone have any idea?

What are you not looking at when you're just looking at the weights? Why is it exactly? Sorry, what did he says so, like? One thing you can do, for example, is, can have a bias that's minus a thousand, so that unit is always zero. Like, you can look at it, and you can argue, oh, this is detecting.

I don't know this particular pattern, and therefore, you know, it's going to be used upwards when it does things. But if the bias is? Very negative, and you have a relu, then basically that unit is not used, and you can't tell whether a unit basically it just because it's sort of like a.

Because you start at the bottom going upwards. When you do this kind of visualization, you never check whether you can check what this this unit is Computing, but you don't check whether the rest of the network is actually using that unit. So, therefore, if you start making a hypothesis that is using this unit to do X.

That might be wrong, like the unit might be trying to check whether X is in the image, but doesn't mean that the rest of the network is going to use it to do anything for its prediction. And it happened in practice. If this can happen in practice, yes? Usually, it shouldn't necessarily happen.

In practice, it can happen in practice. If you mess with the optimization process the wrong way, like, for example, you have a strong, uh, like L2 regularization that you apply on the hidden units or something like this, and the model decides that the easiest way to? Uh, minimize that regularizer is to to do the biot, make devices negative, or something like that, like you could.

You could like it could happen, but it's unlikely depending if you have a standard pipeline that is student, and everything is nice here. Um, I don't understand the emphasis on the first layer because? Oh, it's just because it's a bit conceptually a bit easier. So when you go to the next layer with for this kind of filter visualization, usually what you do usually take the weight.

You take the inverse of the activation function and usually activation functions don't have an inverse, so you take something that you approximate as an inverse. So, for example, for relu, the inverse is relative. That's what people typically use, which makes no sense because it's not. But anyway, you take the weight you apply relu, and then you multiply with the with the weights and the layer below to kind of compose.

So easy to collapse. This whole thing to a linear model and then visualize what that looks. And. Yeah, so I put the first layer because I didn't want to talk about this inverse, that I think is wrong. But this is what people used to do. Maps is the most standard thing that people know nowadays in the early 2000s.

This is what almost everyone was doing, looking at these filters and and then trying to. Sort of approximate. The activation function with sort of the expected behavior or some inverse or something like that and then and then get this to be like a linear model and sort of collapse it into.

Into again, sort of like a linear map, and then look at the weight visualize it. Okay. Let me. So today, the idea was, we talked a little bit about representations and and sort of how you parameterize a model. And today, we wanted to talk about learning. And optimization. Um, so, um.

Maybe I, so this is a slide that presented in the past before, so this is not meant to be necessarily correct. Mathematically, it's more meant to be very kind of, like, illustrative of what's going on. I'm just gonna try to explain it in words because maybe words are way better.

Um, but the whole point I'm trying to make here is that there is a distinction between optimization and learning and generalization and and. Um, ml mostly cares about generalization and learning. Uh, but you know? Optimization is more the thing you can formalize and the thing that there's a lot of theory for.

So, what is the difference between these two. So ideally, when you have some loss, some loss, which is just a distance between the output of your model and some label that you have. What you care about is. Minimizing this loss, so minimizing this distance. Over the entire domain, right?

So, like, if you're doing this for images you care about over the entire domain, and this is sort of. What learning and generalization is trying? Generalization is all about, so it's it's not about it's. It's about making sure. That this function H matches the underlying process that generates the label y, right.

So, basically, you want to have good predictions for all possible images. But what, you know, Computing this integral is intractable. We don't have it. We don't have the data for it. You know, you don't have all possible images that ever existed. So, what we can do instead is we can approximate this integral by a sum.

So what we have is you have a bunch of samples? And what we can do is you can minimize the distance. Between the model output and the labels on this data set that we have, right? But this is different from what we care about. Like, we don't care about.

We, we already have the labels for these data points, so the fact that we get a function that gives me the correct label for a data point that I have in a training set. It's not useful, like I already had the light bulb. The only thing I hear is how well is this function responding to data that I do not have, right?

So, I care about this integral. I care about the generation part, but the only thing I can work with is the optimization part. Um and. We will talk a little bit of how? Optimization can help you reach some of your goal in terms of generalizing. But the first part of what I want you to present today was the focus on the optimization.

So for now, like this is what we really care about, but for now, let's ask the question if I have some function. H does parametrize by Theta? And I have a bunch of data points x i y i. How do I basically? Find the Theta size that this distance is minimized on on the training set.

So, if I have some random loss, so this is the last compute on the entire training set. And looks like this. Like, how would you go about finding the Minima of this function? Anyone has any idea? Got 20 cents? Yes. So, how would? How does gradient descent actually work so?

You know, the first thing that you that that you see, is that, like, okay, this looks very ugly, and there is no way of having some kind of close form solution to it. There's there's nothing there to to do that. So, what you have to do is you have to do some kind of iterative search to find your minimum.

So, how does gradient descent works? Um, and I'm again ignore the math, but I think this intuition to me at this is quite useful. So, what you do when you look agent is sent, you do the following. You say, I have no idea how to optimize this function. But I know how to optimize linear functions or even quadratic functions for those.

I have close form Solutions. I know how they work. And, and I know that any function can be approximated by a Taylor expansion right, and the Taylor expansion goes from linear quadratic. And you know, or third time, and so forth. So, what I'm going to do is, I'm going to pick some points.

I'm random point where I am. I'm gonna say, let me approximate the function. I'm trying to minimize by a linear function. Uh, so, so the the red dotted line is linear function. I'm approximating my. And now, the nice thing is. I know where the Minima of this linear function is.

Because the linear function, I know how to minimize it. The issue is that the minimize somewhere at Infinity, either plus or minus infinity, depending on how this looks so it's not? It's not that informative because it's probably it's not the minimum of the of the black of the. Of the initial function I wanted to to to to minimize the the black line here, but the reason it's not matching is because my data expansion is only true locally.

So, then, what I'm gonna say is gonna say, I'm gonna construct the trust region. So, I'm going to say, okay, I can only look within this circle. And then, I'm gonna ask myself, what is the Minima of this red line that I know how to minimize within the region where I think this approximation actually is reliable.

Um, and you can write this down and this is turned out to be. Um. Um. A constrained optimization. That will look something like, minimize. You know, this is the first order tailor. So, this is my linearization of the of the function minimize L Theta plus. Uh, Delta, times the gradient.

Such that the Delta, the thing that I'm minimizing for the step that I'm minimizing for, is small enough that this approximation is actually reliable. And then the way you solve this constraint optimization is you use LaGrange multiplier to take the constraint and push it back into the equation. As I said, the math maybe is not that important, but probably many of you know how the garage multiplier works, but anyway?

So the way you do that you use learners multipliers to move this here, but you don't. You don't care about. Efficient, right? You, you treat that as a as a hyper parameter. So, what you get when you push this up is you get that. The update is actually just gradient descent.

Because you know, this is like when you push this up. This is a this is the second order term, so this becomes a quadratic. Uh, with the identity as the coefficient for the second order term. So when you're trying to solve that you, you basically just gradient. You just get a gradient, so you take this formula.

You take the gradient of it, and you check to see when the gradient of it is zero, and then what you get out of. It is sort of you are your your usual update, where you have the learning rate, which comes from this LaGrange multiplier thing. Uh, does that make any sense in terms of this works?

Or, like, intuitively, what you're doing so when you're doing this iterative map. Like, right here in the center, any kind of iterative optimization. What you're really doing is, you are starting somewhere. You're making a local approximation of your function to something similar that you can solve, and you're restricting yourself to the area where this approximation holds.

And then, you're solving this approximation rather than solving your problem. Within that area, you take that step. You get there. And then, you repeat this process, and you keep going like this. This is kind of the idea of most of these iterative searches.

So this is the same thing. So here's the math a little bit sort of more like spelled out, so you had this term. This terms becomes which you can write it as a Theta squared. I'm kind of ignoring I'm kind of. Being very loose with, like a proper metrics, notation and stuff like that.

And just like, just to make it look a little bit simpler. So, how do you solve? This, uh, so once I put my constraint into the main equation, I get sort of this this equation, that's of a second order equation. How do I solve this? I take the gradient of it and set it to zero.

And I basically sort of get the the update I was looking for, and this gives me the the standard gradient descent update that everyone knows where. Like, at every step, I I move in the direction, the opposite direction of the gradient, right. I take. Uh, and decide just sort of the steps of driving that.

Are there any questions before I jump to the?

Yeah, so you said we took the constraint and put it back in there in this equation, and the Costa Rica is the the normal that Theta is actually less able to. Yeah. So how do we? How did we actually reflect it in the first question? Right. So, like, I multiply.

What it says, is that the solution of this constraint problem is equivalent with the solution of this? Formula where you just add the constraint weighted by coefficient. This is the LaGrange coefficient. This ETA that you end up here. So, this Norm of theta is this term here. And this boosting is sort of the coefficient that comes from.

Um, so this is how you solve sort of constrained optimization, so you have. I think it's for equality, and then if you have inequalities, there's sort of the extended form. Well, anyway. I don't remember how they have a particular name, but in the end, you end up with the same thing.

Is you basically just move your constraints within the equation, but you end up waiting for them and typically what you do if you want to solve this as a. Properly is there is a way for optimizing for this coefficient. Outside. So, you need to. You need to self solve for the coefficient and then plug in the right coefficient here.

And then, when you solve this, you get the exact proper solution, what we do when we're doing. Machine learning is. We're not. We don't care that much. This is, we don't have that much to properly solve this. So, what we're saying is that we're gonna replace this ETA by some constant.

Uh, because? In some sense. Like we. We don't even know what Epsilon should be. Let's put it this way. Like, we, we don't know, like how big this area needs to be, such that my first order still holds. We just know it has to be small step. That's, I mean, you can compute it.

I mean, this is all about sort of adding curvature and all of this stuff into the equation and trying to get a sense of how quickly my loss surface changes to get a sense of what is the the area where I'm gonna trust my approximation? But for now, we're talking about that.

So we, we just randomly pick an Epsilon because we don't necessarily know what you know, because we don't care about what Epsilon is, except that it's very small. We then as well here. Don't don't care about. Solving for the LaGrange multipliers. We just take this form and we just replace this ETA that further on.

It's going to become gamma because. You know, one over two ETA. We're gonna replace this by the lining, right? And then we're. Hypertune for lining, right? All I know about lining rate is the steps should not be too large, because if they're too large, then basically, what happens? You're not respecting this trust region here and?

Moving outside of? Where the approximation hold? Um, but yeah. So, basically, we'll press that this one by the Norm's point. Yeah, yeah. So, this constraint becomes just, uh, yeah, I should have put a square here because you don't want the dealer square roots and stuff like that. Sorry, like, typically, um.

When you say Norms, we always means Norm squared. Okay, so that we don't need to worry about the square root. It actually does not matter what Norm you pick. You can use whatever Norm in this in. In some sense, you know, as long as you're restricting your step size, that's all that matters.

Okay. So, in order to be able to apply gradient descent, so do this all of this. What we need is we need a gradient of the loss with respect to the parameters. Um. And a way to compute. This is the backdrop algorithm, right, that probably everyone knows and the way the background back propagation algorithm works.

You started the loss. Yeah, the top of the loss. And then you apply the chain rule, right? You compute the gradient of the loss with respect to the output and then multiply this with the Jacobian of the output with respect the previous line and so forth, um. And. At some point, I forgot when there used to be this discussion on Twitter, where Jurgen was trying to claim that backdrop is just a chain rule.

And it's from the 1700s. So, I guess one question is, is this just a chain roll, or is there anything special about back problem because this is a proper paper right in the 90s. I think it's a paper from Jeff Hinton and others that introduced a back propagation algorithm.

There's then afterwards, a paper about back propagation through time algorithm, and these are, you know, those kinds of papers like thousands and thousands of citations, and so forth. So, is this just a chain rule that we knew for hundreds of years, or is there anything special about it? Like, what do people think?

I think there's a joke.

Yeah, so the think about back propagation. So, just to defend the paper and Jeff and the others. So, there is a trick here, which is. Um. Backrogation is about engineering, so the idea in backropagation is that it gives you a particular way of computing the gradients. That leads to minimal flops and memory usage.

Like the and, and basically, the trick for a simple MLP the trick. It's kind of easy to understand is the loss is always a scalar. So, when I'm Computing the the gradient of the loss of respect to the output, this would be a vector. And then. From then onward, I have Vector Matrix multiplication, which will result into a vector, so the state that I have to carry on as I go back to the architecture is always a vector.

If I start from the bottom to apply the chain rule, the general does not have an ordering. The general is just well. I can start in the middle and go anywhere, I want. But if I do that, I have to keep around Matrix Jacobians. They're actually quite large. And.

The computer has to spend when you do a matrix Matrix multiplication and sort of the memory that you have to keep is much, much higher than if you do a vector Matrix. So, this is. Basically the. The main thing about back propagation is, it is not? It is the chain rule.

I mean, there's nothing else you can do. You're trying to copy the gradients, but it is a way of arranging your computation to make it quite efficient. And when you have a what's happening? Yeah, about that.

Okay, okay, so when you're? And machine learning. It's all about how do you make these things run efficiently? I mean, that's what made machine learning work. So, so that is the secret. And when you have more complicated, uh, computational graphs, like, for example, you have skip connections. You have recurrencies.

You have all this kind of stuff. Um, it becomes even more apparent that there is a particular strategy of how you're supposed to computer gradient how you're supposed to go through it. That will lead to minimal memory and and and flops. I have a question. Yes, you said a question, but like, this is in 90s.

Yeah, around 50 years, why the one introduced anything that used substantial Improvement that through that propagation? Is it because the bike propagation is already complete and there is no room for improvement? Okay, to just take? For granted, uh, in, yeah, so I mean. Your meaning. Okay, so first off, let me ask the question.

So, you mean improvements in how you compute the grades or improving our Hardware and optimization, like kernel wise, but you are not doing anything related to the algorithm itself. So, in terms of how you compute that gradient. There's no many other options, so this is, in some sense, if your goal is memory and and computer efficiency just to computer grade.

I mean, for optimization, there are other algorithms instead of gradient descent. Like, you can do other things, but just sort of the question of how you compute the gradient. Uh, if the goal is minimal compute and minimal memory, uh, backdrop is the optimal thing you can do. There is no other ordering of your gradients that would give you something better.

There is another thing that people do. For various reasons. I don't think I have the slice for it, but we have what people call forward propagation, which is, you start from the input and go all the way to the top. When you compute a gradient. Um, and this is useful.

In different scenarios. So? I'm trying to think of a okay. I'll give you the standard scenario, but it's a bit sort of out there, but the center scenario is for recurring neural networks and the point for economyo networks is when you do back prop. Uh, you can imagine that you have to go in time all the way to the end.

And then, you start at the end computer gradients, and this is biologically not plausible, and also this makes things quite quite slow because you have to wait. If you have a very long sequence. You have to wait until the end of the sequence. If you start from the input, well, it's nice for recovering your networks is that you can compute the gradient and forward at the same time, and you just go forward in time, and that's it.

So this is the memory. Consumption is high is, um? And Cube instead of n or n Square to something like that. Anyway, the memory consumption is pretty pretty high. And people don't actually use this that much, but it is biologically plausible because you're Computing both signals at the same time, which I mean, it's not fully biological plausible because gradients are not values, but at least it doesn't require any move backwards in time.

It's always forward. And it can be useful in all kinds of weird settings. So, this is called real-time recurrent learning rtrl. It kind of it was big. In the 90s, it went out of fashion, and now it's been revived in some niche papers, but it's not sort of mainstream.

Um, and the reason is that mainstream is because the memory consumption is high. And is typically backdrop. Two times still works the best. If you are just like. With the hydride that we have with everything it just waste the best. But yeah, I mean, I don't think there is much room to improve how your computer gradient.

There is room in. What do you do in the gradient, like, instead of ingredient descent, doing sort of like fancier stuff on top of it? Yes, sorry about you saying that you, you just do forward pass. Um, like you mean that you do forward pass and backward person at the same time.

Yeah, okay, yeah, yeah, because you start at the bottom, right? So you like, even for for a few dollar models like, you don't need to go all the way up, and then you can try to do your backwards if you do. If you start from the beginning as you compute H and at the same time, you can compute the Jacobia and all A2, resp.

And then you keep this going forward together, right? So you keep accumulating. So whenever you do a forward, you compute both the forward and the jacobia. Which is, for most cases, the Jacobian is in close form, right? So, you, you just? It's a function of X the and and W the jackpot here.

Um. That makes sense, right? So, so you can confuse the copies as you go forward, and you can multiply them as well. It's just that. You always get a matrix Matrix multiplication for the forward for between jacobians. And the the memory footprint is much higher. So, there's the issue is like computationally High memory high, but you don't need to go up and down.

You just go up, and that's it. Yes, this is this is the trade-off. Yeah, so there are reasons why you would want to do this. As I said, because you don't want to do backwards. Um, but usually if? Compute and memory. Is your bottleneck then? Backdrop is the optimal stuff you know.

Oh, oh yeah, this is sorry. I did not know I have this light, but this is exactly a combinational graph of the backwards mode, so this is backdrop back propagation, so you start from the you you have the? So, you should start with X, but here's exactly with H.

You have H you compute H2, you compute H3, and you compute loss. And then, from these two, you can compute the derivative of the loss with respect to H3. And from this, you can compute the derivative of the loss 2 with respect to H2 by just adding the Jacobian, so you compute the Jacobian and then you multiply the Jacobian and then you get, and this is sort of the computational graph that you get to the backdrop.

Here is the forward mode that I was discussing where you compute H1 and then from H1. You can, uh, and you compute HR and compute H2, so there should be an arrow here as well, so may challenge to you can compute the Jacobian so down. From then onward, you kind of just keep going down and compute them in parallel.

Um. So? Um, yeah. This is sort of the. What's most popular ways of of computing gradients backwards and forwards? And um? This is actually. Yeah, I mean. You know, we're so used to have these libraries. I'm sorry, I'm just. Uh, okay, so I'll we'll try to get a break in five minutes.

I'm just gonna try to finish my. So, we're, we're so used of using something like python type, bread, or whatnot of the things. So, uh? All of these libraries, the way they work. It's exactly like this, so. I mean, yeah, so then you have a sense of what's going on.

So usually, when you build an architecture. The all of these libraries, where they try to build under the hood, is a computational graph that tells you how the computations go. And then, when you call grad, they just take the computational graph, and then they construct the computational graph of the gradient, and that usually is done by reverting, you know, starting from the end and inverting the errors and you for each the way pytorch Works in specifically for every layer.

Uh, basically, it has defined in close form of the Jacobian of the layer is. So when, when it needs to do the backwards, it just sort of goes through the computational graph, you know. And then, for each layer. It knows what the Jacobian R and computes the Jacobian and construct the graph that way.

So, so usually, um? The main sort of data structure that's behind all of this library is some kind of graph, which is the computational graph and most of these automatic differentiation and some other stuff that you that you do. They're all just sort of manipulations of this graph, based on on specific roles.

So this is how these things work. And then sort of the. The hardest job with this stuff. Uh, which I think nowadays new libraries that don't even attempting to do as much is to try to optimize the graph. Like, I remember in the early days when I was working on this, so I, I spent doing my PhD sometimes on piano, which was one of this first libraries to build things.

One thing that we were worried about is you get a graph like this or more complicated, and then you want to check whether the graph is computationally optimal. So, whether like you can, like, do a different graph that has fewer nodes or fewer edges, because somehow you know you can reuse some of the computer?

And um? Most of the times. The gradually we have. There are not computationally efficient. We're not using compute, so there is a lot of room to optimize these graphs, but like to do that automatically for the user without it's. It's actually really hard, and Tiano tried to do this, and we ended up, so we had, uh, like one of these nodes, was to do recurrences.

It had the op scan, and it ended up that when you had a program that had the scan, you had to wait like 15 to 20 minutes for the code to compile, so to say so that you can actually run it because it was trying to optimize the graph and most of the time.

It wasn't able to do anything useful. Um. But nowadays, I get sort of why people do is when they identify some inefficiency and how computations happen. They build a care now, so they build a specialized layer for that particular thing. And I think this is a much more scalable way of dealing with it, but um.

Yeah, in principle, this is sort of the. The main thing behind this libraries is right. Building the occupational graph, using that to to generate your gradients and other kinds of derivatives or other kinds of things you might care about, like Computing, hessions, and whatnot. It's the same. It's like you can use this graph to to build sort of the graph to computer Hessian, and uh, then manipulation on this computational graph to make it faster.

This is kind of the the bread and butter for for things like pythonic index and so forth. Um, okay, let's take a short. Oh, there's a question, and then we can take a short break. When doing it, I think there is.

We talked about is? Radiant descent, which is iterative process where you are linearizing the system, or you're taking a personal retail expansion of your model, and then you're restricting yourself to some kind of trust region where you assume this. Uh, this linearization holds. And what gives you what this gives you is a constrained optimization where you're trying to minimize the simpler function.

The the approximation that you've made within a constraint on how much you can move? You typically use LaGrange multipliers to deal with the constrained problem to to transform it into an unconstrained optimization Problem by moving the constraint within the equation. This becomes the second order equation that you can solve in close form and what you get out of it.

Is your typical gradient descent step, right, which says move in the opposite direction of the gradient step. But this is the intuition of what's going on is you're. You're solving this problem that you don't know by taking like by making many local approximations and solving the local approximations to move forward in all of this process.

In order to, to do this, you need to have the gradient, the way we compute the gradient because neural networks are just sort of a big composition of functions.

The chain rule in order to minimize memory and compute. And, and that's sort of, what the back propagation algorithm is all about, and in. Auto diff world. Maybe, uh, it's worth mentioning. So, again, there is also like, in some sense, you know, back propagation. There is a paper and everyone celebrated and whatnot, and it is sort of how people in machine learning refer to the algorithm in Autodif world, which is a different Community, a different field.

They would call this backward mode, and the other one would be the forward mode. And I think this probably existed before back propagation. People did not know this. It just that's when people in machine learning realized that they need to do their Synology to save money. Things. If you look across Fields, you know, things get reinvented all the time.

And in different ways. So this is backropagation, and on the next slide. So this is that is an exercise, but it's just something we're gonna work through is notice in an exercise. It just sort of like, generally, you know. You can see what happens if you do backwards and and, and um, when you have a much more complicated computational graph, so you have this tag where you have, uh, from H1?

Computing this to different embeddings H2 and H3, and sometimes you can put the loss and then you know to compute this. What you can do is you can take the Brute Force approach, which is, you enumerate all different Pathways that go from the loss to the input. So, in this particular case, you have two Pathways, one that goes through H2, the other one that goes through H3.

Path independently, and then you add the terms. So this is what in this exercise is called Brute Force. This is not what back propagation algorithm would do. This Brute Force. I mean, it doesn't look like it here, but it is inefficient. Like, if you have a much more complicated dark with a lot more branching, like, just enumerating all possible paths and doing each pass individually leads to a lot of wasted compute.

What backpack would do is basically this algorithm which what it does is basically. Pathways have a common node. You basically collapse them together, so you, you start both Pathways in parallel. You compute the gradient of the loss to H2 the gradient of the loss of H3, and then at this step, because these two Pathways going to the same node.

You sum them, and then from this, so. So, in this particular case, this only saves you one extra Jacobian multiplication. So, if you look at the computations that you have to do is you have to do one last message multiplication, which doesn't seem like much. But when you have a complicated dag, if you do this.

Nation as soon as you can. When you have a node if multiple ages going out of it. You're gonna end up saving a lot of computer memory and then sort of. Fundamentally, the backdrop algorithm, and it's usually framed or some kind of like Dynamic programming algorithm, which gives you sort of the most optimal way of decomposing things.

So? Yeah, I mean, this is the extent that I'm gonna talk about back propagation, but maybe in the homework. Oh, this has reminded me, um, I will. I think today we will try to send the first homework out for people to to start looking at it. It's gonna be due next week on Wednesday, so there should be plenty of time.

But. Some exercises are a bit more open-ended, so I thought the more time you have the better and just to kind of see how, how that goes. Okay, so back from. Okay, so we're still at the back propagation algorithm, so this is a typical neural network. This is the gradient if I take a simple MLP, I also realize that I never mentioned these things are called MLPs.

I think everyone knows that are called them one piece. But if I take a scene, simple MLP MLP stands for multi-layer by subtron, but that's sort of your standard neural networking for Network. I, I write down the chain rule written sort of in the way backdrop would do it from the loss to the beginning.

I get the series of jacopians, so in. When you're doing effect, prep on MLP. You basically have an alternation between two types of jacobians, right? It's the Jacobias that correspond to a linear projection. And usually the the Jacobian of this is just the weight transpose. They should be a transpose there.

But anyway, there's a transpose when you come to the Japan. You have the Jacobian of the activation functions. The activation functions are typically diagonal because the activation function is alignment wise operation. So, the Jacobian itself is diagonal, so you don't actually when you when you multiply with these things, you don't actually need to do a matrix Vector multiplication.

You do an lmy's Vector multiplication, which is kind of faster. But the point that I wanted to make is. You end up with a lot of dacobians that are the Jacobian of the activation function, um? That's sort of. So your choice of activation function matters like we haven't talked that much.

We talked about relu from the point of view of expressivity and what you can do with it, but I haven't talked that much of how do you end up picking an activation function, what you're looking for. So, this is the. Sigma evacuation function, which used to be the default for neural networks and then the level, the one we've discussed and in blue.

You have the um, the forward the the function itself and with the dotted yellow lines. You have the the Jacobian, the derivative of this function, right to see how it looks. And one thing that you notice, for example, for sigmoid. Is that the derivative is actually quite low? So, when you have a very deep model?

And you have to multiply many of these jacobians. You can see that. You know, they can only do one thing which is shrink. The norm of your signal? Um, and then you can bound how fast it shrinks the the norm of your of your signal. You can kind of get a sense of how quickly your signal will vanish, but overall.

That is sort of the reason why sigmoids are not in fashion anymore is because sort of their properties. When you look at the backwards. They're not very nice because what they do is they have this very fast. Um. They, they shrink the, the, the norm of your gradients very fast.

And, um, basically, once your gradients disappear as you go through the chain? You, you stop learning. Because if you don't have a gradient that connects your loss to the input, it means that you cannot find out what is the relationship between your input and the loss, so you cannot really learn anything right?

You cannot learn what is? Y given X? Um, on the other hand. Relo has its own pathology, which is, while detective, the gradient is one, so there's no, no vanishing of the signal. But if the value is not active, the gradient vanishes immediately. Initially. People are worried about travel because it has this region of zeros, so there were.

He was, if I have a model, I initialize it with values. What happens if suddenly in a given layer? All my hidden units are dead are in the zero region. Then there's no signal flowing through. Then there's no learning happening, and my model is. This one is maybe how people were thinking about this or in the early days.

What happens in practice is that that's a very unlikely event. Actually using the standard internalization and so forth. That almost never happens. And in general. Like, if you really want to approximate how relu is what is doing to the backward signal, you might say that relu like multiplies the signal with 0.5 to kind of take an expectation, or you say half of the time is 0 half of the time, and so on.

It's maybe like scaling it down. It's not really 0.5, but anyway, it? It scales down the signal much. Not as aggressive as sigmoid, and that's sort of what. What kind of matters in this thing? Question yes, when does it happen then? When does it become zero? When does it cost the the rather one is a zero when the input is negative?

So, in practice, like, when does that happen because you said there's almost never happens? Oh, um.

It is in both States quite often, like. Actually. So, so you have two states for a unit that are very problematic when you have a value. One is when the unit is always zero, so then the unit is that, and you're basically losing capacity even though you have those weights.

Like, you're not participating in the model at all the other time. When everything is problematic is, the unit is always on. If you have a layer and the unit of the layer, I always on. Basically, you have. You don't have an activation function. You have two linear projections one after the other.

So, again, you're losing capacity. So the optimal use of your activation function is a 50 of the time. Is that and 50 percent of time is active because that's when you actually have access to non-linear behavior from that unit? Uh, so you don't want to be on any of these sites.

Uh, there was another question, yes. And can we just use the leaky relu to be saved from the zero problem? Yeah, definitely. You can use the I mean, and people nowadays don't don't use Railway anymore, right? We have jet glue. Whatever all of this variants, they're all very similar.

But yeah, some of the things that they do, is they soften this this area, and sometimes they have a quick self? But yeah, you can definitely do that. I'm just saying in practice relu is not that big of a problem either, so you can use it safe? Yes, yeah, one of arctified value.

Yeah, and we also use it to actually welcome this issues.

Is the switch that is the gear glue. There is the Leaky relu, and then there's a bunch of at some point. It used to be very popular to come up with activation functions. It kind of died out. There is a an argument that I'm going to talk about a bit later about JetBlue, or or basically, so they call it an activation function.

I wouldn't call it an activation function. It means that's an activation function, but they have this thing where you Branch out, you have, you know, it's like a it's a it's an activation function with parameters you Branch out. You have two different projections, and then you multiply them together, where on one side you have gel low activation function, and the other side is linear.

So, um? In Transformers, though it's being. Now, replaced Again by a different thing, but it used to be the default. And that used to work way better than all the others, but it worked better because you have this sort of gating Behavior. This multiplication behavior, not because of the activation function itself, which in the end was sort of very similar to all of this.

It was jello, which is. Version of Bello. So, so this doesn't matter. So, um. But. In general, most activation functions are a variation on this shape. Um, I think there is still sort of. Open question of what is, like, an optimal activation function? Um, and I think there is still room.

To come up with new activation functions, but I think. The way I would do it if I would have to do research on this topic or if I would try to do research on this topic is, I think there is enough evidence that we didn't domain. Um. All activation functions at some point kind of start behaving the same, particularly if you have sufficient scale, and you optimize your model properly.

So, if you use, like? Not a super fancy Optimizer, but a decent Optimizer that can forg for some issues of the curvature. Like, add them, you know, like, Adam, W, or whatnot? The version of that then, um? And and? You know, you allow yourself to scale as much as you need?

Um, you'll not see any difference between activation functions, but another way to think about activation functions is as part of the architecture. They Define the inductive bias of the architecture. So, the fact that you use value or something else my biases might bias you towards certain kinds of solutions.

We not necessarily know how to describe this mathematically, but there definitely. It's part of your exactly bias, so if I would do, uh, like architecture, search to find new activation functions or anything like that. I would look for activation functions that behave differently out of domain. It's a bit of a lost cause at the moment to come up with the righteovation function that does better than the ones that exist.

I think they all kind of do the same. There is one space where this still matters, which is, uh, from what I understood, for example, a Gemini scale when they do the extra large models. Because everything is so expensive. Like, if you have an activation function that? You know, it's a few less flops than the other, but kind of behaves the same.

It still kind of matters to them because you know, like flops, matter a lot when you have these huge systems or something that helps you on the hardware somehow. Then, sure, I, I would buy that. But if you're like a normal researcher where you're not thinking about this extreme life scale scenarios that I don't think it matters.

Which activation function you use, like, any of them are, is fine? Looking for out of domain? I still think that. The choice of activation function can make a huge difference. Because they they sort of shape. The inductive bias of your model, and I think that's more of a. Reliable space to look for Activation functions.

But yeah, I mean, for, for my lecture, I'll mostly talk about relu because I find it simple, but almost anything that I say. You can replace it with leaky relu or whatnot, and it's kind of the same story.

Uh, what? I was talking about the extension I was talking. Sorry. When I was saying about the extreme scale about flops, isn't that about outliers. It's about, like, making things sort of cheaper, so some of these activation functions might involve say the X function and expect, actually, not that cheap to compute compared to squaring or multiplication.

So, if you can get the same effect or the same kind of behavior with an activation function that uses simpler operations or fewer operations. Atomic operations, then, that can be a big win at when I say extreme, it's all about scale and compute, right? Um, like the?

Uh, so you know anything that you can save there. It's, you can. It can mean a big thing. Um, but even that I don't think is that active, even there. I think people are pretty traditional in their choice of activation function, because it's also like a place where you don't want to take too many chances like you might have this activation function that looks the same as a relu, or like the jaglu or whatever is your activation of choice.

Are not 100 sure. So, like, it's not worth it. Taking that risk, you know, if you have your pipeline, everything is there. You just kind of go with what's there. Um. So? The other thing I wanted to say is that? When you think about optimization and you think about gradients and all of this thing, like, one thing that's important in neural networks is, like, so far, in the talk I've been.

Presenting things from a functional point of view, right? You have a function in computer gradient that function you can go with it, but like, what's really important is to understand how the signal propagates through the network. I mean, I think this previous slide is exactly about that, right? So, like, if you have the wrong activation function as you go backwards through your network from the loss to the input.

The signal might vanish at some point because of the activation function or something else. To a big extent. This is. Maybe one of the main tools that you have to debug architectures, like looking at how the signal propagates with the architecture, is actually quite useful and quite important. And um?

Of. This was one of the big breakthrough for for deep learning. So, and this is comes back to like things like. They were doing sort of initializations for neural networks, and then you have. Coming in, and others who did sort of other variations. And all of these are about.

Uh, signal propagation, and usually the the kind of questions that they ask is. How should I initialize my model, and how should I use my activation function suching the such that, in expectation, the variance of the input does not vanish. As I go through the architecture, so this is usually the way things are being trained and usually look at a single layer and say, like, okay, I have variance one at the input of this layer.

I want lean expectation, sort of to see variance one or the output, and this is sort of the Machinery that they use. I don't know if.

Is too small, then you know you. You get sort of your gradients Vanishing concentrating around zero as they go through the architecture. If your insulation is too big, you can get your model to be too saturated. And again, when the, when the activation functions are saturated. Signal does not propagate well when the activation is just right.

You sort of kind of if you push a gaussian through you, kind of keep having your gaussian as you go through the layers. This was kind of the the philosophy. I mean, you can look at the papers to see sort of the math and how they derive this, but this is sort of the philosophy of intellization and and activation functions is.

I feel like I'm I'm going through too much math, and that's not useful, but maybe the one thing that I should say, which is kind of important is when they do this kind of derivations. If you look at the math, you sometimes make very strong assumptions. Um, like? Yeah.

So, for example, you assume the 10 age is close to linear function, and you get rid of 10h because you don't know how to deal with that age or things like this. So basically, all of this initializations are the output of, you know, applying this formula that you try to maintain variance and then taking kind of some strong assumptions throughout because you know, there's some mathematical element that maybe can be dealt with, but it's messy.

So usually, machine learning and things I messy, you just say, well, I'm just gonna skip it, and I'm gonna assume this is linear. I'm gonna assume this is independent from that, even though they're not and and kind of go forward.

Or not, and then you apply it and it kind of works well in practice and then sort of. You're good to go. Um, so this is the philosophy of ventralization. But decentralizations. And we're going to come back to this. This internalizations because of the assumptions you're making and the sort of these approximations in the math, but also because of what you're trying to do, which is you're looking at a single layer and you're looking at the signal, not.

Preserving variants in expectation. This is not necessarily the correct thing. This is almost, I mean. By definition, you're looking at the average case scenario. It doesn't look at the worst case scenario so. Any of this in Translation. If you take an MLP and you make it deep enough, say, 100 layers or more, you'll see that always the signal Advantage.

Because you're not looking at the worst case scenario. And and and, and then sort of, like when you make the architecture very deep, that that is sort of what happened so? And then, and the truth is. It's really hard to initialize the network size that the signal does not vanish when you have a very deep architecture.

Yes, is it? The question of initialization itself, the vanishing signature visualization it's? So, if you really want to solve it by initialization? And I think James Martin, so he's like a big personal organization so far has a paper on this, but I have to check because he's been working for it for several years.

I don't know if he put it on archive yet or not, but he's been working for it for several years. He's just that kind of person who never puts things on archive. Uh, it's also like a paper that's really hard to read. It's like 100 pages or so, so that's not something you're going to read.

But like, if you really want to solve your financialization, it turns out because of the non-linearity. There is no really close form solution of how to do this. So, what he does is basically he runs. If I remember correctly, like an optimization, like a layer wise optimization process to find out the right instalation of each layer as you go through the network, which doesn't feel a very practical thing, because if you just need to run like an optimization process to each layer just to insulize the model, so then you start training on something else.

Feels quite wasteful, so you could technically do it, but because of the non-linearity, the installation is very data dependent. It's not sort of something that you can bring close for, and it's probably never going to work very well.

Are sort of a much more pragmatical way of dealing with the problem, and it is sort of the reason why we can train these deep models. Like, without normalization layers, you'll never be able to train something that has 100 layers anything like that. Yes, uh, what was the target when they were coming up with these optimizations?

Like, where they're trying to minimize the variance of calculations across layers for the initializations about, so they were just trying to keep the variance in expectation as you go from one layer to another, so you want if you have a input variance. You want the output variance to be the same.

And this ensures that we're not going to have any value. It doesn't work in practice when we start stacking things. A lot of times, this kind of ensures that through that layer, like if you take the layer independently, you're not going to have Vanishing gradients. Doesn't pan out the right way when you start stacking things.

That's kind of the issue, but because you're doing something that's more like an expectation. You want the variance not to disappear. You're not. You're not looking. So, let me like, basically, the issue is, you're not looking at the worst case scenario. So, this slide? This is, um. Work from Andrew Sachs.

Uh, so Andrew sucks. Likes to look at deep linear models, which is basically just an MLP without activation functions. And that's all silly, because that's just a linear model, but the whole point is that you can make sort of theory around it and, and then sometimes the theory makes sense here.

And here's looking at like, what happens to the signal if you just have a deep linear model, so you remove the activation so you can do math. So, what happens if I put 100 layers of those, and how do I make sure that the signal doesn't vanish, and in this particular case, you have a closed form solution, and that is each of your um.

So, then this is kind of taking care of the worst case scenario. So, no matter what signal you're putting in, the matters can only rotate it. It can never like shrink or expand or anything like that. So, in a in, for a deep linear model, if you, if all your weights are like that, then you don't have any Vanishing gradient, and you can have a thousand layers and you still have radiant flowing and everything is fine.

The issue is that because of the non-linearity and in particular all the non-linearities that we have. Or shrinking the signal, or like they're maintaining it in some region like relu and then shrinking in the other, or like shrinking all over the place. So usually what that means is that your linear layers they need to expand the signal.

If you are the signal not to vanish because you know your activation is shrinking it, but the way that happens is sort of data dependent and all of that. So it's it's kind of complicated to make sure. That's why normalization is such a lot more practical because you don't care why the signal is getting smaller, you're just getting it if you.

If you notice the thing that's too small, you make it big again, and you keep going. And I said.

Or which is? Sometimes when you do that? You're messing up with the optimization process. People don't really care about that, but like, for example, I'm going to talk about this one together. But, for example, for bench Norm. Your, your Computing, the gradient of a different function at every mini batch.

Like, you don't have a function that you're minimizing. Like, if you, if you, if you're being very technical from a mathematical point of view, because because of the renormalizations that you do you, basically, you have a different function at every patch, and somehow you're hoping that this process doesn't diverge, and it's actually going to converge to something.

But, you know, it's one of those things where, if you even look at the paper like, no one even asked the question. Like, is this gonna affect convergence of my Optimizer from a theory point of view? Like, just people don't care about kind of details, yes? Oh, there is digital connections.

Yes, there is digital connections help as well for the signal because. When you have residual connections, if you look at the jacobians, the jacobians will have the form identity plus the Jacobian of the residual path. So, in that sense, you're ensured that the signal will not vanish because it always can go through the skip like, you know that the Jacobian can only expand, it cannot shrink.

You have the opposite direction where the gradients can explode because you have identity plus something that's positive. But yeah, residuals definitely help. I mean, the the? The formula that works that everyone is using is residual plus normalization layers. I mean, Transformers resonates. Everyone has that like, actually, nowadays. I will probably struggle to find an architecture that doesn't have residuals.

It does not have their normalizations, like all of them have. It's sort of by default.

Like, what I'm talking here is about the old world that doesn't exist anymore. Back then, we didn't have residuals and organizations.

Oh this grass, so these graphs. Showing sorry, what is this? So, this one is basically when they use this sort of normal initialization, and they're trying to show so. So, I think the the ones where the relative, uh, where the signal doesn't vanish. So, the one that are OT, so these are the ones that use this orthogonalization of the weights that allowed the signal for every, and this is the depth of the model.

So, 50 layers 100 to 100, 400, 800, and so forth. Oh no, this is the depth. I don't know what is 200 is?

Of the layers, and they show that if you have this orthogonal utilization, basically, all models aren't able to minimize the loss, so the loss goes down. Well, if you don't have this orthogonalization, the, the deeper the model at some point, it just kind of breaks, and there's no loss.

This is just like, uh, sorry. So, it means that orthogonal initialization tends to do better than gaussian instrument. Yes. Linear models. If you go to a Relo model, the orthogonal translation is not going to help you. Because of the non-linearities, this is the under sax work, and he is playing with these deep linear models where you can do things in, and he's just trying to make the point that, like, it's, you have to care about the worst case scenario and, and, you know, like, this kind of schemes that are preserving variants?

And so, the gaussian here would also preserve the variance the same sort of severe formula would do for for the relu.

That's sort of the point here. I mean, the paper is more than that. The paper goes over other stuff as well, but this kind of graphs here is. To give you that sense that, like this, the schemes that we do here. They helped a lot, and that's the reason why, you know, deep learning to fall, but they're useful when you have architectures of like two or three hidden layers.

When you have that, this decentralization is enough. When you go from like three layers to 50 or 100, this is not gonna be enough. Your signal will still vanish. The wrong direction. Um, and, and then this is the point that Andrew is making here. Uh, but obviously, the. The final solution that the community kind of ended up with is that the answer is not fixed the initialization, but as these additional things like skip connections and layer Norms.

Utilization on its own potentially can fix it, but it's really hard. And it's not sort of how you want to approach the problem.

There are no other questions. But let's jump into this so. The next bit. I want to talk about is. Um, so this is how gradient descent works. Um, and you can see that. Depending on your loss at any point in time. So, this trust region that you can have where the approximation holds can be wider or narrower, depending on on sort of how things change so.

If you pick the wrong lining, right? So if you hit the wrong step side, you either end up taking a very long time to reach your minimum because you're taking very small steps. And you could have taken big steps, or if you take two big steps, you're gonna jump over it.

And you know your system is not going to convert, you're gonna have this. Behavior. So? Yeah, so it's like, smaller trust regions means a higher LaGrange and penalty. Low learning rate and small steps, slow convergence, and a bunch of other issues. Larger trust region is low language and penalty, highlighting rate, light steps, and instability and lack of convergence.

So, the question is, can we do anything to figure out what is a good step, like, how do we get a good step? Yeah, I have a question. Can we dynamically like compute the size of the trust region, and then make the learning rate Dynamic with it. Yeah, so this is kind of what this section was.

Kind of going to that direction, so? There's maybe different ways of doing this, but one way to do this and this is sort of kind of. Selfie math is to ask the question how much is my gradient changing? And if my gradient is not changing very fast, that means I can take a large step because things are stable.

If my gradient is changing very fast. That means I need to take small steps so that I can recompute the new Direction and I'm moving in. So it's that kind of the high level hypothesis, and then if you. But I mean, ignore the one over anyway. If you just look at how fast the gradient changes, so you take the limit of, you know, the distance between gradients in some direction?

Once you add Epsilon to Theta, what you get back is the second derivative. This is the definition of the second derivative, and if you want to look at the inverse of this, because this is how much you should know, so you should move a lot if the gradient doesn't change.

If this number is small and you should move less if this number is big, so you take one over it then. What you get back is the inverse of the head shell.

Second order method. It's like you compute the curvature, the curvature is a measure of how fast my gradient is changing, and you can use this to rescale your step and say if the the the gradient is changing fast. Then I move slow into the gradient and what it helps, particularly so.

This is sort of if you're in one dimension. If you're in multiple Dimensions, you're basically scaling in each Dimension, independent key. Kind of picture is something like this where you have a valley where you want to actually have high curvature. One direction you have low curvature? Hd will end up jumping from one side to the other of the valley because the step would be light if you'd use the Hessian.

It will scale down on the high curvature direction a lot, so you end up with a direction that just goes down the valley, ignoring, without jumping from one side to another. This is kind of the. Nice type picture of. Of what this should do? Okay. So, um? I'm a bit confused.

Is it that, are we getting many are using LED with the momentum, or like? That's scaling in different dimensions. Yeah, so it will turn out that. Momentum is a way of approximating curvature as well. Without without Computing technology derivatives? Yeah, exactly? And then Adam, uh, or RMS prop is?

Yet another way of computing, second order derivative in some sense, approximating second order derivatives in axis a line. So it basically makes the assumption that the Hessian is diagonal and just looks at the diagonal elements and approximates the second derivative of the square of the gradient, which we will see that under certain assumptions.

It's actually not a bad idea. This. This particular thing is not technically used because this is very expensive. Computing the Hessian is very effective. This is more just a sort of a motivational thing. There are methods that are trying to use things that are not diagonal, so you might have heard about things like kfac or mon or any of this.

Okay, so there's all of this family of new optimizers that are coming out that everyone is excited about them, but They're not used, not at scales, because no matter what you do, Computing this time is quite expensive. And it turns out that you're better off taking a lot more smaller steps that are cheap than taking a big step that's expensive.

There's some Corner cases where taking big steps that are expensive is better. Default, you're better off doing the alternative. That's one thing. The other thing is the. Engineering over the engineering of doing. This is pretty messy as well, so Adam, it's like a one line of code in pytorch.

Their libraries that are trying to make more and kefac to be the same, but it's not really like you end up needing to understand the algorithm a bit more, and you can put a lot more to get there. So, so here, like, the head should not be, uh?

Compute, like, it's like if you rotating to this Valley not to be axis aligned right to be sort of like vertical. The Hessian will still do the right thing, because what it does it basically rotate things in with the eigenvectors, and then for each eigenvector. It knows how fast things are changing in this direction and then correct by that.

So, so the true, like, method that really uses the full hashion doesn't care about the orientations if I would be using something like Adam. Able to deal with this only. This is actually the line by X is the line. I mean, this is one parameter, and this is the other parameter, and you know, one parameter has high curvature.

The other partner has low curvature. Adam connects the learning rate per parameter, so it can give you a small lightning rate for this parameter. But if this is not per parameter, it somehow like, imagine you've rotated your parameter somehow. So, this is leaves living parameters, then Adam will not be able to deal with this.

Yes, what about? Uh, method like LL oh so, so sorry, let's go back. So those are good ways of approaching the the nutrient stock. So, there is the the map Peak, which is, I want to do something like H21 times the radius. There is a little bit it's usually do H, plus Alpha I to the minus one, just to make sure that, like, everything is well conditioned and the computers, there is some regularization to this age Matrix that we present practice, but you know, their formula, which is H, minus L to the gradient, and there is a question of how do I implement this.

How do I compute this? Rpfgs is just one partitionary for that.

Out things. Versus the Haitian is sort of like a Samo Grandpa updates, and you do rank one updated itself. In fact, it's a different approximation of this. There's like a bunch of methods there. Just sort of the what they differ is how they approximate that setting. So, what they're trying to do, technically, is the same thing.

And then they don't require really require much actually using them. Yeah, so generally, like the second order methods, one of the selling point is they don't require hyper parameter tuning, because now you don't have a lining rate in practice. You do end up with a learning rate. It's not really that, but the learning rate is considerably more robust.

Size differences is being taken care by the session, which is trying to estimate the step size that is true. Like all of these methods that your hydro parameters? Faster as well, because taking sort of these optimals at every time step. When it comes to applying Neptune neural networks, they do not work.

Work, but it's not an easy style of your thing. So, if you are in sort of non or convex optimization world or even non-commerce organization world. But, like you're working with, like, uh, nothing neural networks, any other kind of objects you have different kind of functions you want to optimize.

Usually, the second order methods are way better.

Considering more popular, they're not popular outside of the machine learning community, but they're more more popular because they take into account the structure of the neural networks doesn't have an addiction, and it's a great opportunity functionality. One, like, knows that you have this layer, and it does sort of sometimes lots of log structure, approximation of the hashtag, and it does some and help.

It's better because we know that there are some pathologies of the hashia and that counter structure work that you can exploit to make computations faster and so forth. And, um.

For this new are kind of method. You'll see that there are many papers and many groups, and many people are excited about them. They don't work, as well as advertise. So, my experience so far with this method is, if you don't care about the number of steps that you take.

You care about the number of hours on the cluster that you need to run your turn.

Steps instead of a thousand.

Um, the other thing that is not discussed as much. But is that this method? In a stochastic setting. Is in a non-sarcastic setting. But in neural networks, you rarely are in a non-schastic setting. You can never compute a gradient on the entire data set. You have to use mini batches you have to use, like small amounts of data, because.

You have to complete it on small amounts of data, which introduces noise as well. Yeah, noise. And these methods do not deal well with noise.

Are they not supposed to be equal this one and this one? Yes, uh, yes, I, why did I? Why did I say it's not equal? Exactly. Yeah, yeah, they're exactly, but I don't know. I always trying to be careful, and I feel not pretty person, but this is the example of the second order?

I mean, okay, so the way I wrote my way, so this is more like the way I wrote it depending on how you think of it. It's deceptional in the scale, like this is more like a directional derivative. So I'm just Computing the Hessian along Wonder you have to do for all directions I don't know.

Like, maybe in better notation, like, actually write the formula of the entire Hessian the way usually seen. The Hessian is more like formatting, which makes a bit more sense because otherwise, it's a bit unclear how you pick Epsilon and stuff like that. And what does it mean in the limit of Epsilon goes to zero.

We like the limit of the norm of Epsilon goes to zero, maybe, or something like that if some of your Matrix. So it's a bit weird, so that kind of reason I've I've didn't. I didn't spend time to think, what is that I correct annotation, um?

Mattresses and stuff like that?

But one question is okay, we have a formula. This is supposed to be. Ignore all the Practical aspects of it. What happens if I apply this formula to optimize for the function? The people. First of all, what is the minimum of X Cube?

On this, but like, okay, yes, but what is the minimum of x cubed? I'm sorry. So we, those support.


LECTURE 4:

And following the literature, there is. Thing about, like finding preconditional matrices, right? So you have this, you define, you know, you pick a piece instead. It makes your learning faster, and there's some choices of three that don't make things go much faster. In particular. The water. We're gonna talk about it.

Is natural gradient? Which in the end, like Adam, is very related to it. So the way natural. So this natural gradient, I don't even have the citations here. The natural gradient algorithm comes from Amari. This is a paper. It's a very interesting paper if you ever bored and want to read a paper.

So, this is. Uh, Amani started working on natural gradients in the 60s. I think the natural gradient paper for deep networks is like 80s or 90s. Um, it's actually hot paper to write, and Amari sunchi Amari is like a Japanese researcher. And. Until recently, I would say the Japanese Community was kind of isolated from the rest of the machine, like the U.S European community, and you can see that in the paper like it was the most painful paper that I ever eaten.

Because the notations are completely different, like he uses Einstein's summation, which I wasn't used to. He calls by sees a instead of B, and the weights are called. I forgot, like, no, no notation makes sense. Like, if you look there, it's like nothing of the standard notation holds, like the activation functions are weird the way he uses them, but it basically introduces this kind of concept, and all he's saying is, like, uh, I can't explain a high level that the intuition here is, like, we're doing the usual thing, where we're taking a linearization first lawyer, Taylor expansion of our laws, and we want to minimize this, and we want to create some trust regions some constraint so that we don't move too fast, right?

So that the thing costs? But we want that constraint to be in a functional space, so his point is, and this is something that's easy to notice. Is that there are many values of theta that give you the same function, so your your model is not parameterizing a unique way.

So, for example, one thing you can think of is you take two neurons and you swap them around. So technically, you change nothing. Because if you swap here and you swap above, you get exactly the same function. But if you look at the weight matrices. The weight matrices have swapped columns, so it's technically a different Theta, right?

If you're, if you decide two different points in the space, right? So, his point is that, um? The, the, the, the, the. Mapping between parameters and functions is not necessarily one to one, and even if it's one to one, you know, sometimes a change in in Theta creates a big change in your functional Behavior.

Sometimes it doesn't, right? So, it's not sort of like, there is a metric there that says that functions don't change the same way parameters change, so all he wants to do is to say, instead of saying, I want to take a step that is small enough that my Theta doesn't change more than Epsilon.

It says, I want to take a step small enough that my function doesn't change more than Epsilon. So, that's the the big the big idea. So then, the next question is okay if I want the distance between functions. How do I make a distance between functions and that's a very hard question on its own, so the choice that I'm already did was saying, well?

We know in neural networks that the output can always be interpreted as a probability, and it goes back to all of that things about probabilities so we can always think of the output of a neural network as some, some corresponding to some kind of distribution. And we know how to compute distances between distributions.

And this is what we're gonna do so, so P. Of Z, given Theta is basically the the distribution that comes out of your model and we can use Kyle, which is a Divergence. It's not a distance, but it's good enough and everyone uses, so I'm going to use the Cayle to measure how much my output distribution changes if I change Theta, yes, very same.

Here is total distance. Yeah, what does it mean in this context? It means that if I do Cal of P G given Theta. Okay, if I reverse the order of this, terms understand the same. Yeah, it's not semantic. That's the only that is the only one thing that's missing from hard version to be a proper distance.

It's not symmetrical, but otherwise, you know, it's zero. When these things are equal and everything else kind of the the? Triangle inequality, I think, holds. Anyway, there's like a bunch of properties that could fall. It means how different they are. Yeah, yeah, so it measures a way of differencing.

Just, it's not symmetrical so that that's kind of important and. That also means that if I reverse the order here, I'm going to get a different algorithm. I did because it matters when you do your expansion. Okay, so, so this is sort of what he was going for. Let me kind of try to skip out a bit straight.

So now, the question is, okay, so we've done the first step. So, we've we've decided how to make our trust region. But obviously, this time is, is nasty, right? I have the style term. I don't know what to deal with it. So, the next step in this process is okay.

I want to replace this constraint with a constraint that it's a lot more pragmatic that I can actually use to do what I'm doing. Um. So the way he does. This is, uh, he takes a secondary Taylor expansion of the kayam and then plugs in the secondary Taylor expansion of the coyote instead of the Chiablo.

And the reason for that is because once you take the second order expansion of the Kyle, it will turn out that that simplifies a lot. So if you write down. Again, I don't want to go necessarily through the map, but I'm just going to say a few words, and you guys can look over it whenever you want to.

Uh, if you, if if you, if your instinct is, but like, if you write the formula of the kael? Um and. Um, here, I wrote the sum of the values of g, p, z times log in Z minus. So, because you had the ratio of Pz times the log of the other thing, and then you you convert this into expectation because it's easier.

And then you start doing the the Taylor expansion around Theta for this term because you're gonna get the Delta Theta. What happens? And you just have to trust me. Is that the first order in that expansion disappears? Because you get log of pz Theta minus log of P z Theta, so that's zero.

So it cancels out. And then the first order term of the scalar expansion, the one that's just in terms of the gradients. Disappears again. And the reason that the reason celebrating more technical. Let me see if I have it. Sorry, do I have the math? No, I don't have the map right here.

So, the reason there is that you can reverse all of this, the expectation and the derivative are linear operations, so you can reverse their order. So, what you can do is, you can push the expectation inside, and you take the gradient of the expectation of this first return, and then you push the expectation inside, you get a one here, so you get derivative of 1 with respect to something which is zero.

Anyway, you can look this out. But technically, because of the expectation the first time disappears as well, and the only time that is left is the second order card. Which is Delta Theta, the second order derivative of log of P z, given Theta Delta Theta. And this is kind of nice, because now.

The second order derivative of log. Is going to be your metric. Yeah, so if I go back to the constraints, all of these terms disappear. And I really don't have the. No, I don't have it. If I go back here, basically, I end up replacing this tile. By a formula that looks.

Do I have anywhere from Adele? Sorry, I don't, but you replace this guy else by something that looks like Delta Theta second derivative of log P Delta Theta? So, the second order derivative of log P, which is a hashian. This would be my new Matrix. This would be this will.

This will end up being what I'm conditioning with. Um.

This second order derivative. Of of a loss? You can. You can always use. But, am I expecting this here, or am I just kind of? No, I'm not. You can always rewrite this so you can use the layout architecture. And you can see that this this this, because because you know the loss.

Is the loss function applied to the output of your model? You can rewrite it as some of these terms. This is basically just using the chain rule, so you say, like? So I have? Um, the the first derivative of the loss is the derivative, you know, because of the chain rule is the derivative of the loss with respect to the output times the derivative of the output with respect to Theta.

That's just the chain rule, right? That's just writing those two terms. And now, when I'm taking derivative of this again, to get the second derivative, I have the product rule. So, because of the product rule derivative of the product of these two matrices becomes this term class this term.

And this is kind of like a scheme that's being used often and in the second order derivatives and the reason for using this scheme is because the way people usually reason about this is, let's say, I'm gonna drop this Matrix. And the reason I'm going to drop this Matrix is because when I'm close to convergence, this loss is going to be close to zero, so this time is going to be close to zero, so it doesn't matter.

And I'm gonna keep this this Matrix. And this Matrix is usually called the gauss Newton approximation of the Hessian. If everyone heard the term. And the nice thing about this term is that it has this form of Jacobian transpose Jacobian. And you know that when you have this form, this has to be positive definite.

So, this is how people usually get rid of negative eigenvalues. And it turns out that when you do natural gradient. Okay, so I know I put these comments because I was going to derive all of this on the board, but I kind of directed in India, but the whole point here is that.

If I'm looking at this? So I have? This delivered the second derivative. Of. P. Of Z Pizza. So, this is the term that I'm our expectation over. Of this. So I can? Without working too much into it, I can say. That this can be approximated by. Derivative of log.

Times derivative of log B. And there is a transpose somewhere. Uh, I'll put it here. I'm not sure if it's the right place. But basically, you can do the expansion, and you get the distant things. Like, what about the Delta Theta? Sorry, what about the Delta Theta, so the Delta Theta will come in front of this.

So for now, I just took the the this and I wanted to expand. So, the reason I'm trying to do this is because I want to prove well. Uh, I can. I, I'll I'll send to everyone. I'll I'll update the slide with a link. Actually, if you look up this paper that I'm author of the revisiting natural gradient itself.

In the paper, you have this step by step the math in the appendix. If you really want to go through it, so you have the step by step of how you go through there to that, but that for here. Like, I just want to give you the impatient, so the the big change of natural gradient, so you you replace this with this functional distance, and at the end of it, when you take the Taylor expansion?

You end up with a Hessian again. So, one question you could have is like, what have I solved? I had a Hessian before. I did not like it because it had negative curvature. I went through all of this exercise to change it to this distance and whatnot. And in the end, I got another Hessian.

So, is it good that for anything? So the trick, the tricky bit is, you can? Try to expand that that Hessian and you can exploit that. This is an expectation of that Hessian, and it turns out that if you do that, you get that thing is equivalent. To an expectation over the outer products of gradients.

So, the advantage of this form? Is that this is positive definite Pi construction or at least semi-positive definite? It doesn't have to be positive, definitely, but it's at least any positive, definitely by construction, so. Yeah, you can just take my word for it, or you can look at any kind of linear algebra book, but I think the any, any Matrix that has the form a, a transpose, or a transpose.

A I don't know what the right order is. It's positive, definitely. By construction, you can prove that. So, to make your understanding. And then I solved the problem of the neonatos having a very large negative eigenvalues, exactly using this. Yes, because this Matrix does not have negative eigen values.

So, even though in the end, I end up with a hashion. This action does not need recognization, so this plot here is trying to illustrate what's going on. So, the red surface I should have make it bigger. The red surface is the loss function that you are navigating. So, what I'm trying to show here is that once I've plugged in that tile and work it out is, I have a different surface.

And that surface is always quadratic. It doesn't have a saddle and what I'm doing is I'm I'm trying to take a decent step on the red surface using the curvature of the green one. That's basically what we're doing when you're doing natural gradient. We're creating this additional function, which is the Kyle, and we're just taking the curvature of the Kyle and apply it to the function we cared about, and this makes sense because what we're really doing is solving this constraint optimization?

Well, what we're saying is we're solving our original problem, like the first order approximation of the problem, where we constraining how much the distribution is changing in a Kyle sense, and by doing that. We really end up with this algorithm. That it's a precondition HTD, so it has this form where this preconditioner ends up being the outer product of gradients.

So, then it doesn't have any negative curvature. That is the big thing of. I'm natural gradient. Um. Yeah, so I'm not going to work on the board because I feel like we've seen too much math anyway, and I'm worried that it's just gonna look like numbers, but the additional thing about it is, so you have this particular derivation of natural gradient, which says, if I constrain the Kyle and I do a bunch of math, I end up with this preconditioner.

That is just the outer part of Radiance. And then you have this derivation, and this is not natural gradient. This is. Traditional second order methods where people in secondary methods they use this gauss Newton approximation. So, this is a different derivation where it says the Hessian of my loss that I care about is equal with the.

Outer product of the gradients from the output to the input. Times the session of the loss. Plus, the gradient of the loss times the Hessian of the of the outputs. So, this is. This is the sort of a big thing in the optimization world. This is called the Gauss Newton and the reason I wanted to bring.

I mean, there are many reasons I probably had this slide, but maybe one reason I wanted to bring this out is because it's kind of interesting, sort of, uh. Maybe just add a bit of History to make it a bit more fun. It's how these things evolved. So, so am I did natural gradient early on?

But then people weren't actually using it. So for a long time, people were kind of ignoring it. Then there is a paper from Joshua. That is claiming to do natural gradient, but it's actually not doing natural gradient. So, in that paper, they messed up the the preconditioner, and they use a different Matrix.

And, and then sort of. In parallel, you have folks like James Martin's that were using this gaussians and approximation, and they were claiming that Gauss Newton works better than natural gradient because it is connected to the Hessian and whatnot. And as sort of later on. There is a paper that actually shows that gross Newton, a natural gradient, is the same thing because it turns out that this Matrix is exactly the second order derivative of the chiral.

And this is always true if you. Matching activation function analysis. So, if you pick like a reasonable loss and activation function, which you typically do, even if you don't know about it. So, like if you have a negative log likelihood and sort of soft Max or whatnot, or if you have mean Square error and and linear output, then this gaussian approximation is exactly the same as Amari's Fisher.

And all of this. So, the reason why this is important is because it tells you, besides, sort of the motivation that we had here. It tells you why this makes sense. This makes sense because this turns out to be an approximation of the Hessian of your true loss. Because it turns out that this Kyle.

That Thailand, which is exactly. This Matrix is related to the loss of your function. By this term, which typically is seen as being zero.

I think I lost half of you, but the whole story was really about. Uh, natural gradient is usually presented as a preconditioner, because it's not about using the Hessian of your losses, but it's about using a different Matrix, but it turns out that this Matrix has a lot of relationship to the tragesian, so this is sort of the whole point I'm trying to to get to you is that this other products of Radiance?

It's for for very for various reasons. It's a very good proxy of your curvature. Um. And the reason. This is interesting, because if you squinted it, this is very close to the formula that Adam uses. So, Adam takes gradient Square. And we know that radial Square. Now, if you assume here that every Matrix is diagonal, which is not, then this will be just gradient square, and this is sort of a proxy of that.

So, so this is kind of just telling you that you can approximate the curvature of a function by just looking at the square of the gradients. Do you expect this part? Yes, correct words. Adam is a special case of this. So? That's what I'm trying to convince you. They are connected that there is some glitches on the way.

But yes, you can think of Adam and this particular paper if you want sort of a much more formal way about it this. Bayesian learning rule from NTS, Khan, and others are really trying to pin this down and trying to argue that Adam is just a diagonal version of natural gradient.

So, this is sort of. The whole, the whole spill of that paper. It's it's, it's a bit more complicated, um, sorry, I think. Uh, it's a little bit more complicated for a few reasons. Okay, let me let me go why it's a bit more complicated, so? Um, the first reason.

That is a bit more complicated. Is that in the gauss neutral approximation and in the natural gradient? Uh, what you do when you're Computing this Matrix is you have your gradient from the output to your Theta. So, if your function is, so it's from y to Theta, right? This is not from the loss to B guys from y to Theta.

Um. In Adam, you're using the gradients you're using to do your optimization. So, using the gradients that started the loss and go to Theta, you don't use the gradients that started Y and go to Theta. Um, in the literature. This thing is called the empirical fissure, so you have the fissure, which is what Amari introduced and the empirical fissure, which starts from the loss instead of starting from Y.

And reason why the literature around natural gradient. If you ever find this topic interesting and decide, looking at the papers is a bit messy. It's because people confuse these mattresses all the time. Like, you have papers that messed up empirical feature with the proficient and replace one for the other.

They're not the same object. They're different mathematical objects, but people use them interchangeable. So, a first step to go from natural gradient to Adam is to replace the true fissure by the empirical fissure. Um, the other thing that makes Adam a bit different is that you take the square root.

And and and the square root thing is a little bit of. No one really knows, but it works really well kind of thing, so. There is different ways to argue for it. So, one way to argue for it is if you take a matrix. Any approximate that Matrix by the diagonal, so you remove all the of diagonal elements, and you just look at the diagonal then.

Usually, if you look at the eigenvalues of the diagonal versus the eigen values of the full Matrix, you're usually overestimating. The eigen values and taking the square root is you're basically pushing them back down the correct that over estimation. But it's this is a very handy argument, but that's something people use.

Another argument that maybe some people would like is if you look at the formula for Adam. And you pick units for the different quantities in there. It turns out if you don't have the square root, the units don't work. I don't know if you guys know, but in physics, people do this all the time.

If you want to figure out if you have the right formula for something, you just plug in the units and you check that the units come out the right way, right? If the units don't work out, you've messed up the formula. It's not the right one, so it turns out that if you, if you, if you, if you, if you assume like this quantity is a unit like, you need a square root to make the units work.

It's another argument that I've seen for that square root, but really that the answer is that the square root really helps quite a bit and and the reason. And okay, let me give you another better reason of why you have to start it. Yeah, another reason is because you don't use the true feature you use the empirical fissure.

So, what is the difference between these things, so the thing the difference between these things is when you are convergence. When the loss is zero, these times becomes zero, and the true fissure, which is this. This function becomes your hash shape. It's the same as the Hashion, so that's fine.

But if you look at the empirical fissure, the empirical fissure is the outer product of the gradients of the loss with respect to Theta, but if you had convergence the gradient of the loss with respect to Theta is zero. So if you square this thing, you get zero. So, if you look at that formula where you have Grady and over radian squared.

As radians go to zero, that thing goes to zero even faster. Because you're defining high gradient zero. So, what that means is that Adam becomes unstable when you're close to convergence if you don't have the square root. Because if you're looking at that limit, as the gradient goes to zero.

Like that thing, explodes goes to Infinity because you have G over G squared. And the skirtle solves this. Doesn't. Well, it makes it better, which is, which is good enough. It seems for convergence. Because the the rate at which G over G squared goes to. Um, I guess in in this case would be like plus infinity because it'd be like one over GSG goes to zero.

Like, it doesn't go the same at the same speed when you have G over G squared. Yeah, this kind of stays constant, right? Uh, because if you have G over square root of G, squared is equally one more or less up to some variations. So, so this makes things stay a lot more stable.

So, these are being the kind of arguments that I've seen so. Okay, I'm I'm gonna push a little bit through this, so I, I can appreciate this part of the of the live channels. It's a bit kind of heavy. Optimization ends up being that way. I wanted to make another note that is kind of useful.

Which is, we did all of this exercise because we are worried about negative eigenvalues. But one thing that turns out is that you still need to regularize the fissure idea. The Fisher is this Matrix that we've been Computing. This other products are playing with. And the reason you still need to regularize it with the identity.

Is because you, it's semi-positive, definite, so you don't have negative eigenvalues, but you have eigen values that equal zero. And that is still a problem. Because that means you can't compute the inverse. Uh, so in practice, you still regularize this Matrix to get rid of the zero eigenvalues, but at least you know that the only thing you need to do is to correct for zeros.

So you can add like a small Epsilon just to make it invertible without in. Well, I don't really waited by something small. Yeah. Anything, uh, positive? Well, you need to be careful because, like, um. There is this, I mean, you have the condition number, right, which is the smallest I can the, the largest second value divided by the smallest cycle values.

So, if the condition number is very large, so if you look at those, if you have the spectrum of eigenvalues, and you look at the extreme of it, like the largest and the smallest. If the gap between them is huge, then usually numerically. Things are not very stable, like when you try to do the inversion directly.

Things are not very stable, so. You know you, you don't just want to add 10 to the minus 12 and say, okay, I don't have any zeros. My motorcycle value is 10 to the minus 12, because then numerically, your inversion might still be unstable. So, you want to add something that's big enough so that the gap between the largest and the smallest is sort of reasonable, so that the inversion process is.

But this is more in the numerical stability, kind of space, and things like that. It's not anymore about dealing with negative curvature. It's really about trying to make your computation stable when you're trying to the inversion. Um. The other thing that I wanted to mention and sort of. When it comes to practical methods that are trying to do the second order methods, like one thing that they're they're doing, is that they're using a block structure and the reason they're using a block structure is because people have been arguing that the off the blocks that correspond, so block structure.

What does it mean? Is that your computer Hessian per layer? So you don't look at how a layer affects a different layer. You just say, I'm going to compute the hashion for each layer independently. And you know if I'm looking at my whole Hashion? It means I assume all of these blocks are zero.

And I only have non-zero elements in this blocks on the diagonal. Um, and this is just a trick to save memory and compute. And, and the reason is because people have argued that these blocks tend to be very small in norm and. Locks here have, like, so this is just another.

And I'm gonna skip that slide because it's too much math. Um. So, this is sort of a high level, so we, we've done all of this. But then? Um, this is at a high level. Yeah, so this is sort of what natural gradient is about. This is sort of what's economic methods are about, so it's Computing these sessions and it's kind of dealing with negative curvature.

But then, like when it comes to? Training models for? Standards scenarios. Um, it turns out that. Keeping things cheap is crucial to make them work. So, maybe to repeat something I was saying in the break is all of these methods can be very useful when you have. Domains, in which the low surface is well-behaved and the example I gave was, for example, you're trying to simulate the navier stocks formula or something like that you have some neural network that is trying to predict sort of the evolution of this system.

Um, this is sort of mathematically if you look at the loss of trying to predict this. This is going to be something that looks very ugly, so that's a place where you want to use this powerful optimizers. But if you what you want to do is to learn to classify imagenet?

Or you try to do language modeling, which maybe many of you are thinking about, then it turns out that this is not as important, it turns out that actually, the laws behaves quite well and usually. Usually, the outcome is that keeping the updates cheap is the best way to go about it and then sort of.

This is sort of. What Adam and momentum does Adam and momentum you can think of them as some degree? They're sort of a very crude approximation of all of this business about curvature and whatnot. But they're extremely cheap. And because of that, they dominate. They work so much well, in, in almost all standard scenarios, they they work well, there is another sort of hypothesis in the air that people have been talking about it for a while, which is.

Especially for modern architectures. We've been developing them to work for Adam, so there is this formal. I don't know how to call it this fear of. Um, what we have done with Transformer and other architecture like resnets. We basically tuned the architecture to the optimizer, and if we were to be using a much more powerful Optimizer.

We might have found much more interesting architectures, but like, for us, like these things are intertwined, right. Like, whatever, I propose, a new architecture I run into if Adam, and if it doesn't work well for Adam, I assume that new architecture is not a good idea, and I'm not even going to publish the paper so that that itself is like, what?

Maybe drove things to this place where, like, everyone is using Adam? It's it's sort of this bias in the community, where in general researchers prefer to focus on the architecture, not on the optimizer. So, usually, they pick the default Optimizer and run everything with the default Optimizer. So, therefore, when they're developing architectures, they're finding those architectures that have well-behaved curvature so that Adam really, you actually wear to come.

So, this is just sort of all of the things before it's heavy math. And it looks scary, and it's interesting. But in the end. Like, in most cases, you're probably never gonna end up using that. You're gonna just end up using the simple thing, and the simple thing is is something like RMS proper Adam.

And here, I mean, maybe the momentum should have been come first and then talk about that. But here, basically, you have two terms, right? The first one is the momentum term, where instead of um? Instead of using your gradient. You're using a moving average of your gradient? So that does multiple things, uh.

One thing that it does is it reduces noise. Um, and it sort of like shrinking your gradients in Direction. That way, you have high curvature where the gradient changes directions all the time. Um, but also this is not a noise, and it becomes an approximation or a better approximation of the true gradient.

So, instead of like overfitting on the current mini patch. Now you have this moving average of all the previous mini batches, so it's harder for you to kind of be super biased by the current battery idea. And the other thing that you're doing is you're Computing this moving average of the squared gradients.

And one way of thinking about this is the square gradients is a measure of curvature, and this comes from all of these empirical fissure, natural gradient, and so forth, where you can show that. You can always write your Hessian as the sum of other products of radians, plus another term.

And that term usually disappears. So, by the way, I did not say this explicitly. But when you look at this formula, you see that there is a hashank here in the middle. This session, for most losses, is a constant. So, that's why I ignore it. So, this formula, the gauss Newton, is really just the outer product of gradients this session.

It's here mathematically. But like, if you write it down for existing losses, it turns out to be a constant or something that doesn't matter. So, so you know, so that's kind of the connection with, like the gradient square is part of the proxy of your. Um, approxy, for your for your curvature, and then you take a step that is basically.

The momentum divided by the square root of this and, and this is sort of what you're gonna. Is in practice, and you have this Epsilon. Which is this regularization to deal with zeros? We are taking talking talking about how Fisher has zero eigenvalues because it's the same positive definite like this.

SD can also have very small numbers, and you don't want to divide by zero. So, this Epsilon, it's a regular for that. It acts as exactly the same role, so that's the point. Usually, that Epsilon is super small. I know you guys did a course on RL. Uh, so one hint that I have is that in around, it turns out that it's very useful to make that Epsilon very high.

Uh, the reasons is not that well understood, but the difference between supervised learning and RL is that in a row if you do anagram. It's actually quite useful to hypertune Epsilon and to make it larger than normal. Um, but otherwise, this is what it is, and this is, as I said, I kind of replaced UI type.

This is the momentum this is. This is kind of the top part of Adam. And really like the intuition of momentum and how it connects. The curvature is in this picture, right? Like, if you have a direction of high temperature and direction of low curvature. You're gonna get sort of this zigzagging behavior, like.

Now, if you compute the momentum, you'll notice that in the direction where the gradients agree, you're gonna accelerate, and in the direction where the gradients disagree. When you're doing the moving average. You're gonna squish the magnitude in that direction, so you undo the blue curve that it's a lot more aligned to the to the curvature, so this is highly of intuitively how the momentum works, and it's sort of.

There are different ways of deriving it. Another way to deriving it is to take sort of inspiration from physics, and that's why it's also called momentum, and it's really sort of. Like, if you have a ball and you let it go on a Surface, you know, you get sort of a similar kind of effect.

Um there as well. Um. The last bit that I kind of alluded. And after this, when you're gonna take. A short break and then go a little bit into generalization part. So, another thing that I've alluded for, like, so far, we've been talking about gradient descent, which means we compute a gradient on the entire data set for the entire loss.

In practice, that is not possible. So, in practice, what we do is, we estimate the gradient by using a mini batch. And the way this works is we basically need to make an assumption that our data points are ID, so they are distributed according to the same distribution, and they're sampled independently.

And if you make the distribution, we know that the expectation over this is equivalent, or it's approximated by just picking a few samples and and doing the average of samples. And this is sort of the difference. What people say you should expect is if this is greater than design that social classic idea should look right.

It's moving in the right direction, but it has a bit more noise, um. The sun has been mostly introduced as a scheme to make things scalable. So, like in the early days? Everyone, so that radiant decided the correct thing to do, but we can't afford it. So let's do stochastic, and it's a proxy for it in more recent work, and we're going to talk after the break about that.

It turns out that actually stochastic gradient works better than gradient descent. So, it's not about computation efficiency anymore. It's like, it's the better thing to do. You get better results out of it. Um, and, and this is just sort of. In terms of gaming convention, so toxic gradient means your batch size equals one.

Means you have the entire data set mini basebate and the scientist you fix a minute by size. So, you split your data set into groups of whatever Trend. 10, 20 times 56 or whatnot. Um. And then maybe go to this slide and the other very important thing is. Even with all of these measures of curvature and whatnot.

It still turns out that the best thing to do is to adapt learning, right. Learning rate is not a constant somehow because of the approximations we make, and all of that, like these things, will not tell you the right thing. So, by far, you cannot get the right results.

By far, maybe one of the most important thing is to have a learning rate, Decay, or learning grade schedule. This is a very old-school learning race schedule. This is how you used to do it where you started the constant learning rate, and either you have magic numbers when you divide the learning grade by something or you look at the trading loss or validation loss, and if that one doesn't decreases, then you divide the learning grade as a number.

This is how you used to do this, and I used to call this waterfall learning rate schemes. Nowadays, you have this. Like? A linear warm-up, and then sort of exponential decay of your learning rate. And this is kind of the standard, and usually what you hypertune is sort of the highest learning rate that you start from.

And, and then this game's yeah tend to work very well, so this is just sort of like the standard, uh, I don't have a lot of slides about why and how, but if anyone is interested, I'm happy to talk about that. And here, I'm gonna stop here for a break Midway for like 10 minutes, and then we're gonna continue for the last hour to talk a little bit about generalization.

But. And this is. For a long time. This has been the main perspective on learning neural networks. It's just they're non-convex. They have this very complicated loss surfaces. Therefore, it's impossible to train them to do anything useful, because anything you do, you're gonna get into a local in a local space.

And it got to the point, so I was telling you at the beginning of 2000 that they were like, maybe like three or four groups that were still doing here on that tracks, and everyone was doing svms and other things. So, in svms, like, you have convexity. You have, you know, you have guarantees that things are gonna work out.

So, really, what was going on is like, everyone was worried about this kind of issues, and then he had a bunch of people who are, like, well, you know, maybe it works. It's like, you know, it's. It's not great, not terrible. Like, let's go through it. Miracle of what happened is these people were playing around with things.

Obviously, they got some things right as well to make it work. But then they started seeing consistent behavior, so you actually have a paper it's not cited here. But there's a paper from Demetriarchan and Joshua banjo. Important, but I keep anyone wants to see it, where they actually look at this systematically, right?

They do like sample many random seeds, many starting points, and show that there's like the behavior of the neural network is consistent. And all of this kind of stuff, it's just because, back in the day, this was a big theme, right? This was a big reason why people did not like neural networks because they believe this.

Like, it's all about luck, and there's nothing systematic going on there. So this is sort of where the problem of generalization starts right. Like, you have this loss, like, sure, you can optimize on it and get somewhere. That is better than when you started, but is it anywhere meaningful, and how do you consistently go to somewhere?

That is really good, um?

Is, if you have a neural network and you have some data, you throw the data. The neural network of Adam and things are going to work out, and you don't need to worry about anything. And if it doesn't work, just make the neural network bigger, and it's going to work.

And that's it. Another way of framing it is that people now are viewing the optimization problem in neural networks as being almost comebacks. Maybe it's technically not convex by the behaviors if it's convex? And this is on the back of. A bunch of words that these are different plots here from the different works.

I'm gonna try to explain so historically the way things have happened is after some changes, including maybe switching to relu and some changes to initialization and some changes to SGD that kind of happen in the in the background, and then some stuff with the three, um, layer wise, pre-training that the Jeff and Joshua were doing.

Have started to work consistently. And there was more and more results, you know, like the image, net results, and so forth that people could reproduce and and or get these good numbers. And then you started getting a series of papers that were trying to check what's going on. And these are generally empirical papers.

Sample this paper here. This is the paper from Oriole. Uh, you're a good fellow, and Andrew sucks. And what they do is they take the starting point, or you start Theta zero, they take the point where you end up with convergence, and they interpolate on that line. So, this is a line in the parameter space.

They interpolate verifiedly, and they compute the loss at every point on that line. And they show that the loss is monotonically decreasing. Point is if I walk on a straight line from what I started to, I converge. There is no wall. There is no weird shape of my lost surface.

It's just something that's monotonically decreasing, like in a convex case. Done this on a bunch of examples, and they showed this. And you know? From now, this hypothesis that look things looks almost comebacks. If you want to go on, the more methy heavy side. This is the much methy heavy side, and this is actually a pretty cool paper.

I really like this paper by Yando fan. I'm an author so I'm biased. There's another paper by Andy Coon that says exactly the same thing, and people usually cite the animal controls paper. So this is right on my paper here, but this is a different take. So this is.

Statistical physics. They come up with this Theory looking at. Um, gaussian. What what? What are those, uh random, gaussian Fields? Which I do not know what they are, but there is a there's. There's something that's called random gaussian field that's been studied in statistical physics, and they notice that they have this very interesting property that as you expand the dimensionality of the space in which this random gaussian field exists.

Have this, this property that the error that you can get and the index they become strongly correlated. So, what does that mean because there are, this sounds a bit heavy. So, what that means is that the lower the error? The lower the index and the index is the number of negative eigen values, so the lower the error, the less negative eigenvalues you have.

So, in basically, if they, if this correlation is very strong, like, is shown in this picture as the direction that the space grows, then all the. All the? Um, um. Fixed points. That have? Zero negative eigenvalues. They also have very low error. So, let me rephrase this, and this is sort of kind of the theme of where we are now in thinking about the problem.

So, what that means is that as you blow up the size of the model? Your, your all your local Minima will basically have the same error, so they're all going to be very similar to your Global Minima. What you're gonna get is an exponential number of Saddles, and that's why we talked about sellers before.

So, the intuition right now is that you get sort of. The only thing you need to worry about is settle points. You don't need to worry about minimize, because, yeah, I don't think I get the intuition. Why all my local Minima become like there are between them and the development and becomes small?

Why they're all kind of go to the bottom? Yeah. Uh, so.

Yeah, so, so, okay. So, the way this paper works is really taking these results from physics. And here, there's some like proof of this, because the particular kind of thing. And then they, they try to connect this random gaussian Fields, or if both machines and connect both machines and your networks, and then that way, they say, oh, this have to behave the same.

And then they empirically compute this kind of plot that shows the same kind of strong correlation. The intuition of why this should happen. It's a little bit kind of hard to say, like the the usual framing. So this is this is something young, like, who used to like to say is that if you're in, like, if you're in a high dimensional space, the probability that all direction will point up is very low, so as you increase the space, you're always going to find a way of escaping anything that is bad.

That's sort of the way. How, how he is framing it, but really, I think what is happening is, um.

Something weird that's happening with distances where things becomes closer to each other or more equally distanced from each other, and somehow that helps you navigate that space a bit better. It is not clear to me what is going on either, and that's why I call this a myth because we don't necessarily have any kind of.

Grounding of why this has to happen. All I can say is that basically has been observed a lot and and people have realized or or started, kind of connecting the dots and starting saying, like, really, it seems in a standard training regime if I'm trying to. This is another thing is, like.

If you train, or if you stop through the training and you try to validate your Hessian. And you try to see whether it has negative eigenvalues. You'll find that it always has negative eigenvalues. And the thing is, if it has negative eigen values, it means that there is a way to escape to go lower because it means that, at best, you're in a saddle.

So, these are just things that people have observed practically. And, and this is sort of. The the background in which we we are right now. That doesn't mean that Allah surface is not yet behaved, and this is just sort of a few examples of how things go wrong, and I'm going to try to explain it.

So in this first paper, this is sort of more of a joke paper, but it's pretty kind of funny. What is trying to say is that if you give me an image so you have the Christmas tree image, I can find a place in the Lost surface where the loss looks exactly like that image, so there is a Subspace in the loss that has that shape.

So, the point of this paper is trying to say that. If you want in a Subspace of the Lost surface. So the low surface is arbitrarily ugly, but the only trick is that it arbitrarily ugly, far away from zero. So, I mean, there is some like layers here. So, the usual, maybe let me?

This. This deep learning? It's the way it exists and the reason it exists. Is that because everyone who's doing this experiment follows the pipeline there is? There is a protocol of how you train this model, right? There is a scheme to initialize them. There is a way to do the to the garden and so forth.

So the point here is that? If you go far away from the standard utilization, you get into trouble. And one way to get into trouble is, you can have this kind of funky theoretical work that I'm trying to say that. Like, look, you can find any shape you want in the surface, or sort of, maybe even more trivial stuff is.

You can Define your training set in such a way that all your relus. If this is a relevo model, this is what this paper is trying to do. You can, you can buy by, play with your data points of how you pick your data points. You can find the data set such that, for one layer, all your values are zero, and then there's no lining happening.

And there's a bell local Minima because you're random.

Here, which is, there is a way of initializing the model. To make it have zero training error. But be chance on your validation set? So you can completely break sort of disability of the model to generalize by just playing with the initialization. But the core idea, the intuition I'm trying to say here, is that things look almost kind of convex empirically and and things look well defined, and they're sub theoretical reasons that are a bit hand wavy in the sense that there's some very strong mathematical results in statistical physics and understand, like.

On networks that people have been employing. But all of these things look good as long as you follow as long as you do things properly. And by properly, I mean, you have proper initialization and you use sort of a standard Optimizer and you have normalized your data and your data.

You know, there's no battle. Into your data so? As long as you're following the protocol and you're doing sort of normal lining, things look good as soon as you move far away. Things look bad. There is this funky paper. This is just for people who, like, pumpkin. So, this is from.

We have Belkin, who also did double descent, um. Something that, like it's to me, is kind of surprising. In the end, it doesn't mean anything, but it's kind of surprising. So, the three quivarello Network is I can multiply with alpha one layer and one over Alpha. The other layer and I have the same function.

And that's just because if Alpha is positive, it's not going to change the sign. So, the relative doesn't matter. So, really, I have W1 times, Alpha times, 1 over Alpha times, W2. So that becomes W 1 and U2.

This means that the space is always curved because this one of our Alpha is sort of curve. So, if I have a global Minima, your Global Minima is never going to be a point. It's going to be a region, and then region is always curved, because that region has to follow this one over Alpha, because if I have a point on in on, that is a global Minima or a local Minima.

I can construct another local Minima by just multiplying with 1 over Alpha and Alpha the layers below so I can construct this curve. So, what that means is that, and this is sort of the the punch line of the paper you have a look at Minima and Zoom. No matter how much you zoom in, this always going to be curved, and it's always going to have negative curvature.

Attendant attention that you make is that things are locally convex. Therefore, I can use gradient descent and all this stuff. So basically, all of this light. It like details don't matter, but all of these slides are they're trying to say. Is that? Surface or neural networks are relatively complex as a mathematical object.

And and their papers that are trying to highlight this. But at the same time, as long as we use them following the standard recipe, they are extremely well behaved to a point where people don't worry about them anyway. So, um? We'll now try to go into some more reasoning of why things are well behaved and in some sort of the usual reasoning of where this generalization power comes through, not necessarily connected to those early work, but something that's a little bit more like Hands-On and makes a little bit more sense.

And to do that, I'm going to start from the standard kind of point where people start when they start talking about this. And usually, you know the way you would introduce this is you look at a plot like this, and you say, okay, I have some data points, and I'm trying to fit something to them.

I try to fit the line. It looks something like this. I try to fit quadratic. Maybe look something like this, and then I try to to fit a ninth order polynomial, and then it'll look something like this. And if I'm just looking at my training error and my application error?

Whatever. But then, if I'm looking at these plots, like which one seems more reasonable for the data that we have? Yes, probably this one, right? So, there is a sense in which this is our kind of the overfitting underfitting. So, there is a sense where, like, just driving training error down is not good, right?

There is a there is a point where, like, you're not doing the right thing and and and usually. What, um? What we care about in neural network and the way we kind of find this. When is the right time to stop not to get into the sort of nine order fit, it's usually this is done by the usual train test split.

So, so here, like, uh, I'm trying to say is that well what we really care about is to be able to generalize in domain more or less at the moment. Because we're talking about in domain and the way we can ensure that we're not losing that property is, we rely on statistical learning, which basically what that means is that we assume it takes some distribution Pi, and we've sample data from the distribution in ID fashion.

So, every time we want to compute this integral, like the laws that we really care about, the only thing that we have to do is we need to have some unbiased samples so that we can estimate the loss outside, and this is kind of like how things go interactive.

You have a bunch of samples, okay, calling the training set. These are the ones we're allowed to trade on, and then we have someone by example, so called the validation.

Starts increasing. It means things are bad. Um, and, and this underpins. It's basically the entire machine learning. I mean, it's not a big thing, but this is sort of the standard thing, right? Any anything that you do, you always have. You know, when you're dealing with data, you always or training, set the validation set, maybe a test set, you know.

You can do. Other things, but this is sort of the standard recipe, and you always train on training site and use validation set to to to see where you go. This is not the only choice, as I said. And this choice on its own is problematic, particularly in becoming problematic.

For llms, so foreign people now have a really hard time dealing with them, because what is really hard to know there is, whether your test set is included in your training set. A real problem, like with the size of the data that we have and the way the data is being collected and to the ability that we can understand the data.

This sort of? This. This concept of IID some, you know, same thing ID from the from the distribution kind of breaks down and actually. I think it's one of the big problems that that sort of this this the community has in this space and. I'm not sure how much people are actually thinking about it, but it's really harming to the point where.

Is becoming harder and harder to know. Whether a model is actually better or not than the previous one, like, you can get better metrics on different dimensions, but somehow the model still is not better because you're not measuring the right thing. And then this comes from this, uh, this, this, this.

This distributional assumption that it's hard to maintain, so the other choice. And I think I had a slide for it. I'm not gonna go too deep into it, but there's another choice that I think sounds kind of interesting. I think it has problems as well, but it's this choice is given by the minimum description line principle.

Solomon of induction and other funky things like that, but basically all it's saying is that? The model that compresses the data. The best is a model that will generalize. This is kind of the principle, so the idea there is if you look at the? Any beats you need to store the model?

And. How many beats you need to store the data, so you look at these two terms. Basically, this is how many bits the model takes, and this is how many bits the data takes given that model. The thing that minimizes this tool? That is. I was going to generalize the better, and the nice thing about this is if you have this prequential approach.

Is that you don't need a distribution, so maybe let me explain how this works the way, the way it works, particularly just looking at this term. Is that you take a single data point and you fit the best model you can with the data point. And then, you take two data points and you feed the best model you can to data points, and you keep going for every data point and you always fit the best model that you have, and this will generate a curve, which is almost like a training curve, but not really because you for any subset of data you fit, sort of the best you can.

And then you compute the area under this curve.

The thing that has the lowest area under the curve is the thing that's going to generalize better. There is another intuitive way. This has been framed, but it's a lot more vague, but maybe it's useful. And it's not actually in the paper from Yorkshire, which says they basically the thing that learns the faster.

Is the one that's going to generalize the batter? Because it has the right inductive biases when? Like, if, yeah. So, so, basically, the the the concept here is that you don't look just at the loss that you get on the training set at the end, but you look at how quickly you get there.

And if you get faster, that means that somehow you are kind of have the right structure, and you're exploiting the right structure to get there. Therefore, you're gonna generalize as you move, move faster. And yeah, because it may interesting and is better than the idea or spin Transformers, for example, because it has a better.

After some scaling. Yeah, so in some sense. Yes, and no, so three. To be fair, this is a binish. So, in some sense, I can. Yeah, like the CNN do have the right indoccupiers when it comes to images because they assume sort of this positional invariance that the Transformers do not, and therefore you expect that the Transformers.

A lot more. To learn sort of the the true bit of of, so it's almost like if you want, you know, we stopped. We talked something about strong generalization, like in cognitude, basically including something about the images that you know it has to be true, which is like, somehow locality is important when you process images and Transformers have to discover this by themselves.

So, in that sense?

Um, confidence will generalize out of domain instead of in a more meaningful way. When, when, when you have this stress, this concept of locality somehow. And if the Transformers don't get it, exactly right when they they don't learn this exactly right from the data. They will struggle a bit more now.

Empirically, I know everyone is switching to VIT and they tend to work better. I not necessarily a vision person, so I don't know exactly sort of the reasoning behind it, but I assume it's a mix of everyone wants to use Transformers because it's sort of very popular, and you have the right libraries.

And it's a question of also scaling like certain things scale better with, um.

Quite well as well. Which is, if you have two objects that are related to each other, but they're not present close to each other. The only way you can make the connection between them is if you go very high in a convolutional layer. So when I'm going to end up talking the promotionally, I'm gonna show you that.

But basically, the depths now controls sort of how much of the image you're actually see in kamazoo Unity seeing so if you want to connect two two dots that are apart like, here, you need to go pretty high in the area here, but the problem is as you go higher.

The hierarchy you're throwing away high frequency contact, so you're only looking at a sort of a smooth version of the image. So that makes it really hard to reason about, so if you have occlusions, for example, coconuts have a really hard time dealing with occlusions because they have a really hard time connecting objects that are not connected, especially because there's some some something in between them.

Transformers don't have this issue. So, Transformers in a single layer they can jump around because they don't have this.

That is useful. You could also argue that it's not exactly the ground trust indirectly bias, either because you have this kind of occlusion things, and then it works against you so. But, but in, in spirit, that's kind of what this preferential staff is trying to say, I think you had a question as well, but I'm not sure.

Not, not sure. Yeah, I'm just trying to link. The mdl principle with, like, because that's more about, like, Efficiency, right, and, and like, and, and the idea of, like, generalization, then, is it that the cons that the information or the? Is better. Now, if it's more efficient, so it's?

The gambial principle is not. It's something that looks. It's like an information theoretic kind of principle, so it's something that looks at the data. So it's not really about on which thing you're converging faster. It's really like the true definition of it is exactly the one that I said, where you independently for each subset of the data.

You train the best model you can with that data. So, so it's like, like the way. Okay, so the way this has been done in practice. So, like this, experiment at the bottom, so this is from your experience where it is one person that likes this a lot. He is really looking at the area under sort of the training because you can't afford to train sort of a resnet, like a million times.

For every new data point that you have, so you just kind of when you say you retrain for a new data. You just do a single HDD stuff, and that's it, and you start from the previous one. So, so, in that sense, like, I'm gonna show you, if this is even your question, but in that sense, like this principle, like the way it's been used in practice, has been bastardized to be which one trains faster, but really, really.

To test us, like, what is the cutoff of data points that you need to get low training error? And if you can get away with fewer data points. That means you have a better model, so that's kind of the conceptually what's going on, but this has been bastardized into this because this is the Practical bit.

So let me just check what the next slide is. Oh sorry, yeah, I feel it. I'm a bit lost. Yeah, between the two Concepts, like, I understand the second one, which I've been discussing now about the affirmation. How does it take to the faster convergence of the like, so there is no theoretical link?

As such, it is sort of how this original, so this original ndl concept, that is, I'm not mistaking the first paper might be from the 60s, but it's been developed a lot in the 90s and early 2000s. Marcus Hutter is a big name that like to work on this quite a bit.

The way this has been translated to actual day-to-day deep learning is by converting this sort of.

And. The the reason for that is because they were just arguing like a proxy of retraining on this new data set. Is, just take the previous solution that I have and take a single HDD step on a new data set, but there are some information based networks are, for example, if I want to evaluate a language model on some text.

Yeah, I can use the aplexity or text, very character, or whatever. Yeah, and ideally, if that plexity is very low, I can't save. This model can generalize well in this, like purpose or language. I mean, the complexity is the same as the laws. The perplexity is really the likelihood, just like.

In a different range, but it's really the same likelihood object that we have right here. So, what this is telling you is not just, you shouldn't just look at perplexity. That would be sort of the usual way of doing things. If you do training validation set, but it's to look at.

Which model needs the least amount of data to get the perplexity dial? This is kind of the concept that we're that this is going for, and as a matter of generalization, just to kind of connect it today to day. So, the reason I thought it's kind of useful to present this.

This is a bit of a niche thing. So, like, you're not gonna find this in traditional sort of deep learning courses. People are not going to talk about the prequential, but I, you know, just to say that, there is a reason I, I put this in the slides.

It doesn't require a distributional assumption. It, it seems I, I didn't talk about this experiment, but it seems it can figure out spurious correlation. That is to be seen, like, this is just one experiment and one one claim from one paper. But what is interesting, is that, uh, in large?

Language models in empirically at places like Google and Facebook and whatnot. People are actually using this thing without them knowing so. So, what's happening nowadays because of the scale and the computational cost, and I can tell you this from from my own experience is when they do the pre-training stage, they do not have an explicit validation set where they look at the perplexity for that.

Select models based on, I guess, in their mind, on the training perplexity, so you know the training things and they're looking at a collection of the training, but like, if you look at the code and what they're doing in practice again, this is for computational saving engineering kind of tricks, is they usually, you know, because you're doing back back propagation, you first do the forward you get your perplexity, and then you do your backwards and computer gradient and apply the update.

So basically, what they do in practice is they first evaluate on the new data point and then take the step on the data point, which in some sense is very close to what this is saying, and then the other thing that they do, because that point is noisy, is they do a moving average, which is kind of corresponds to looking at the area under the curve.

So, this is usually the metric that, in practice, big groups ends up doing when they have these big models that you know they take most of the computer they have and also the data they have. And, um.

Not exactly done correctly, but close enough in spirit and. This is sort of a claim that. Jorgen and Marcos Hutter by people I know. But there are demon. I work with them. This is a claim they would make. Obviously, they have a bone in the fight. Because preparation evaluation is the thing they came up with, but they're trying to argue that people are actually using it day-to-day.

Think all about the point, suggesting, oh, the the people. So, in large language models, when they have to train these big models, so there's usually in. Lamps. There is the pre-training stage, and then the post training stage. Again, I I don't like these names, but anyway, the pre-training stage is really.

Just have lots of data. Uh, and you train the model from scratch and the only thing you're looking at this perplexity. Post training is usually when you start doing. You know safety training, you're trying to remove hormone language you're trying to do. Rlhf, you do instruction tuning to make the system listen to instruction and all of this.

These are all like fancy names. What it really is post training. It's a bunch of stages of fine tuning on very dedicated data sets, or we very dedicated objectives, and it's in post training that they have like evaluations that are different than perplexity, right? They would they would use different kinds of ways of lighting the system.

Rather, they have qna kind of things, and they look at how well it answers the questions, or they have all kinds of other. They have human evaluations and all of that stuff that all usually happens all in first trade. But the pre-training, which is the bit that takes most of the time, like 99.9 percent of the compute and energy are not lost in the free training.

It's really just perplexity driven, and it's just like predicting the next open. You have lots and lots of data, and the model just predicts the next token.

Computer that you need to use is so big that people usually do not like the way you do. This is you run your training model. You would ask your your training model code to save the weights regularly, and you have another process that you spend on a plaster that would load up that model and try to evaluate it on some set right, and, and so you get some kind of sense of how your validation error goes up.

But like, what people have started doing recently recently, I mean, maybe the last three four years is they don't do this validation job anymore because it's too expensive anyway, and things are expensive. What they do is they actually look at the training error that's sort of their claim. And they say, I can, only I it's sufficient for me to look at the trading error, and that will tell me when to stop.

And if my mother is doing well, and if everything is fine, like I can use the training error instead of validation error.

That you can have. If you just talk some training together, but there are two differences that they do, that makes that not usual training error, so one is the way they compute. It is the exploit the fact that if you want to compute a gradient, you first have to go forward to get a loss so that you can back propagate, because that's how backdrop works.

So, therefore, they they first evaluate on the data point, and then they take the step. Not on data that you've already trained on. It's on the data that you're going to train on, so your computer loss before you train on it. So in that sense, you're not biased, right?

Because you first use the data to evaluate, and then you use the data to trade on. The second thing is they do this moving average, and the reason for that is because otherwise, things are too noisy. But when you do the moving average. It's almost like you're integrating this area under the third string.

Sort of the the it is link, but it's connected to it, you know, yes, why? Another device using this idea of predicting that the data point and doing the gradient step on it? Why would the idea just describe? Yeah, yeah, yeah. And then, I update my weights. Yeah, what?

Everyone is not using it, even if I'm doing a very small CNN about it, and this is the closing like 20 of my assets for validation. Yeah, yeah, I mean, usually, like, okay, so there's no good theoretical grounding for this. If you talk with these kind of people that work in this space, they would say it's working because we never have enough computes to overfit the the llm anyway, so it's fine.

And that's why we can do it. But like on small models like, you know, maybe you sometimes even do it, and you don't realize, but the point is smaller models. People just look at validation error. They don't ever look at the training area like, I mean, you look at the training error for pathological behaviors, but they just you have the validation error.

It's easy to compute most of the scripts you find online would automatically have Computing the validation error, so it's just a practice thing. I mean, I think sort of what, uh, York and Marcus Otter would like is for everyone to do frequential evaluation, and that no one to use the.

But the truth is, it's um.

First of all, really says you need to train the best model you can. For every data like, this is not what we're doing so, and there's no strong Theory to say that this proxy that we're doing is good enough. So, anyway, this is like, this is also new stuff.

Like, it's. It's one of those things like, maybe in some future, we'll end up doing this. I kind of doubt it because I think there is a big gap between the mdl principle and and the sort of trick that we're doing in practice. That might break things quite a bit.

Still kind of interesting to think about this this way. Um, there is also. Plus the time. There's also maybe I should mention it because I find this a bit interesting. Um. This also sounds a little bit like black magic, so if you, if you listen to Marcus Hatter, he really makes it sound like this is the answer to all problems like, you don't need distribution.

You don't need things, but there is a secret in mdl as well, and the secret in mdl is that that Ariana this is, according to me, is that that Ariana, the curb that you are measuring is only meaningful after you've seen sufficient data and what the sufficient data means.

No one knows, so if I have a process that is something like for a million steps via linear function. After a million steps, BF abstractive function that are generating my data like, until you get to the switching point.

To model your your thing, and then when you try to evaluate it as you go further, you'll see that it doesn't work anymore, because at some point it become fantastic. So really, there is this concept that, like, sure, the thing that compresses the most is the closest to the true program, because that's the most compressed form of the thing, but that is, if you've seen sufficient of the data generated by that process, some somehow covers the entire program.

Well defined in the ndl literature, right? So, in the ideal literature, they say, well, you can do this trick. And if you do it. It's fine, but I think there is a Dutcher there where, like, you can always get the wrong decision if you somehow do not go far enough in how much data you see.

Yes, yeah. But at the scale, you said, this is implemented in practice, then that that isn't really an issue. Yeah, for for the nlm stuff, that's not an issue, but I'm just saying this more of. The right way to do model selection, like, say for toy things that's not necessarily clear to people that there is a point where you can't really trust the system unless you've interacted with it enough.

And I think that's never. If you read newspapers, he never talks about that. He only says, I don't need a distribution, and this works, and it's sufficient to look at that end of the curve. And that's the only thing you need. So, how much time I think we have quarter 15 minutes, so I'll try to push for more more slides because I was hoping to be further away in my back?

But yeah, so okay. So this was a this was like a side note this whole frequential learning, but if we go more traditional learning? And this is maybe things you already know. This is kind of the picture you'll find in a book and the kind of thing people will tell you, which is, you usually have these three regimes when you're playing with the model capacity, which is one is you're under fitting.

You have a small model you're trying to train on the data. And then, in this scenario, the the validation loss, the test loss, and the train loss. They track each other, and they start going down. Model is just right. It has the right capacity. And that's when the the test and the train are the lowest.

And then, as you keep training, or as you make the model bigger? The training will keep going down. But the, but the test is going to go up. And this is when you're gonna start fitting the noise. So, this is exactly. This picture, right? So, this is when you're in a good fit, and if you keep increasing your amount of capacity, you're gonna get up this, right?

You're gonna get to to drive the training error even lower, but at the the expense of the of the validation. And, and how does this play for neural networks the way people think of? This is the capacity is the size of the mountain, right? So, of course, you have the linear model versus maybe a quadratic model, but then, in the neural network space, once you move to new natural space.

All neural networks are, you know, Universal approximators, but their expressivity. Their capacity is defined by the size of the waves, like how many ways you have, how many neurons you have. So usually, this is also intuition. You know, you need to pick the right size model for your problem.

Like, that's not too big. Does it relate to the double design Theory? Yes, this is exactly related to double descent, so this is sort of what an old school kind of introduction to to generalization will tell you, but that in practice, we know that this is incorrect. Is, we know that the bigger the model, the better.

And if you keep making it bigger, it's even better. And this is sort of where double descent comes in and maybe, oh, I had a different slide before that. So before I talk about double descent, I just want to say that you can control things by by the model size, but you also can control things by the amount of training steps.

The way to intuitively understand that is, even though you have a neural network that's infinitely wide. It's Universal approximator is the expression that you want. If you limit the number of steps of a GD step that you do, then you limit the number of functions you can reach, right?

Because you can't go through all the parameter space, you can only travel this far, so that's another way of limiting capacity. So, number of steps that you do is the same as limiting number of parameters. You can either make the model very big and limit the number of steps or you, you just have any many number of steps, so just attack.

Yeah, because I'm kind of like swapping between this. Uh, and, and maybe that's a bit confusing. And and, and, and another way of controlling capacity is through regularization. Sorry, I'm gonna get to double descent after this. Uh, and and regularization is really like if you have a loss that has multiple local minimas or multiple minimas.

You have a regularizer that has only one. Minima say, this is an L2, so it has a minimize zero when you sum these two surfaces together. You basically make this Minima have different values now. Because you know the regularizer will prioritize the Minima that's closer to zero. Away from zero to the other higher value to it.

So that's a way of thinking of, like how the regularizer solves the? Um, the problem is by prioritizing things that are closer to you. I'm gonna skip over that because it looks like a lot of mans. So this is just really showing, and this we talked about it before.

This is showing that, like if you're being proper Beijing. But if you're being proper patient from being proper, Beijing, you get basically a regularization term, which is your prior. So, I think we talked about this, so usually if you're being properly patient when you try to optimize things, you get that your objective is the negative log likelihood, so it's how well you fit the data.

Plus, how well you respect your prior, and if you pick your prior to be sorry. A gaussian Center zero that will end up being an L2, so that basically says that I prefer solutions that have small arms. So, so this is just sort of a probabilistic way of getting to the regularization turn, and it's basically sort of just a natural outcome of having a prior, um.

And um, this is kind of. Yeah, maybe there's not that much to say around here. I'll come back to this when I actually talk about contents, but just to tell you, there's we have many techniques to regularize all of this are constraining the problem. So here, now you have to learn that both these sleep versions of the image are a catch, so you have to learn more.

So, you use more of the capacity, so you know? Slide. Okay, there's quite a few slides before the double descent. Do I have time to control that much light? Jump to the double descent. Me, just try to go through all of them. I, they should be after this. You should be the double descent.

So, we're we're on the regularization point. There is another observation, so we talk, so these are more traditional ways, or at least sorry. This one is a super traditional way of regulating the models. This is sort of like textbook, I don't know 2000 way of regularizing the model. The there is other funkier ways of regularizing a model.

This idea was thrown out was from from your girl in 1997, but is this idea of flat, sharp minimum? So there is this effect that people have found it was very surprising for a long time. We were doing mini batches GD because we couldn't afford to do anything else and at some point gpus caught up.

There were extremely powerful, so people were like, let's just do full GDs because you know, now we can. And that's the right thing to do. And then, when you do full GD, it turns out that. Is my choice, and then sort of. There was this big debate of what's going on, like, why is HDD helpful?

And it turns out the the one of the hypotheses that is kind of sticking around. This turns out that the noise is helpful because it allows you to escape sharp minimum, so the intuition here is you have a loss like this where you have something that's very narrow, and then a flat Minima because of the noise in in.

In SGD, you cannot converge here. The noise will push you out, so the only only minimal you can converge to is the Minima that's wide enough that it's much wider than the variance of your noise.

More like mdl kind of principle for why this is useful. So the argument is the flat minimized a much more compressed. Like, you need less bits to describe it, but another way to think about it is if this is my training loss. I don't have a slide on this, but if this is my training loss, you can imagine that the test loss is just basically a noisy version of this is this, but like shifted a little bit and whatnot and the the thing here is for the Teslas.

If you shift a little bit here because this is narrow, you're already. Your loss is going to be very high, but because this is flat if you add any noise, the loss will kind of stay the same. So that's another intuitive way of understanding Wi-Fi system and and the the concept.

Here is the stochastic gradient descent.

Just a quick question. Yeah, sorry if you have time. Yeah, why? Most people are not utilizing this like they focus on regularization of learning, great schedule or whatever. But for example, like we don't see something like? Batch size schedule. Example I start with, like, very large to regularize the noise.

Exactly, yeah, why? Generally, training people don't mess around so much with flashlight. Just select effective value like, 64. Yeah, so I don't necessarily have a good answer. Like, my expectation is probably the engineering work that you have to do to do. That doesn't buy you, you know? It's basically not worth it for the improvements that you see, because I mean, depending on your setup.

Like, if you're playing with C5 or whatnot, probably there's no engineering work. But if you're playing with like a large data set. To serve it in different ways throughout and sort of how you're sampling and and things like that. There is another reason maybe correlated to that which is with the change of the batch size.

You need to change the learning rate. There is a Formula of going around of, like, what is the correction of the learning rate? If you change the bite size for that formula is not ground truth, so you know, then you can't really stop Midway to like return your learning rate because you change your bad size.

Um, and then, like, you know, if you're doing Adam or things like that, you might need to, like, I actually even haven't mentioned this. But in Adam, you have at least two hyper parameters, which is the the beta one and beta2. So these are the the moving average of your momentum and the moving average of your sub square gradients those might depend on the base size as well, and maybe you need to.

So, it's just sort of all of this side questions that people don't want to answer. So it's always just easier to have a fixed by size and tune it whether, like, like.

Which is sharpness away minimization, which is a take on SGD, which the main goal of Sam is to find flat Minima. So there are algorithms and that was quite used to some point. So there are variations of atom and so on. That are particularly framed because of this and trying to exploit this.

How much time do we have? Five minutes. I'm not gonna get to the double descend, but to link that for tomorrow, so I'm just gonna continue this. So I decided to say because this is actually a cool thing, so you have to slash everything. This is actually by far.

The intriguing hypothesis that is true in a lot of spaces and. It's like one thing that is kind of helping to explain a lot of the things that are going on. So, so I think if there is any kind of take home messages, you know, understanding the sharp, flat minima Behavior.

It's kind of important because detecting a lot of things. There is another framing of this, which I'm not going to go into the math of it, but uh, um. Is incorrectly they. They did this thing where they're looking at the updates that you're doing. And they work backwards from that.

To find a loss where if you would follow that loss, if you follow the the flow of that loss. So, if you describe personally, you get exactly the behavior of this and and the whole point of this. Okay, so the detail doesn't matter. The whole point of this is that they managed to identify an alternative loss.

Your normal updates are minimizing. And that alternative loss is your original loss, plus the regularization term on the norm of the gradients. And this regularization term on the normal. The gradients is basically trying to do the same thing. It's saying that the Minima you find it has to be flat, so it's coming up in many places.

And the interesting thing about this line of work is that GD has this form. Actually, you get a different form. And then this is kind of interesting, because now you can write down what is the implicit regularizer that GD is imposing, and what is the implicit regularized that SGD is imposing, and because they behave differently, you know, regularized that are different, and you can sort of understand where the difference is coming from.

So, the difference? Sorry, I kind of stiff over this weekly, but I'm gonna stop after this. Just penalize the norm of the full gradient from the entire data set. For HGD, you penalize the gradient on each of the mini batches. And then, if you're thinking about the space. Say, if you have some directions that cancel out in the averaging of the full gradient, you're not going to impose flatness in those directions.

While here, you're going to impose flatness more widely around you. So, that's why HGD works better is because it finds flatter minimum sort of all around. And the last part of the slide is that? While flatter is better. There are also some questions, so one, for example, you assume that about from this hypothesis that adding noise is useful because it helps you go flatter.

It turns out that it's only certain types of noise that are useful, like chat. Ocean noise is not going to help you. The optimization is going to go, so it's very frustrating, so it's. There is something definitely about noise, but it's also definitely about the kind of noise you get from stochastically sampling data points.

Like, for example, here we try to use noise from data augmentation and that hurt you so getting rid of that noise. Actually, you get better performance. Does this relate to the way you mentioned the Deep learning models perform better with natural data? Yeah, yeah, so here is like, we're basically trying to see.

Yeah, I mean, this is just one facets of that, but we're trying to see data augmentation, where we pick standard data augmentation for images. If that noise is better, it's also as good as as noise coming from sub-sampling, the data to kind of force you to generalize better, and the answer is not.

One such a Minima, but like the the nature of the noise that comes from stochastic gender data makes it flatter in the right way, and what that means we don't know. But it's not like any anything that you add on top of your HDD updates will help you. That noise.

Okay with this. I think I'll have to stop here because otherwise I'm gonna run over, and I think that's been quite a bit. Let me see. No, okay, so there's quite a few more slide. Oh, quite a few more sliders. Yeah, okay, so we'll, we'll catch up on that.

Tomorrow morning, I think, oh, thank you, thank you. Thank you.