Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Building makemore Part 5: Building a WaveNet

Andrej Karpathy
829K subscribers
<__slot-el>
<__slot-el>
217K views 2 years ago
We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...). Links:  …...more
...more

intro

0:00

starter code walkthrough

1:40

let’s fix the learning rate plot

6:56

pytorchifying our code: layers, containers, torch.nn, fun bugs

9:16

overview: WaveNet

17:11

dataset bump the context size to 8

19:33

re-running baseline code on block_size 8

19:55

implementing WaveNet

21:36

training the WaveNet: first pass

37:41

fixing batchnorm1d bug

38:50

re-training WaveNet with bug fix

45:21

scaling up our WaveNet

46:07

experimental harness

46:58

WaveNet but with “dilated causal convolutions”

47:44

torch.nn

51:34

the development process of building deep neural nets

52:28

going forward

54:17

improve on my loss! how far can we improve a WaveNet on this data?

55:26
Building makemore Part 5: Building a WaveNet
3.8KLikes
217,087Views
2022Nov 20
We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...). Links: Supplementary links: Chapters: intro 00:00:00 intro 00:01:40 starter code walkthrough 00:06:56 let’s fix the learning rate plot 00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugs implementing wavenet 00:17:11 overview: WaveNet 00:19:33 dataset bump the context size to 8 00:19:55 re-running baseline code on block_size 8 00:21:36 implementing WaveNet 00:37:41 training the WaveNet: first pass 00:38:50 fixing batchnorm1d bug 00:45:21 re-training WaveNet with bug fix 00:46:07 scaling up our WaveNet conclusions 00:46:58 experimental harness 00:47:44 WaveNet but with “dilated causal convolutions” 00:51:34 torch.nn 00:52:28 the development process of building deep neural nets 00:54:17 going forward 00:55:26 improve on my loss! how far can we improve a WaveNet on this data?

Follow along using the transcript.

Andrej Karpathy

829K subscribers