ch06 - 02_bonus_additional-experiments

[d-kleine]

I am wondering why the code for training only the last layer works, because it only passes the code:

for param in model.parameters():
        param.requires_grad = False

...

if args.trainable_layers == "last_layer":
    pass

...

I have set it more specifically, with the same results:

for param in model.parameters():
        param.requires_grad = False
...

if args.trainable_layers == "last_layer":
    # Get the last layer
    last_layer = list(model.children())[-1]

    # Make the last layer trainable
    for param in last_layer.parameters():
        param.requires_grad = True

...

---

What do you also think about adding test for the two last layers + two last transformer blocks, like from your article about finetuning LLMs:

[rasbt]

I am wondering why the code for training only the last layer works, because it only passes the code:

Oh that's simply because we first make all layers untrainable, and then we replace the last layer with nn.Linear, and nn.Linear is trainable by default. And that's because nn.Linear uses nn.Parameter, which has requires_grad=True by default.

[rasbt]

That's not a bad idea. I will put that on my list ... Right now, I need to finish chapter 7 (the deadline in 2.5 weeks 😅)

[rasbt]

After the last chapter is done and submitted, I plan to add way more things here and there to the repo

[d-kleine]

Alright, thanks! 👍🏻

[rasbt]

And no worries, I am planning to add bonus material to the repo for a long time to come. Have such a long list of interesting things :). Big thanks to you by the way for all the invaluable feedback! I'll make sure you get a big shoutout in the Acknowledgements section!

[d-kleine]

Sounds great, and thanks for the shoutout! Looking forward to it! 🙂

[rasbt]

That was a good suggestion, the performance of the last two blocks is quite good!

[d-kleine]

Thanks for the implementation - yeah, the performance is a little better! 👍🏻 🙂
I just have tested it, but there was an issue ("two_last_blocks" instead of "last_two_blocks"). Pushed fix via #171