FDE Times: Build a Large Language Model (From Scratch) — write a GPT so you can answer client latency and fine-tuning questions
A 6 November FDE Times review of Sebastian Raschka’s Manning book (368 pages, Sept 2024) says FDEs should retype chapters 2–4 for attention and KV cache intuition, then pick chapter 6 (classification) or 7 (instruction) by the client’s problem — not rush straight to fine-tuning.
FDE Times reviewed Sebastian Raschka’s Build a Large Language Model (From Scratch) (Manning, September 2024, ISBN 9781633437166) for Forward Deployed Engineers. The book will not make you a researcher; it helps you answer the client questions API-only experience leaves vague: why longer prompts cost more, when a KV cache helps, and whether the job is labelling or free-form generation.
The official rasbt/LLMs-from-scratch repo mirrors a working order: text data, attention, assembling a GPT, pretraining, then two fine-tuning paths, with LoRA in appendix E. Code is meant to run on an ordinary laptop. A rough estimate in the piece: for a 1,000-token prompt and 200 generated tokens, a KV cache cuts token passes from about 220,100 to 1,200 by not recomputing earlier keys and values — attention to stored keys still remains, and memory still grows with sequence length.
Chapter 6 (classification) suits ticket-sorting measured by test-set accuracy; chapter 7 (instruction) needs a scoring scheme before you claim improvement. The review warns against skimming and against jumping to instruction fine-tuning when the client only needed a labeller. Suggested reading order: chapters 2–4 closely, skim 5, then 6 or 7 by client mix, then LoRA — and put a measured fine-tune on GitHub so the learning shows up on a CV.
Source: fdetimes.net