Join the crewMy track

Classes / Systems class / Systems Department

Systems Track: Spacecraft Links and Flight Software Basics · Rank 5 cornerstone reading · about 90 min of reading

Cornerstone reading: Attention Is All You Need

The skill this builds. Read a cornerstone work in our field and apply its lessons to situations it does not spell out.

This paper introduced the Transformer, the architecture under today's large language models and many vision and robotics models. If you build, buy or operate AI systems, it explains what is actually happening inside them and why they scale on modern hardware.

What to read: Read the abstract and sections 1 through 4 (Introduction, Background, Model Architecture, Why Self-Attention). Skim sections 5 and 6 for the training setup and results.

Start with the problem the authors set out to fix: recurrent networks process a sentence one step at a time, which makes training slow and hard to parallelize. Then look at what they replaced recurrence with.

Work through Figure 2 and equation 1 slowly. Make sure you can explain in plain words what queries, keys and values are, why the dot products are scaled, and why the model uses several attention heads instead of one.

Table 1 is the heart of section 4. Compare self-attention, recurrent and convolutional layers on cost per layer, sequential steps and path length, and think about what each column means for long inputs.

The test has 10 questions. Half check what the work says; half ask us to apply it to a situation the work does not describe word for word. We need 8 of 10 to pass, and we can retake it after studying.

Sources

Module quiz

10 questions, 8 correct to pass. You'll see what you missed and why, and you can retake it.

Back to the track home →