The topic of Understanding Transformers Part 12: Building the Decoder Layers is currently the subject of lively debate — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
In the previous article, we just began with the concept of decoders in a transformer.
Just like before, we use the same sine and cosine curves to get positional values based on the embedding positions.
Since the <EOS> token is in the first position and has two embedding values, we take the corresponding positional values from the curves.
As a result, we get 2.70 and -0.34, which represent the <EOS> token after adding positional encoding.
Next, we add the self-attention layer so the decoder can keep track of relationships between output words.
Note that the weights used in the decoder’s self-attention (for queries, keys, and values) are different from those used in the encoder.
So far, we have seen how self-attention helps the transformer understand relationships within the output sentence.
However, for tasks like translation, the model also needs to understand relationships between the input sentence and the output sentence.
Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment’s permalink.
For further actions, you may consider blocking this person and/or reporting abuse
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
Why it matters
News like this often changes audience expectations and competitors’ plans.
When one player makes a move, others usually react — it is worth reading the event in context.
What to look out for next
The full picture will become clear in time, but the headline already shows the dynamics of the industry.
Further statements and user reactions will add to the story.