The topic of Understanding Transformers Part 5: Queries, Keys, and Similarity is currently the subject of lively debate — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
In the previous article, we explored the self-attention concept for transformers, in this article we will go deeper into how the comparisons are performed.
The first step is to multiply the position-encoded values for the word “Let’s” by a set of weights.
Next, we repeat the same process using a different set of weights, which gives us another value (for example, 3.7).
We do this twice because we started with two position-encoded values representing the word “Let’s”.
Now, we use these query values to measure similarity with other words, such as “go”.
To do this, we first create a new set of values for each word, similar to how we created the query values.
We will use these key values along with the query values to calculate how similar “Let’s” is to “go”.
Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment’s permalink.
For further actions, you may consider blocking this person and/or reporting abuse
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
Why it matters
News like this often changes audience expectations and competitors’ plans.
When one player makes a move, others usually react — it is worth reading the event in context.
What to look out for next
The full picture will become clear in time, but the headline already shows the dynamics of the industry.
Further statements and user reactions will add to the story.