RoPE

There are many good resources explaining the transformer architecture online, but Rotary Position Embedding (RoPE) is commonly poorly explained or skipped entirely. RoPE was first introduced within the paper RoFormer: Enhanced Transformer with Rotary Position...