Introduction to Few-shot Named Entity Recognition

Published 2021-10-07 22:56
Tagged on nlp transformer ner

 You might (or not) have seen that I have recently published a tiny library called FSNER. I am pretty much sure if are reading this article, most probably you have searched for something similar already. Well, at least I did. Today, I will explain the paper titled “Example-based Named Entity Recognition” by Morteza Ziyadi, Yuting Sun, Abhishek Goswami, Jade Huang, Weizhu Chen, which I implemented in the FSNER library. To identify entity spans in a new domain, it uses a train-free few-shot learning approach inspired by question-answering. I will start with the same example given by the authors of the paper. The sentence goes like following- <p style="text-align: center;">***I cannot play Minecraft with error code 0x111.***</p> And, we have three different types of entities (Game, Error Code, Device) and their corresponding support examples. You might have already figured that there is no such “Device” type entity in the example sentence. In the following table, you will see what is expected behavior in the case of an entity not being available in the sentence.  What it does is, basically, try to find sentence level similarity and token level similarities to predict the start token and the end token of an entity. When there is no similarity with the support examples for a certain entity, it just points to the [CLS] token as the start token and end token. This is achieved through passing negative examples during training, i.e., entities that are not available in the sentence. I mentioned that the recognition is done via finding sentence and token level similarities. I will explain how is that accomplished.  At first, let's take one support example and a query. From the above figure, we can see that we pass our example and query sentences into the BERT model and capture embeddings (T_CLS, T_1, T_2) of all tokens. Now, for the support example, we embedded the target entity with two special tokens: “<e>” and “</e>”, and that's how we already know which ones are the start token and end token. We take dot products of the start token's embedding and all query tokens' embeddings. We do the same for the end token and end up with the scores of each token being the start and end tokens. As you can see from the red vertical bars (represent scores of a token being start token and end token) that the 4th token is the start token and the 5th token is the end token. Slicing the substring from index 4 to 5 will yield “Los Angeles”. That's it! For multiple support examples, we cannot do the same. Even if we just sum the scores for multiple examples, we need to weigh each token's similarity score based on the similarity of sentence structures of the query and the example. To achieve that, the authors introduce sentence-level similarity. We will shortly see how is that done. The representations of the query and the jth support example (assuming we have K support examples for each entity type) are defined by the vector summation of their token embeddings.  To measure the sentence-level similarity of a query with the support examples, they define the atten function, which is:  where T is a hyper-parameter (temperature of Softmax function). Now to calculate the scores of a token being start and end token, they use the following formula:  Previously, we only had the dot products. Now, for each support example, we take the dot product of a token (of the query) and the start/end token of the jth example. We multiply the dot product with the sentence-level similarity (atten). Now, we have K such vectors as start token scores and end token scores for all the support examples. Just summing up all these vectors will provide us with one vector that contains the score of each token, being the start token. The same goes for the end token. The loss function is the average of the span start and end prediction losses using cross-entropy, as shown below.  That was the gist of the paper, which I implemented in the FSNER library. I hope you have had a quick overview of the example-based named entity recognition. Note: All the equations and the dot-product diagram shown here were directly captured from the paper. FSNER Library: [sayef/fsner: Few-shot Named Entity Recognition (github.com)](https://github.com/sayef/fsner)

---