0

I am currently updating the NER model from fr_core_news_lg pipeline. The code used to work about 1 or 2 months ago, when I last used it. But now, something happened and I can't run it anymore. I haven't change anything from the code, just wanted to run it again. But I received the following error:

Traceback (most recent call last):
File "../nermodel.py", line 174, in <module>
ner_model.train(med_label)
File "../nermodel.py", line 102, in train
optimizer = self.nlp.entity.create_optimizer()
AttributeError: 'French' object has no attribute 'entity'

The error points to the part of the code where I update my NER model with new examples:

def train(self, label, n_iter=10, batch_size=50):
    # creating an optimizer and selecting a list of pipes NOT to train
    optimizer = self.nlp.entity.create_optimizer()
    other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']

    # adding a named entity label
    ner = self.nlp.get_pipe('ner')
    ner.add_label(label)

    with self.nlp.disable_pipes(*other_pipes):
        for itn in range(n_iter):
            random.shuffle(self.train_data)
            losses = {}

            # batch the examples and iterate over them
            for batch in spacy.util.minibatch(self.train_data, size=batch_size):
                texts = [text for text, entities in batch]
                annotations = [entities for text, entities in batch]

                # update the model
                self.nlp.update(texts, annotations, sgd=optimizer, losses=losses)
                print(losses)
    print("Final loss: ", losses)

A single training example, so that NER learns that 'consultation' is an entity, goes as follows:

('et la consultation post-réanimation', {'entities': [(6, 18, 'MEDICAL_TERM')]})

I've updated SpaCy to the most recent version, and downloaded again the fr_core_news_lg model, even tried this in a new python environment, to no avail. Which makes me think that there's a change in the pipeline or in SpaCy library. Googling around, I wasn't able to find precisely an answer for this. Does anybody have a fix for this?

EDIT: Provided more details.

Rafael
  • 23
  • 6
  • Is it possible that you used spacy 2 when the code worked? – krisograbek May 15 '21 at 03:52
  • The spaCy APIs changed when v3 was released - that's the reason the big version number went up. If you don't need to change your pipeline downgrading to spaCy 2.3.5 would be the easiest thing. In this case it looks like you're relying on the old pipeline API, which has changed quite a bit. I could help you update your code for v3 but it'd help to have a bigger sample to understand what you're trying to do. – polm23 May 15 '21 at 07:07
  • @krisograbek I could've swear I was using spacy 3, but that may be the case. Regardless, I had to download the french model again, prior to updating spacy version, and it didn't work. That's when I decided updating spacy as well. – Rafael May 15 '21 at 18:58
  • @polm23 any help would be appreciated. I'll update the question with more details. – Rafael May 15 '21 at 18:59

1 Answers1

1

I think this code should work for you:

def train(self, label, n_iter=10, batch_size=50):
    # creating an optimizer and selecting a list of pipes NOT to train
    optimizer = self.nlp.create_optimizer()
    other_pipes = [pipe for pipe in self.nlp.pipe_names if pipe != 'ner']

    # adding a named entity label
    ner = self.nlp.get_pipe('ner')
    ner.add_label(label)

    with self.nlp.disable_pipes(*other_pipes):
        for itn in range(n_iter):
            random.shuffle(self.train_data)
            losses = {}

            # batch the examples and iterate over them
            for batch in spacy.util.minibatch(self.train_data, size=batch_size):
                for text, annotations in batch:
                    doc = nlp.make_doc(text)
                    example = Example.from_dict(doc, annotations)
                    nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
                print(losses)
    print("Final loss: ", losses)

To break it down a little bit further, in spacy 3 there are two changes:

  1. They got rid of entity in nlp.entity.create_optimizer()
  2. We don't pass texts and annotations directly to nlp.update() but with Example
krisograbek
  • 1,642
  • 10
  • 16
  • just adding the import statement for Example: from spacy.training import Example – Ash Feb 25 '22 at 01:31