Idiom to get or create mutable collection from map member fields

Question

I have a type with HashMap members which hold a HashSet as value. If the HashMap does not contain a given key, I need to create a new HashSet and add it to the map.

Since I have multiple similar members in my type I want to create a single method to handle that.

Sadly I always end up with multiple mutable borrows add_to_index.

How can I solve this or is there a better idiom?

use std::collections::HashMap;
use std::collections::HashSet;

struct TopicIndex {
    all: HashSet<String>,
    topicsByName: HashMap<String, HashSet<String>>,
    rootsByName: HashMap<String, HashSet<String>>,
    endsByName: HashMap<String, HashSet<String>>,
}

enum IndexType {
    Roots,
    ByName,
    Ends,
}

impl TopicIndex {
    pub fn new() -> Self {
        TopicIndex {
            all: HashSet::with_capacity(32),
            topicsByName: HashMap::with_capacity(128),
            rootsByName: HashMap::with_capacity(128),
            endsByName: HashMap::with_capacity(128),
        }
    }

    // [...]

    fn add_to_index(&mut self, name: String, topic: String, indexType: IndexType) {
        let mut index = match indexType {
            IndexType::ByName => &mut self.topicsByName,
            IndexType::Roots => &mut self.rootsByName,
            IndexType::Ends => &mut self.endsByName,
        };
        // first mutable borrow occurs here
        let mut setOpt = index.get_mut(&name);
        if setOpt.is_some() {
            setOpt.unwrap().insert(topic);
        } else {
            let mut set = HashSet::with_capacity(32);
            set.insert(topic);
            // second mutable borrow occurs here
            index.insert(name, set);
        }
    }
}