Accessing a non-mutable hash map inside another hashmap

Question

Basically what I've got is a hash map which has another hashmap as it's value.

I'm new to rust so what I'm trying to do is insert into an already existing hashmap, when I try that the compiler complains that neighbors is not mutable.

But if I make it mutable by doing Some(mut neighbors) => { then I get another error:

error[E0596]: cannot borrow `*neighbors` as mutable, as it is behind a `&` reference
  --> src\main.rs:24:13
   |
23 |         Some(mut neighbors) => {
   |              ------------- help: consider changing this to be a mutable reference: `&mut std::collections::HashMap<u32, bool>`
24 |             neighbors.insert(j, true);
   |             ^^^^^^^^^ `neighbors` is a `&` reference, so the data it refers to cannot be borrowed as mutable

What's confusing is that I'm inserting a mutable reference so I would expect it to stay mutable.

Here's the complete bit of code:

use std::collections::HashMap;

fn main() {
    let matrix = HashMap::new();
    let step1 = add_edge(matrix, 1, 2);
    let step2 = add_edge(step1, 5, 1);
    let step3 = add_edge(step2, 5, 2);
    let step4 = add_edge(step3, 2, 6);

    for (&i, secondLevel) in step4.iter() {
        for (&j, &value) in secondLevel.iter() {
            println!("Calling {}, {}: {}", i, j, value);
        }
    }
}

fn add_edge(
    mut matrix: HashMap<u32, HashMap<u32, bool>>,
    i: u32,
    j: u32,
) -> HashMap<u32, HashMap<u32, bool>> {
    match matrix.get(&i) {
        Some(neighbors) => {
            neighbors.insert(j, true);
        }
        _ => {
            let mut neighbors = HashMap::new();
            neighbors.insert(j, true);
            matrix.insert(i, neighbors);
        }
    }

    match matrix.get(&j) {
        Some(neighbors) => {
            neighbors.insert(i, true);
        }
        _ => {
            let mut neighbors = HashMap::new();
            neighbors.insert(i, true);
            matrix.insert(j, neighbors);
        }
    }

    return matrix;
}