How to reset button after event fires (using removeEventListener)?

Question

I'm currently working on a simple color generator that produces a random hexademical values (#FFFFFF), and changes the background color based on the hexcolor provided.

The program works well enough, but the button element doesn't seem to trigger the submit() function as its supposed to.

button is the reference for the HTML DOM element that will trigger the change in background color. I used button.addEventListener('click', submit()); to listen to the event trigger. It only works if you refresh the page, but not if you click on the button. Here's a link to the github repo.

// DOM elements
const button = document.getElementById('submit');
const para = document.getElementById('paragraph');

//Hex color variable
let hexaDecimal = [];
const arrangement = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 'A', 'B', 'C', 'D', 'E', 'F'];

// Function that randomizes a character 0-F, and pushes that to the hexaDemical array
const setColor = () => {

    for (let i = 0; i < 6; i++){
        let randNum = Math.floor(Math.random() * 16);

        hexaDecimal.push(arrangement[randNum]);
    }
  
  hexaDecimal = hexaDecimal.join("");
  hexaDecimal = '#' + hexaDecimal;

  return hexaDecimal;
}

const submit = () => {
    let newColor = setColor(); 
    document.body.style.backgroundColor = newColor;
    para.innerText = newColor;
    button.removeEventListener('click', submit());
}

button.addEventListener('click', submit());