How to replace an image with a new one with the same filename, in runtime in a WPF application in MVVM

Question

I am trying to get an image from the user through an openFileDialog, then change an existing image by the new one the user picked and then save that new image to a folder within the application with the exact same filename. Adding an image for the first time is fine but, trying to replace the image if the user wants to change it is troublesome. Whenever i try this , i get the error:

System.IO.IOException: 'The process cannot access the file 'File Path' because it is being used by another process.'

I understand that my application is using the file to show the image, but can i end this process somehow and then save the new image with the same filename? I tried using File.Delete(filename), but it gives me the same error.

Screenshot of the WPF application: screenshot

I have a property in my view model

private BitmapImage _previewImage;
public BitmapImage PreviewImage
        {
            get { return _previewImage; }
            set { _previewImage = value;
                NotifyPropertyChanged();
            }
        }

the image is added and shown like this :

public void AddImage()
    {
        
        OpenFileDialog openFileDialog = new OpenFileDialog()
        {
            InitialDirectory = Environment.SpecialFolder.MyPictures.ToString(),
            Filter = "Image Files |*.png;*.jpeg;*.jpg;*.gif",
            FilterIndex = 0,
        };

        if (openFileDialog.ShowDialog() == true)
        {
            string uriString = openFileDialog.FileName;
            PreviewImage = new BitmapImage(new Uri(uriString, UriKind.Absolute));
        }

        
    }

Now when i want to execute following code, it throws the error on the line where i start the Filestream, again, only when i already have a image saved an i simply want to replace the image with a new one but keep the same filename. For an initial save this works fine:

public string SaveImage()
    {
        if (PreviewImage != null)
        {

            string extension = Path.GetExtension(PreviewImage.UriSource.ToString());
            string filename = @"\" + Exhibition.Title + extension;
            string fullpath = Dir + filename;               


            using (FileStream filestream = new FileStream(fullpath, FileMode.Create, FileAccess.Write))
            {
                BitmapEncoder encoder;

                switch (extension)
                {
                    case ".gif": encoder = new GifBitmapEncoder(); break;
                    case ".png": encoder = new PngBitmapEncoder(); break;
                    default: encoder = new JpegBitmapEncoder(); break;
                }

                encoder.Frames.Add(BitmapFrame.Create(PreviewImage));
                encoder.Save(filestream);
            }

            return filename;
        }

        return null;

    }

This might be of relevance, because i feel like this might be the problem ? Whenever i want to edit my exhibition, I set my image in my constructor with this directory, like this:

string Dir = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location) + @"\..\..\Images";

public NewExhibitionViewModel(int? exhbitionId)
        {
            
            if (exhbitionId != null)
            {
                Exhibition = unitOfWork.ExhibitionRepo.Get( 
                    x => x.ExhibitionID == exhbitionId,
                    x => x.ExhibitionArtists.Select(y => y.Artist)
                    ).SingleOrDefault();
                
                //Exhibition.ImageFile = "\ExhibtionName.jpg"
                PreviewImage = new BitmapImage(new Uri(Dir + Exhibition.ImageFile));
                
                ...

            }
                
        }