Ok, after some trail and error I managed to find a solution that works:
Sys.setlocale("LC_ALL", "Hebrew")
library(readr)
zzt <- read_csv("D:/your_file_path/file_name.csv", locale = locale(date_names = "he", encoding = "UTF-8"))
The first line sets Hebrew as the local language.
The other two line are part of the dataset uploading process (save in the 'import' GUI), note the two additions inside the locale brackets.
Make sure the CSV file has 'UTF-8' encoding. If not, open it via Notepad and save again with this encoding in the CSV format.