The REPLACE statement consists in the following algorithm:
MySQL uses the following algorithm for REPLACE (and LOAD DATA ... REPLACE):
Try to insert the new row into the table
While the insertion fails because a duplicate-key error occurs for a
primary key or unique index:
(https://dev.mysql.com/doc/refman/5.7/en/replace.html)
So you can't keep a value from a line which is going to be deleted.
What you want to do is emulating a "ON DUPLICATE KEY UPDATE" logic.
You can't do that within a single LOAD DATA query. What you have to do is to load your data in a temporary table first, then to make an INSERT from your temporary table to your destination table, where you will be able to use the "ON DUPLICATE KEY UPDATE" feature.
The whole process is fully detailed in the most upvoted answer of this question : MySQL LOAD DATA INFILE with ON DUPLICATE KEY UPDATE