Can not remove file a second time

Question

I made some dcode that will change one line by copying the first file with the correct line replaced. Then the first file gets deleted and the temp file gets renamed. This is the code:

            File countFile = new File(DEF_PATH + DEF_PATH_EX);
            File tempFile = new File(DEF_PATH + DEF_PATH_EX_TEMP);

            PrintWriter pWriter = new PrintWriter(new BufferedWriter(new FileWriter(tempFile)));
            BufferedReader bReader = new BufferedReader(new FileReader(countFile));

            String lineRead;

            boolean hasChanged = false;

            while ((lineRead = bReader.readLine()) != null)
            {
                String[] dataLine = lineRead.split(":");

                if (dataLine[0].equals(userString.substring(1)))
                {
                    lineRead = String.format("%s:%s:", dataLine[0], Integer.toString(Integer.parseInt(dataLine[1]) + 1));
                    hasChanged = true;
                    System.out.println(lineRead);
                }
                pWriter.println(lineRead);
            }
            if (!hasChanged)
            {
                pWriter.println(userString.substring(1) + ":1:");
            }

            pWriter.close();
            bReader.close();

            countFile.delete();
            tempFile.renameTo(new File(DEF_PATH + DEF_PATH_EX));

The problem is I can not do "countFile.delete()" when I removed it already 1 time before. So then renaming is a no go as well.