-1

I have a bash script in Linux (CentOS) that redirects output to a file. It takes a few minutes to run:

./myBashScript.sh >> file.csv

I have a java application that tries to read the file (only read) for further processing:

    File file = new File("file.csv");
            Scanner input = new Scanner(file);
            while (input.hasNextLine()) {
                String line = input.nextLine();
                // do something...
            }

These processes are on Crontab. It works all fine, but apparently whenever the bash script is redirecting the output to the file, the Java app can not read it. The logs shows "File not found" exception!

Is the file locked?! How can I fix the problem?

Tina J
  • 4,983
  • 13
  • 59
  • 125
  • 3
    ...mind you, if by "can not read it" you mean "can read it, but it's empty", that's a whole different question (one that probably relates to [buffering](https://mywiki.wooledge.org/BashFAQ/009)). – Charles Duffy Jul 31 '18 at 19:45
  • @gtgaxiola no this is a different question. I don't want to read from while it is being appended continuously. I'm asking about lock. – Tina J Jul 31 '18 at 19:46
  • 1
    @TinaJ, ...the bash code you quoted has no locking. Could be your operating system or filesystem has constraints, but that's not the case on POSIX-standard Unixlikes. – Charles Duffy Jul 31 '18 at 19:46
  • @TinaJ yet I believe the answer can be applied (while can't read... sleep) – gtgaxiola Jul 31 '18 at 19:46
  • @CharlesDuffy Yes, there is a possibility that the file is empty. But my logs shows "File does not exits" exception. – Tina J Jul 31 '18 at 19:47
  • 4
    "File does not exist" / "file not found" isn't a locking problem; a file being locked creates a completely different error. Make sure your Java code is actually looking with the right name, in the right directory, etc. – Charles Duffy Jul 31 '18 at 19:47
  • These processes are on Crontab. It works all fine, but apparently whenever the bash script is redirecting the output to the file, the Java app can not read it. – Tina J Jul 31 '18 at 19:49
  • 1
    Please include all the details necessary to let someone else see the problem themselves. Keep in mind that processes started by `cron` are liable to have `/` as their working directory, *not* the user's home directory or otherwise the default you'd expect in other circumstances. – Charles Duffy Jul 31 '18 at 19:50
  • 1
    ...so a good place to start is to change from `File("file.csv")` to `File("/home/someuser/file.csv")`, as appropriate for the actual location; see if that gets you a more specific/useful error message. – Charles Duffy Jul 31 '18 at 19:52
  • 1
    @TinaJ, we do not dispute that you observe a problem, but the way you characterize it seems inconsistent with the way Linux file management works. In particular, that one process has a file open for writing does not prevent other processes from opening that file (unlike in Windows). – John Bollinger Jul 31 '18 at 19:52
  • @CharlesDuffy yes. The code has the real file path. I just entered a fake one here for exemplification. – Tina J Jul 31 '18 at 19:53
  • 1
    I'm not just saying "real" path, but to use an *absolute, fully-qualified* path (one that starts with `/`, so it has the same meaning no matter which directory you're in), and actually copy-and-paste it between the two locations in your code to catch any typos. – Charles Duffy Jul 31 '18 at 19:54
  • OK. Apparently I just need to somehow make sure the bash is finished, then run the Java app. Or once the whole bash is finished, just rename it! These trick should solve the problem – Tina J Jul 31 '18 at 19:55
  • 1
    I frankly have trouble believing that that would solve your problem. Nothing on Linux prevents a Java process from opening a file that a shell is actively writing to. It would solve *other* problems (reading partially-written files is a fast track to race conditions), but... well, those other problems are topics for different questions. :) – Charles Duffy Jul 31 '18 at 19:56
  • Is it possible that your bash script *deletes* the file before (after some delay) it starts rewriting it? – John Bollinger Jul 31 '18 at 19:56
  • Silly question here -- are you sure your bash script didn't have DOS newlines? If it did, it could be creating a file named `file.csv$'\r'`, which would explain why the Java program couldn't find `file.csv`. – Charles Duffy Jul 31 '18 at 20:02

1 Answers1

-3

OK. Apparently I just need to somehow make sure the bash is finished, then run the Java app. Or once the whole bash is finished, just rename it! These trick should solve the problem

Tina J
  • 4,983
  • 13
  • 59
  • 125
  • If that's what you want to do, consider using advisory locking. `flock -x file.csv ./myBashScript >>file.csv`, and `flock -x file.csv java -jar myJavaProgram.jar file.csv` will ensure that only one process holds the lock at a time. – Charles Duffy Jul 31 '18 at 20:00
  • NMDV. Nevertheless, although ensuring that the Java program does not run concurrently with the `bash` script might indeed solve your problem, that would be because of an incompatibilty between the two *residing somewhere else* than the script command you asked about. And if a file renaming is required, then that's even more so indicative of a different problem. – John Bollinger Jul 31 '18 at 20:00
  • That said, if the approach described in this answer solves a FileNotFoundException, it's solving it only incidentally (as a side effect of reworking whichever code was *actually* causing the error). – Charles Duffy Jul 31 '18 at 20:01
  • @CharlesDuffy, isn't it the case that the `flock` approach will be ineffective at providing mutual exclusion if one or the other program deletes the lock file? – John Bollinger Jul 31 '18 at 20:04
  • 1
    @JohnBollinger, definitely true. If either program is deleting the file, that logic needs to be brought into scope of the question (and the lockfile should probably be made a separate inode from the data file). – Charles Duffy Jul 31 '18 at 20:05