1

I often need to look up log files, problem is the log files are burried in dozens of folders each and to look up 50 would take literally an hour or more.

I have been using a batch file to scan the drive overnight and compile a list of all directories in the following format

Z:\folder\folder2\folder3\folder4\folder5\folder6\folder7\ <about another 20 folders > \log.txt

Current command is: 

dir /b /-d /-p /s /A:-D > directories.txt

The txt file has like 500 thousand lines of this.

Then when I need to look up a set of logs I would run another batch to pull a set of 50 logs based on scanning that txt file.

Problem with the current solution is that with the log database growing it is now taking 12+ hours to scan the directories. Which makes it un-runnable overnight. And I need to run this every night to keep the logs current.

Question:

So question to you guys, what is the best way to do this? I cant change any of the directory structures (this is a database of logs used by hundreds of people), and I dont really know any languages other than Batch scripting. But it seems like Batch is limited and doesnt allow me to do any of the following (which would solve my problem)

  • Skip directories that have not been modified in the last 48 hours
  • Skip subdirectories of folders with specific keywords in the name

If I could do the above 2 with batch, it would probably take the txt file output from 500 thousand lines to maybe 3 thousand.

Etheryte
  • 24,589
  • 11
  • 71
  • 116
Duxa
  • 966
  • 2
  • 14
  • 27
  • This is not complete solution but it can give you an idea. As you know Python is easy to learn. For your operation you don't need to learn it completely. So only look this to open your mind. http://stackoverflow.com/questions/120656/directory-listing-in-python – hakki Jun 05 '14 at 18:22
  • What are you asking, to find the files and output them with their paths or to do this but only for files modified in the last 48 hours? Please make your question more concise. – Ian Stevens Jun 05 '14 at 18:46
  • @IanStevens: he is hoping to speed up things, by skipping older files, but to know a files age, the directory entry has to be read. So this would not make anything faster. Skipping some directories would speed up, but will need some code. – Stephan Jun 07 '14 at 19:34
  • @Stephan Yeah, I tried Cygwin running through bash script, and its taking just as long if not longer, I am avoiding a ton of directories during the process too.. I wonder if cmd is just way faster than cygwin on Windows. – Duxa Jun 09 '14 at 19:18

2 Answers2

1

If you can install Cygwin on the machine, that would give you access to using bash along with grep / find / etc. tools which are standard in Unix/Linux/BSD and will run on Windows under a Cygwin shell. Then you can use the various solutions like:

Grep inside all files created within date range

Otherwise, you're probably going to have to invest in learning Powershell.

Community
  • 1
  • 1
tgharold
  • 721
  • 6
  • 15
  • Yes I can install Cygwin, ill look into this. Thanks! – Duxa Jun 05 '14 at 22:10
  • I installed Cygwin, question is... what commands do I need to use. Using "find" seems to look for the key string inside each txt file. Which makes it take longer. I just need it to look at folder names, and if it hits certain ones to skip all subdirectories (based on keyword as part of the name). – Duxa Jun 06 '14 at 01:50
  • The 'find' command, assuming you use the correct -mtime argument will then only grep inside files within the last 48 hours. For instance "find -mtime -2" will only list files that are within the last 2 days. You could also choose to search only for directories with a particular keyword in the name, modified within the last N days, and pass that list via xargs to a second find for the actual grep. – tgharold Jun 09 '14 at 20:10
  • A good way to test 'find' is with "find . -mtime -2 | wc -l" which will tell you how many things your 'find' command found. And "find . | wc -l" will tell you the total number of things that could be possibly be found. "find . -type d | wc -l" for the number of directories, etc. – tgharold Jun 09 '14 at 20:12
0
>"output.txt" robocopy z:\ "%temp%" /l /s /nc /ns /njh /njs /ndl /fp /maxage:2 /xd folder7 otherFolder

Robocopy (from windows vista and up, but available in W2003 Resource Kit Tools) can be used to get the recursive list of files with the indicated conditions: modified in the last two days and excluding some folders from the process.

MC ND
  • 69,615
  • 8
  • 84
  • 126