Stack Overflow
Stack Overflow

Questions tagged [disk-io]

Input/Output operations involving local persistent storage.

Common topics using this tag:

  • I/O-Bounded operations
  • General interaction with local storage

Useful Links

138 questions
188
votes
17 answers

Fast Linux file count for a large number of files

I'm trying to figure out the best way to find the number of files in a particular directory when there are a very large number of files (more than 100,000). When there are that many files, performing ls | wc -l takes quite a long time to execute. I…
Charles
18
votes
2 answers

How to measure file read speed without caching?

My java program spends most time by reading some files and I want to optimize it, e.g., by using concurrency, prefetching, memory mapped files, or whatever. Optimizing without benchmarking is a non-sense, so I benchmark. However, during the…
maaartinus
  • 44,714
  • 32
  • 161
  • 320
15
votes
6 answers

disk I/O error with SQLite3 in Python 3 when writing to a database

i am a student just starting out with python, and i was tasked with creating a relational database management system. I think i came pretty far, but i seem to have hit a wall. This is my code: import csv import sqlite3 conn =…
Xander Blaauw
  • 157
  • 1
  • 1
  • 4
11
votes
5 answers

Store/retrieve a data structure

I have implemented a suffix tree in Python to make full-text-searchs, and it's working really well. But there's a problem: the indexed text can be very big, so we won't be able to have the whole structure in RAM. IMAGE: Suffix tree for the word…
juliomalegria
  • 24,229
  • 14
  • 73
  • 89
10
votes
4 answers

Is is possible to write to different parts of the same file from multiple threads?

Can I write to different parts of the same file concurrently from multiple threads (on a typical PC)? I mean there's only one disk head, so the writes can be only performed in some order anyway i.e. not in parallel, right? Edit: I'm writing a…
szx
  • 6,433
  • 6
  • 46
  • 67
9
votes
1 answer

StrictMode: StrictModeDiskReadViolation when creating SharedPreference

I have a project with dagger setup with following provider method: @Module(...) abstract class AppModule { @Module companion object { ... @Provides @Singleton @JvmStatic fun provideSharedPreferences(@AppContext context:…
azizbekian
  • 60,783
  • 13
  • 169
  • 249
9
votes
2 answers

Performance: BufferedOutputStream vs FileWriter

I've always used a FileWriter to write text to a file in Java. Apparently you can also use a BufferedOutputStream as well. After reading both javadocs carefully, I can't seem to tell which was is faster/more efficient. So I ask: is there a…
user1768830
8
votes
4 answers

Can I/O latency cause a simple UPDATE to take seconds in MySQL?

My MySQL application is experiencing slow performance when running some UPDATE, INSERT and DELETE queries. In this question, I will only discuss one particular UPDATE, because it's enough to demonstrate the problem: UPDATE projects SET ring = 5…
Adiel Mittmann
  • 1,764
  • 9
  • 12
8
votes
1 answer

UNC path does not work with .NET?

I am running a very simple program, which is trying to list files in a folder on the same machine, which is specified using UNC format(as described in http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx): static…
whywhywhy
  • 278
  • 1
  • 5
  • 15
7
votes
1 answer

C Disk I/O - write after read at the same offset of a file will make read throughput very low

Background: I'm developing a database related program, and I need to flush dirty metadata from memory to disk sequentially. /dev/sda1 is volumn format, so data on /dev/sda1 will be accessed block by block and the blocks are adjacent physically if…
Chia
  • 102
  • 6
7
votes
4 answers

Improve performance of first query

If the following database (postgres) queries are executed, the second call is much faster. I guess the first query is slow since the operating system (linux) needs to get the data from disk. The second query benefits from caching at filesystem level…
guettli
  • 25,042
  • 81
  • 346
  • 663
6
votes
0 answers

Should getExternalCacheDir() be called outside Main Thread?

I am having a call to getExternalCacheDir() function on my Android App, and this is causing an ANR for some of Android 10 users having the following devices (HUAWEI P30 lite, Galaxy A40, Xperia XZ3). I am having the following stack trace for my…
AmineG
  • 1,908
  • 2
  • 27
  • 43
6
votes
1 answer

Can anyone explain me the difference between NET I/O vs BLOCK I/O in docker stats?

I am getting the following stats for my redis while running docker stats command. I want to know about NET I/O and BLOCK I/O. I read few documentation they are saying about BLOCK I/O is something related to disk operation. But I am not sure what…
Karthikeyan
  • 123
  • 2
  • 2
  • 7
6
votes
2 answers

Why Mongodb performance better on Linux than on Windows?

I created a programme to test sharded MongoDB performance on Linux(Ubuntu) and Windows(Server2008). With inserting large quantity of records, Windows's disk's active time is very high(100%), then performance is very bad. But on Ubuntu, the disk's…
vjHuang
  • 71
  • 1
  • 1
  • 3
5
votes
5 answers

Fastest way to merge millions of files

There are 81 million files (!) stored in one directory on a remote machine. All files end in ".paintedHaploDiversity". I would like to merge those files into one called allOutputs_3.5 in the parent directory. More specifically, each file contains…
Remi.b
  • 17,389
  • 28
  • 87
  • 168
1
2 3
…
9 10