I have a folder on my desktop with 3,000+ small .txt files in it. How can I get that folder into Hadoop with all of the .txt files instead of uploading each one separately?
I am using Ubuntu OS with Hadoop 3.1.2
Asked
Active
Viewed 28 times
0
-
Loading the file into HDFS in many ways. This one has detail explanation https://stackoverflow.com/questions/28213116/hadoop-copy-a-local-file-system-folder-to-hdfs – Skanda Shastry Nov 17 '20 at 06:46
1 Answers
0
HDFS is not meant to store "small files" of any type. You should compress them into a BZ2 archive, for example, then upload that to HDFS.
Bzip archives are readable by most Hadoop libraries.
OneCricketeer
- 179,855
- 19
- 132
- 245