I have to create a proxy for file uploading but the problem is that I don't have to store an entire file on the proxy server. My proxy have to accept very big files (files with size up to 80 GB) and store an entire file in the object storage (openstack swift). But proxy don't have to keep an entire file in the RAM or on the disk. Proxy have to act like a pipe.
How to create such system?
Asked
Active
Viewed 1,090 times
1
Enamul Hassan
- 5,266
- 23
- 39
- 56
Oleksandr
- 3,574
- 8
- 41
- 78
1 Answers
2
Simple create a server socket to accept the requests and for each request received, open an output stream to the target (filesystem, database, cloud storage, ...). Then do a loop which reads a chuck of bytes, e.g. 4K bytes, from the input stream and then write those 4K bytes to the output stream. Continue this, until there are no more bytes to read from the input. Close the input stream and flush and close the output stream. See here for an example
