We receive around 10 million images per day ranging in size from 3kb to 200kb. At peak times it is around 400 images per second. It is an average of around 30kb per image. At the moment all these images come into a single server with a 1TB NVMe SSD for storage. At night we move the days images to an archive server.
At peak times users want to read the latest images as they are being written but there are delays as it appears the server is attempting read and write at the same time. What is the best way to be able to succeed in read / write to the same volume and be able to scale easily going forward?
I've started looking into distributed file systems like SeaweedFS. Is this the right way to go? Are there better options than SeaweedFS?
Thank you
