Compare if two file are the same over the internet

Question

Here is my scenario - I have a windows store app. I have a local file, and a link to a file on the internet. Is there a way I can check if these two files are the same, WITHOUT downloading the file from the link?

The code used to get the file is this:

private static async void SetImage(PlaylistItem song, string source, string imageName)
{

    HttpClient client = new HttpClient();

    HttpResponseMessage message = await client.GetAsync(source);

    StorageFolder myfolder = Windows.Storage.ApplicationData.Current.LocalFolder;
    StorageFile sampleFile = await myfolder.CreateFileAsync(imageName, CreationCollisionOption.ReplaceExisting);
    byte[] byteArrayFile = await message.Content.ReadAsByteArrayAsync();

    await FileIO.WriteBytesAsync(sampleFile, byteArrayFile);

    song.Image = new BitmapImage(new Uri(sampleFile.Path));

}

Answer 1

The usual solution is to keep a hash of the cloud file somewhere, usually in the file's metadata and compare it with the hash of your local file. Checksums are unsuitable for this operation because they have a very high chance of collision (ie different files having the same checksum).

Most storage services (Azure Blob storage, Amazon S3, CloudFiles) actually use a file's MD5 or SHA hash as its ETag, the value used to detect changes to a file for caching and concurrency purposes. Typically, a HEAD operation on the file will return its headers and ETag value.

If you have the option of picking your own algorithm, choose SHA256 or higher as these algorithms are highly optimized and their large block size means that calculating hashes for large files is much faster. SHA256 is actually much faster than the older MD5 algorithm.

What storage service are you using?

EDIT

If you only want to check files to avoid downloading them again, you can use the ETag directly. ETag was created for exactly this purpose. You just have to store it together with your file when you download it the first time. That's how proxies and caches know to send you a cached version of a picture instead of hitting the destination server.

In fact, you can probably just do a GET on the file with the ETag/If-None-Match headers. The intermediate proxies and the final web server will return a 304 status code if the destination file hasn't changed. This will halve the number of requests you need to download all images in your list.

An alternative is to store the Last-Modified header value for the file and use the If-Modified-Since header in GET

EDIT 2

You mention that the ETag header is null, although your code doesn't show how you retrieve it.

HttpResponseMessage has multiple Headers properties, both on the message itself and its Content. You need to use the proper property to retrieve the ETag value.

You can also check using Fiddler to ensure the server does actually return an ETag.

EDIT 3

Finally found a way to get an ETag from Youtube! The answer comes from "How to get thumbnail of YouTube video link using YouTube API?"

Doing a HEAD or GET on a YouTube thumbnail from ytimg.com does NOT return the ETag or Last-Modified headers.

Using YouTube's Data API and doing a GET on gdata.youtube.com on the other hand, returns a wealth of information about the video. An ETag value is included, although I suspect it changes whenever the video changes. This may be OK though, if you only want to download an image when the video changes, or you don't want to download the image a second time again.

The code I used was:

var url = "http://gdata.youtube.com/feeds/api/videos/npvJ9FTgZbM?v=2&prettyprint=true&alt=json";

using(var  client = new HttpClient())
{
    var response = await client.GetAsync(url);
    var etag1 = response.Headers.ETag;
    var content = await response.Content.ReadAsStringAsync();
    ...
}

Answer 2

You could calculate a hash of the file contents like git does. Use MD5 or similar. Then you only need to check if files have the same hash.

Answer 3

If you want to do a comparison without downloading and you are the one who has placed the file over the internet. Then ideally you should place a checksum of the file uploaded. Then before uploading a new one you can just check the checksum of local file and the one on the server. if it is not equal proceed with the upload else cancel it.

Answer 4

Directly? No. If the file online is also provided with a Hash, you can get a high probability of successfully checking the equality of the files, though.

Answer 5

Now with your update, it's kind of clear what your code does: it downloads an image from a given URL and stores it in your application data folder under the given filename. You want to download any image only once.

It's still unclear to me how you call this code, but the solution to me looks like you just need an "URL to filename" translation. So, in psuedo:

BitmapImage GetImage(string sourceURL)
{
    string filename = GetFilenameForURL(sourceURL);

    BitmapImage image;

    if (!FileExists(filename))
    {
        image = DownloadAndSaveImage(sourceURL, filename);
    }
    else        
    {       
        image = ReadImageFile(filename);
    }

    return image;
}

This does not account for images that have been updated on the server. If you want to do that, you need to save metadata in the DownloadAndSaveImage() call, for example the mentioned ETag or last-modified date.

Then to save bandwidth, you can do a HEAD or conditional GET request with an if-none-match or if-modified-since header before the call to ReadImageFile() to check if a newer version is available.