Problem: I want to use FileOpenPicker to read a binary file over 4GB into std::vector in UWP
It'd also be great if I could use the default picker, but open it with fstream instead. I'm guessing this is easy to do and quite possible, but I'm having trouble figuring it out.
Background: The file I want to load is typically between 2GB and 4GB in size, and I haven't been able to figure out what sort of guarantee (if any) the typical UWP library options provide. I don't want to stream the data through a buffer, unless the buffer is large enough to fit the entire file.
Irrelevant aside: I feel like someone is going to ask about why I'd want to load a large file into memory instead of streaming it or buffering it, and it's because I need to perform a number of operations on all of the data over multiple iterations, and the disk I/O slows down the calculations a lot.
edit: example: I've tried a number of things. I'd love it if I had something like this:
Windows::Storage::Pickers::FileOpenPicker ^picker = ref new FileOpenPicker();
concurrency::create_task(picker->PickSingleFileAsync()).then([](StorageFile ^file) {
/* some tests */
std::experimental::filesystem::path data_file = to_string(file->Path);
/* access file here */
}
/* or access file here, I don't care where as long as I can do it */
The issue with this implementation is that I cannot actually do anything with the file. It correctly returns the path to the file, but I cannot access it. I've also tried just returning the name and operating on that, but I still can't use fstream on the file. I suspect that's due to the restricted capabilities in UWP. https://learn.microsoft.com/en-us/windows/uwp/files/file-access-permissions I know it's not recommended, but short of just being able to load data into a vector via some msft pointers, I'd at least like to be able to get it via normal ISO C++ operations.
I've also tried implementing something similar to this (a sample from msft): https://github.com/Microsoft/Windows-universal-samples/blob/master/Samples/FileAccess/cpp/Scenario5_WriteAndReadAFileUsingAStream.xaml.cpp but those too fail to load the file. A few samples from that set don't build on visual studio, so I'm not sure if they are any good to check out anyway.
I was able to load a bmp more or less following the examples given, but I couldn't figure out how to load normal binary data using similar constructs. They have a lot of functionality for some image classes, but I don't have images.
I noticed this comment, but it just looked like an implementation issue: Files larger than 4GB need to be broken into multiple chunks to be loaded by LoadAsync.
Anyway, I figure that there has to be something as easy as this for UWP:
std::ifstream ifs(data_file, std::ios::binary | std::ios::ate);
/* find end of file, store in size, reserve memory for vector */
ifs.read(byte_vector.data(), size);
Edit 2: I've resorted to reading random blogs by MSFT people. https://blogs.windows.com/buildingapps/2016/11/28/standard-c-windows-runtime-cwinrt/
The suggestion in this blog was to do something like this:
FileOpenPicker picker;
picker.FileTypeFilter().Append(L".png");
picker.SuggestedStartLocation(PickerLocationId::PicturesLibrary);
auto file = co_await picker.PickSingleFileAsync();
Unfortunately, when I switched over and tried to do a WinRT app, the only thing available was FileOpenPickerUI, which is totally different from FileOpenPicker. The code above also coughs up on the "co_await" line.
Another random MSFT blog post was presented here: https://blogs.windows.com/buildingapps/2018/05/18/console-uwp-applications-and-file-system-access/ This individual reads a file into a string (I know, bad practice 101, but literally half of their documentation does the same thing.)
hstring text = FileIO::ReadTextAsync(file).get();
When I attempt to modify the above line, I can't get it to read into a vector.
I also decided to try doing something like this: https://stackoverflow.com/a/14383816/9824697, where I put the file retrieved from the picker (or attempt to?) into an IBuffer. This also didn't work for me.
EDIT 3: Ok, so I tried this (more or less from here: https://github.com/Microsoft/Windows-universal-samples/blob/6370138b150ca8a34ff86de376ab6408c5587f5d/Samples/Simple3DGameDX/cpp/Common/BasicReaderWriter.cpp)
create_task(picker->PickSingleFileAsync()).then([=](StorageFile ^picked) {
/* some error checking */
CREATEFILE2_EXTENDED_PARAMETERS extendedParams = { 0 };
extendedParams.dwSize = sizeof(CREATEFILE2_EXTENDED_PARAMETERS);
extendedParams.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
extendedParams.dwFileFlags = FILE_FLAG_SEQUENTIAL_SCAN;
extendedParams.dwSecurityQosFlags = SECURITY_ANONYMOUS;
extendedParams.lpSecurityAttributes = nullptr;
extendedParams.hTemplateFile = nullptr;
String ^Lame = picked->Path;
Wrappers::FileHandle file(CreateFile2(Lame->Data(),
GENERIC_READ,
FILE_SHARE_READ,
OPEN_EXISTING,
&extendedParams));
FILE_STANDARD_INFO file_info = { 0 };
GetFileInformationByHandleEx(file.Get(), FileStandardInfo, &file_info, sizeof(file_info));
Platform::Array<byte> ^file_data = ref new Platform::Array<byte>(file_info.EndOfFile.LowPart);
ReadFile(file.Get(), file_data->Data, file_data->Length, nullptr, nullptr);
But in the end, "file_data->Length" is always 0, and there's no data in file_data.