3

When I compile my program in debug mode GetOpenFileName(&ofn) never returns.

It works perfectly fine in release mode. The only difference I can spot is that a lot of things are being optimized away in release mode.

OPENFILENAME ofn;
TCHAR szFile[MAX_PATH];

szFile[0] = '\0';
szFile[1] = '\0';

//Initialize OPENFILENAME
ZeroMemory(&ofn, sizeof(ofn));
ofn.lStructSize = sizeof(ofn);
ofn.hwndOwner = NULL;
ofn.lpstrFile = szFile;
ofn.lpstrFile[0] = '\0';
ofn.nMaxFile = MAX_PATH;
ofn.lpstrFilter = TEXT("Images (*.jpg;*.png;*.bmp;*.tga;*.psd)\0*.jpg;*.png;*.bmp;*.tga;*.psd\0\0");
ofn.lpstrInitialDir = TEXT(".");
ofn.lpstrTitle = TEXT("Open 512x512 image");
ofn.Flags = OFN_PATHMUSTEXIST | OFN_FILEMUSTEXIST;
...
GetOpenFileName(&ofn)

I'm compiling using Unicode character set.

If I remove \0 from the middle of ofn.lpstrFilter it works in debug mode, but obviously the filter doesn't function correctly.

This is how the string looks like in the memory in both debug and release mode around the \0 spot:

...snip...
0x00364BB4  70 00  p.
0x00364BB6  73 00  s.
0x00364BB8  64 00  d.
0x00364BBA  29 00  ).
0x00364BBC  00 00  ..
0x00364BBE  2e 00  ..
0x00364BC0  6a 00  j.
0x00364BC2  70 00  p.
0x00364BC4  67 00  g.
0x00364BC6  3b 00  ;.
...snip...

I'm probably doing something silly (I don't have much winapi /w Unicode experience), but I can't figure out what. Any ideas?

EDIT: Updated with current code.

Basically

ofn.lpstrFilter = TEXT("Images (*.jpg;*.png;*.bmp;*.tga;*.psd)*.jpg;*.png;*.bmp;*.tga;*.psd\0");
                                                              ^

works, but

ofn.lpstrFilter = TEXT("Images (*.jpg;*.png;*.bmp;*.tga;*.psd)\0*.jpg;*.png;*.bmp;*.tga;*.psd\0");
                                                              ^

doesn't.

EDIT: Tried reproducing same error in fresh Visual Studio project and I couldn't. It works. Did a diff on project settings and there aren't any mayor differences.

Maiku Mori
  • 7,419
  • 2
  • 40
  • 52
  • 1
    When you say "GetOpenFileName never returns", do you mean that the dialog appears and functions normally, but then locks up when you click OK or Cancel. Or, does the dialog not appear at all? – cbranch Nov 10 '11 at 13:26
  • @cbranch, it doesn't appear at all. It appears to be in infinite loop, but since I don't have debug data for it, I'm left with disassembled data. Haven't tried to figure out what's going on in it since it's a bit of pain. I'm quite sure that for some reason it can't parse the filter and just gets in a loop trying to parse it. But I have no idea why. – Maiku Mori Nov 10 '11 at 13:32
  • Could we assume you do no other initialisation of `ofn` but such you show in the code posted? – alk Nov 10 '11 at 13:41
  • 1
    If there's something wrong with your filter, it's not jumping out at me. You have a extra (redundant) null terminator at the end, but that doesn't matter. Is this compiled to a .exe or .dll? What version of Windows are you using? – cbranch Nov 10 '11 at 13:45
  • The code is at start of open_file() function, so yeah, no other initialisations. And it doesn't work on first call. – Maiku Mori Nov 10 '11 at 13:47
  • 1
    What happens if you set lpstrInitialDir to NULL? I don't think I've ever used a relative path like that. I use either NULL, or full path. – cbranch Nov 10 '11 at 13:50
  • @cbranch, It's compiled to exe. I'm using Windows 7 (64bit, but compiling as 32bit) running Japanese locale at the moment (but no funky paths anywhere). Tried with `lpstrInitialDir = NULL` before, it didn't work. In release mode the relative path works fine. – Maiku Mori Nov 10 '11 at 13:50
  • Look at the Output window. Do you see a "first chance exception"? – Hans Passant Nov 10 '11 at 14:06
  • @Hans, nope. As it turns out it was caused by Visual Leak Detector. Also there shouldn't be any exceptions since I'm compiling it as C. – Maiku Mori Nov 10 '11 at 14:27

3 Answers3

3

1 ofn.lpstrFilter shall be terminated by two zeros: \0\0.

2 Are you sure ZeroMemory() really zeros out ofn's content. You might try SecureZeroMemory() instead.

Referring 2 a note from msdn:

Use this [SecureZeroMemory()] function instead of ZeroMemory when you want to ensure that your data will be overwritten promptly, as some C++ compilers can optimize a call to ZeroMemory by removing it entirely.

I'd simply use memset() .

alk
  • 69,737
  • 10
  • 105
  • 255
  • The second one should be appended automatically by compiler. Even manually adding 2 null-chars at the end doesn't fix the problem. – Maiku Mori Nov 10 '11 at 13:06
  • Yeah I was going to rewrite it using `memset()`, it was partly copy-paste code from documentation which used `ZeroMemory()` – Maiku Mori Nov 10 '11 at 14:19
2

A few things...

1) You are initializing lpstrFile twice.

ofn.lpstrFile = szFile;
ofn.lpstrFile = TEXT('\0');

2) You should initialize szFile (probably your intention in #1):

TCHAR szFile[MAX_PATH];
szFile[0] = '\0';

3) nMaxFile should be the maximum number of CHARACTERS, not bytes. So, you want either MAX_PATH or sizeof(szFile) / sizeof(TCHAR).

4) You are missing a '*' in your file filter: should be *.jpg

cbranch
  • 4,709
  • 2
  • 27
  • 25
  • Good catch, but it doesn't fix the problem. This was probably introduced while I was trying to get this thing working : – Maiku Mori Nov 10 '11 at 13:09
  • Yeah totally didn't notice the #3 since start, but still nothing. Even using the slightly incorrect code works in release and debug if I remove the null char from middle of filter. So it's something a bit more tricky. – Maiku Mori Nov 10 '11 at 13:12
2

I finally isolated the only difference between the builds:

I was using Visual Leak Detector in my debug build. After removing it, it started to work.

I'll try to get to the bottom why it happens, but I just wanted to thank every one trying to help me. I wish I could reward more points.

Some more people having similar problems:

Maiku Mori
  • 7,419
  • 2
  • 40
  • 52
  • 1
    Note for future readers: it seems to have been fixed and release with the version 2.5 (http://vld.codeplex.com/releases/view/619330) – Guillaume George Feb 03 '16 at 07:07