I'm trying to build OpenSsl 3.0 to get updated FIPS capability. The OpenSsl FIPS PDF says install Perl & NASM; did Strawberry latest & NASM latest. Modified VsDevCmd.bat to include nasm in the path. Supposedly three commands in an admin VS developer CLI window will do the trick:
perl Configure enable-fips
nmake
nmake install
With a fresh extract of the code, the first two succeed. The last builds a bunch of test things and it fails on http_test.exe with fatal error LNK1108: cannot write file at 0x0. Given it's written all manner of stuff to the test diretory before this, it seems unlikely to be a permissions issue.
System is w10 build 19045.2846, 12 cores, 32 GB RAM (17GB free), 1TB m.2 SSD (155GB free), VS2022 (17.4.5), Strawberry Perl (5.32.1), nasm (2.16.01), OpenSsl (3.0.0) downloaded May 3, 2023.
I deleted the source directory, re-extracted the code, and tried with just "perl Configure" as the first command, to see if it would build a non-FIPS OpenSsl. When doing FIPS it appears it initially succeeded on the second, nmake only command, and it did so each time I deleted the source and re-extracted. Except after I posted this question it consistenly fails on that second, nmake only command. I have five directories in Recycled and none of them show the initial success on the second command, so I'm now wondering if I missed it. Repeatedly. (grimace) Either way it fails building test here:
"cl" /Zi /Fdapp.pdb /Gs0 /GF /Gy /MD /W3 /wd4090 /nologo /O2 -I"include" -I"apps\include" -D"OPENSSL_BUILDING_OPENSSL" -D"OPENSSL_SYS_WIN32" -D"WIN32_LEAN_AND_MEAN" -D"UNICODE" -D"_UNICODE" -D"_CRT_SECURE_NO_DEPRECATE" -D"_WINSOCK_DEPRECATED_NO_WARNINGS" -D"NDEBUG" -c /Fotest\hmactest-bin-hmactest.obj "test\hmactest.c"
hmactest.c
cmd /C ""cl" /Zs /showIncludes /Zi /Fdapp.pdb /Gs0 /GF /Gy /MD /W3 /wd4090 /nologo /O2 -I"include" -I"apps\include" -D"OPENSSL_BUILDING_OPENSSL" -D"OPENSSL_SYS_WIN32" -D"WIN32_LEAN_AND_MEAN" -D"UNICODE" -D"_UNICODE" -D"_CRT_SECURE_NO_DEPRECATE" -D"_WINSOCK_DEPRECATED_NO_WARNINGS" -D"NDEBUG" "test\hmactest.c" > test\hmactest-bin-hmactest.d 2>&1"
IF EXIST test\hmactest.exe.manifest DEL /F /Q test\hmactest.exe.manifest
"link" /nologo /debug setargv.obj /subsystem:console /opt:ref /nologo /debug @C:\adminUserPath\AppData\Local\Temp\nm9E25.tmp
IF EXIST test\hmactest.exe.manifest "mt" -nologo -manifest test\hmactest.exe.manifest -outputresource:test\hmactest.exe
^--- Successful build of hmactest.exe ------ Unsuccessful build of http_test.exe ---v
"cl" /Zi /Fdapp.pdb /Gs0 /GF /Gy /MD /W3 /wd4090 /nologo /O2 -I"include" -I"apps\include" -D"OPENSSL_BUILDING_OPENSSL" -D"OPENSSL_SYS_WIN32" -D"WIN32_LEAN_AND_MEAN" -D"UNICODE" -D"_UNICODE" -D"_CRT_SECURE_NO_DEPRECATE" -D"_WINSOCK_DEPRECATED_NO_WARNINGS" -D"NDEBUG" -c /Fotest\http_test-bin-http_test.obj "test\http_test.c"
http_test.c
cmd /C ""cl" /Zs /showIncludes /Zi /Fdapp.pdb /Gs0 /GF /Gy /MD /W3 /wd4090 /nologo /O2 -I"include" -I"apps\include" -D"OPENSSL_BUILDING_OPENSSL" -D"OPENSSL_SYS_WIN32" -D"WIN32_LEAN_AND_MEAN" -D"UNICODE" -D"_UNICODE" -D"_CRT_SECURE_NO_DEPRECATE" -D"_WINSOCK_DEPRECATED_NO_WARNINGS" -D"NDEBUG" "test\http_test.c" > test\http_test-bin-http_test.d 2>&1"
IF EXIST test\http_test.exe.manifest DEL /F /Q test\http_test.exe.manifest
"link" /nologo /debug setargv.obj /subsystem:console /opt:ref /nologo /debug @C:\adminUserPath\AppData\Local\Temp\nmA182.tmp
test\http_test.exe : fatal error LNK1108: cannot write file at 0x0
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.34.31933\bin\HostX86\x86\link.EXE"' : return code '0x454'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.34.31933\bin\HostX86\x86\nmake.exe"' : return code '0x2'
Stop.
If I do nmake test it also fails in the same place, which one would expect. I've tried other targets found in the makefile but nothing will build this file. Everything I've read about FIPS indicates all tests must run successfully in order for it to be considered "compliant", so I can't skip this and drive on, or I would.
When I build on Linux (Debian 11) it works fine as frog hair, but ofc that won't run on windows, and even if I could cross-compile, FIPS compliance requires building ON the target platform. What I expect ofc is this same build/install to succeed on windows.
So far about 8 hrs of scouring the 'net have turned up nothing of note. I also appear to have lost access to my github account to open an issue with OpenSsl, which will be done as soon as they reply how to regain that access.
Adding /verbose to the LDFLAGS in the makefile generated massive gobs of output, then "Finished pass 2", then the error.
I notice the prior item in the makefile, hmactest.exe, has libcrypto_static where http_test has libcrypto. Changing that to libcrypt_static (two edits), changes the reported error to be "...cannot write file at 0xA0000" instead of 0x0.
I happen to have VS2019 available as well. Using it's CLI window as admin has no effect ... same error.
I noticed there's another another error from link.exe: 0x454. I have been utterly and completely unable to determine what that means or why it's reported.
dumpbin on the .obj files for hmactest & http_test showing nothing obvious, but I'm certainly no dump file guru, so any issue would have to be pretty blatant for me to catch it.
Thanks for your time...
Addendum: I haven't (completely) lost my marbles. Today the FIPS build does in fact "complete" on the second, 'nmake' command. I search the log I captured, and it shows the http_test target is run without error but I see NO http_test.exe in the output directory. Since it deletes http_test.exe.manifest, then links, then uses that manifest to create the .exe, I must conclude A) the linker creates the manifest file and B) even though it didn't toss an error, the link operation didn't create the manifest file like it should. Since there's an IF EXIST on the manifest -> executable line in the makefile, it quietly skips that. Mysteries:
Why does it rebuild the test directory when 'nmake install' is run, when it's already built all that with the prior 'nmake' execution? The most I would expect is to build the missing http_test.exe, not the entire test directory.
Why does the linker (usually) silently fail to build http_test.exe on the 'nmake' run, yet -always- fails with the 'nmake install' run? The sequence is the same: Delete manifest, link, create executable if manifest exists. It seems like it should silently fail to build http_test.exe on the 'nmake install' just like it did with 'nmake'.
