Hadoop build Failing in Windows: zconf.h is missing from native.sln?

Question

I'm trying to build the following hadoop version on development computer with Windows 10 Home Edition

hadoop-2.7.3-src

Here are the details about my local development environment:

-Windows 10 Home Edition

-Intel Core i5-6200U CPU @2.30GHz

-RAM 16 GB

-64-bit Operating System, x64-based processor

-Microsoft Visual Studio Community 2015 Version 14.0.25431.01 Update 3

-Also added MSBUILD location as C:\Program Files (x86)\MSBuild\14.0\Bin\amd64 to Windows System Environment Variable Path

-.NET Framework 4.6.01586

-cmake version 3.7.2

-CYGWIN_NT-10.0 LTPBCV82DUG 2.7.0(0.306/5/3) 2017-02-12 13:18 x86_64 Cygwin

-java version "1.8.0_121"

-Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

-Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

-Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)

-Google Protocol Buffers protoc --version libprotoc 2.5.0

Also, I've created and system environment variable called Platform and set it to x64

I opened up Developer Command Prompt for Visual Studio 2015 (VS2015)

c:\hadoop\hadoop-2.7.3-src> mvn package -Pdist,native-win -DskipTests -Dtar -X

Unfortunately, I'm getting the following error:

 [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
  ZlibDecompressor.c
c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
Done Building Project "C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default targets) -- FAILED.
Done Building Project "C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default targets) -- FAILED.

Build FAILED.

"C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default target) (1) ->
"C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default target) (2) ->
(ClCompile target) ->
  c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
  c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

    0 Warning(s)
    2 Error(s)

Time Elapsed 00:00:02.49

The aforementioned error has to do with zlib tool.

After researching online, someone said that the following Visual Studio solution file needs to be built successfully in Visual Studio:

....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln

Using Visual Studio 2015 in Administrator mode, I opened up the native.sln file, and immediately saw an error:

enter image description here

Could someone please tell me what steps I have to take to resolve said error?

Answer 1

So there were quite a few steps I had to take in order to resolve the problems.

Within the ....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\winutils directory, I opened up the following solution in Visual Studio 2015:

winutils.sln

Within .....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c , I commented out the following line of code, and made a modified copy of it as shown below:

//const WCHAR* wsceConfigRelativePath = WIDEN_STRING(STRINGIFY(WSCE_CONFIG_DIR)) L"\\" WIDEN_STRING(STRINGIFY(WSCE_CONFIG_FILE));

const WCHAR* wsceConfigRelativePath = WIDEN_STRING("../etc/hadoop") L"\\" WIDEN_STRING("wsce-site.xml");

Also, In the winutils solution's property window, I had to set the platform value to x64 as the screenshot below shows:

Next, I opened Dos command prompt, and checked the exact version of my Windows OS:

ver

Microsoft Windows [Version 10.0.14393]

Also, I opened up the property window of the libwinutils project, and ensured that properties that are marked in the following snapshot had the proper values:

Also, I took the same steps for the properties of the winutils project:

(Sorry, stackoverflow would not allow me to place another picture snapshot, but all you basically have to do is make sure the the winutils project's property are set properly )

I downloaded zlib version 1.2.11 source code. Using Developer Command Prompt for VS2015 ( Visual Studio 2015 ) I built zlib from zlib version 1.2.11 source code using cmake

c:\zlib\zlib-1.2.11>cmake  -G "Visual Studio 14 2015" -A x64 c:\zlib\zlib-1.2.11\
-- The C compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - not found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Configuring done
-- Generating done
-- Build files have been written to: C:/zlib/zlib-1.2.11

Finally, run the build with cmake

c:\zlib\zlib-1.2.11>cmake --build .

In Windows System Variables I have the following variable defined:

ZLIB_HOME is set to C:\zlib\zlib-1.2.11