How to Practically Ship GLSL Shaders with your C++ Software

Question

During OpenGL initialization, the program is supposed to do something like:

<Get Shader Source Code>
<Create Shader>
<Attach Source Code To Shader>
<Compile Shader>

Getting source code could be as simple as putting it in a string like: (Example taken from SuperBible, 6th Edition)

static const char * vs_source[] =
{
    "#version 420 core                             \n"
    "                                              \n"
    "void main(void)                               \n"
    "{                                             \n"
    "    gl_Position = vec4(0.0, 0.0, 0.0, 1.0);   \n"
    "}                                             \n"
};

The problem is that it is hard to edit, debug and maintain GLSL shaders directly in a string. So getting the source code in a string from a file is easier for development:

std::ifstream vertexShaderFile("vertex.glsl");
std::ostringstream vertexBuffer;
vertexBuffer << vertexShaderFile.rdbuf();
std::string vertexBufferStr = vertexBuffer.str();
// Warning: safe only until vertexBufferStr is destroyed or modified
const GLchar *vertexSource = vertexBufferStr.c_str();

The problem now is how to ship the shaders with your program? Indeed, shipping source code with your application may be a problem. OpenGL supports "pre-compiled binary shaders" but the Open Wiki states that:

Program binary formats are not intended to be transmitted. It is not reasonable to expect different hardware vendors to accept the same binary formats. It is not reasonable to expect different hardware from the same vendor to accept the same binary formats. [...]

How to practically ship GLSL shaders with your C++ software?

Answer 1

With c++11, you can also use the new feature of raw string literals. Put this source code in a separate file named shader.vs:

R"(
#version 420 core

void main(void)
{
    gl_Position = vec4(0.0, 0.0, 0.0, 1.0);
}
)"

and then import it as a string like this:

const std::string vs_source =
#include "shader.vs"
;

The advantage is that its easy to maintain and debug, and you get correct line numbers in case of errors from the OpenGL shader compiler. And you still don't need to ship separate shaders.

The only disadvantage I can see is the added lines on the top and bottom of the file (R") and )") and the syntax that is a little bit strange for getting the string into C++ code.

Answer 2

There is just "store them directly in the executable" or "store them in (a) separate file(s)", with nothing in-between. If you want a self-contained executable, putting them into the binary is a good idea. Note that you can add them as resources or tweak your build system to embed the shader strings from separate development files into source files to make development easier (with the possible addition of being able to directly load the separate files in development builds).

Why do you think shipping the shader sources would be a problem? There is simply no other way in the GL. The precompiled binaries are only useful for caching the compilation results on the target machine. With the fast advances of GPU technology, and changing GPU architectures, and different vendors with totally incompatible ISAs, precompiled shader binaries do not make sense at all.

Note that putting your shader sources in the executeable does not "protect" them, even if you encrypt them. A user can still hook into the GL library and intercept the sources you specify to the GL. And the GL debuggers out there do exactly that.

UPDATE 2016

At SIGGRAPH 2016, the OpenGL Architecture Review Board released the GL_ARB_gl_spirv extension. This will allow a GL inmplementation to use the SPIRV binary intermediate language. This has some potential benefits:

1. Shaders can be pre-"compiled" offline (the final compilation for the target GPU still happens by the driver later). You don't have to ship the shader source code, but only the binary intermediate representation.
2. There is one standard compiler frontend (glslang) which does the parsing, so differences between the parsers of different implementations can be eliminated.
3. More shader languages can be added, without the need to change the GL implementations.
4. It somewhat increases portability to vulkan.

With that scheme, GL is becoming more similar to D3D and Vulkan in that regard. However, it doesn't change the greater picture. The SPIRV bytecode can still be intercepted, disassembled and decompiled. It does make reverse-engineering a little bit harder, but not by much actually. In a shader, you usually can't afford extensive obfuscuation measures, since that dramatically reduces performance - which is contrary to what the shaders are for.

Also keep in mind that this extension is not widely available right now (autumn 2016). And Apple has stopped supporting GL after 4.1, so this extension will probably never come to OSX.

MINOR UPDATE 2017

GL_ARB_gl_spirv is now official core feature of OpenGL 4.6, so that we can expect growing adoption rate for this feature, but it doesn't change the bigger picture by much.

Answer 3

OpenGL supports pre-compiled binaries, but not portably. Unlike HLSL, which is compiled into a standard bytcode format by Microsoft's compiler and later translated into a GPU's native instruction set by the driver, OpenGL has no such format. You cannot use pre-compiled binaries for anything more than caching compiled GLSL shaders on a single machine to speed-up load time, and even then there is no guarantee that the compiled binary will work if the driver version changes... much less the actual GPU on the machine changes.

You can always obfuscate your shaders if you are really paranoid. The thing is, unless you are doing something truly one-of-a-kind nobody is going to care about your shaders and I mean that genuinely. This industry thrives on openness, all the big players in the industry regularly discuss the newest and most interesting techniques at conferences such as GDC, SIGGRAPH, etc. In fact, shaders are so implementation-specific that often there is not much you can do from reverse engineering them that you could not do just by listening to one of said conferences.

If your concern is people modifying your software, then I would suggest you implement a simple hash or checksum test. Many games already do this to prevent cheating, how far you want to take it is up to you. But the bottom line is that binary shaders in OpenGL are meant to reduce shader compile time, not for portable re-distribution.

Answer 4

My suggestion would be to make the incorporation of shader into your binary a part of your build process. I use CMake in my code to scan a folder for shader source files and then generate a header with an enum of all the available shaders:

#pragma once
enum ShaderResource {
    LIT_VS,
    LIT_FS,
    // ... 
    NO_SHADER
};

const std::string & getShaderPath(ShaderResource shader);

Similarly, CMake creates a CPP file which, given a resource, returns the file path to the shader.

const string & getShaderPath(ShaderResource res) {
  static map<ShaderResource, string> fileMap;
  static bool init = true;
  if (init) {
   init = false;
   fileMap[LIT_VS] =
    "C:/Users/bdavis/Git/OculusRiftExamples/source/common/Lit.vs";
   // ...
  }
  return fileMap[res];
}

It wouldn't be too hard (much handwaving here) to make the CMake script alter it's behavior so that in a release build instead of providing the file path it provided the source of the shader, and in the cpp file stored the contents of the shaders themselves (or in the case of a Windows or Apple target make them part of the executable resources / executable bundle).

The advantage of this approach is that it's much easier to modify the shaders on the fly during debugging if they're not baked into the executable. In fact my GLSL program fetching code actually looks at the compile time of the shader versus the modified timestamps of the source files and will reload the shader if the files have changed since the last time it was compiled (this is still in its infancy, since it means you lose any uniforms that were previously bound to the shader, but I'm working on that).

This is really less of a shader issue than a generic 'non-C++ resources' issue. The same problem exists with everything you might want to load and process... images for textures, sound files, levels, what have you.

Answer 5

As an alternative of keeping GLSL shaders directly in a string, I would suggest considering this library that I'm developing: ShaderBoiler (Apache-2.0).

It is in alpha version and has some limitations that may restrict usage of it.

The main concept is to write in C++ constructs similar to GLSL code, that would construct a computation graph from which the final GLSL code is generated.

For example, let's consider the following C++ code

#include <shaderboiler.h>
#include <iostream>

void main()
{
    using namespace sb;

    context ctx;
    vec3 AlbedoColor           = ctx.uniform<vec3>("AlbedoColor");
    vec3 AmbientLightColor     = ctx.uniform<vec3>("AmbientLightColor");
    vec3 DirectLightColor      = ctx.uniform<vec3>("DirectLightColor");
    vec3 LightPosition         = ctx.uniform<vec3>("LightPosition");

    vec3 normal   = ctx.in<vec3>("normal");
    vec3 position = ctx.in<vec3>("position");
    vec4& color   = ctx.out<vec4>("color");

    vec3 normalized_normal = normalize(normal);

    vec3 fragmentToLight = LightPosition - position;

    Float squaredDistance = dot(fragmentToLight, fragmentToLight);

    vec3 normalized_fragmentToLight = fragmentToLight / sqrt(squaredDistance);

    Float NdotL = dot(normal, normalized_fragmentToLight);

    vec3 DiffuseTerm = max(NdotL, 0.0) * DirectLightColor / squaredDistance;

    color = vec4(AlbedoColor * (AmbientLightColor + DiffuseTerm), 1.0);

    std::cout << ctx.genShader();
}

The output to the console will be:

uniform vec3 AlbedoColor;
uniform vec3 AmbientLightColor;
uniform vec3 LightPosition;
uniform vec3 DirectLightColor;

in vec3 normal;
in vec3 position;

out vec4 color;

void main(void)
{
        vec3 sb_b = LightPosition - position;
        float sb_a = dot(sb_b, sb_b);
        color = vec4(AlbedoColor * (AmbientLightColor + max(dot(normal, sb_b / sqrt(sb_a)), 0.0000000) * DirectLightColor / sb_a), 1.000000);
}

The created string with GLSL code can be used with OpenGL API to create shader.

Answer 6

The problem is that it is hard to edit, debug and maintain GLSL shaders directly in a string.

It's strange that this sentence has been totally ignored by all 'answers' so far, while the recurring theme of those answers has been, "You can't solve the problem; just deal with it."

The answer to making them easier to edit, while loading them directly from a string, is simple. Consider the following string literal:

    const char* gonFrag1 = R"(#version 330
// Shader code goes here
// and newlines are fine, too!)";

All other comments are correct as far as they go. Indeed, as they say, the best security available is obscurity, since GL can be intercepted. But to keep honest people honest, and to put some block in the way of accidental program damage, you can do as above in C++, and still easily maintain your code.

Of course if you DID want to protect the world's most revolutionary shader from theft, obscurity could be taken to rather effective extremes. But that's another question for another thread.

Answer 7

You can also combine multiple shader sources into one file (or string) using preprocessor directives if you don't want to keep them separate. This also lets you avoid repetition (e.g. shared declarations) – unused variables are optimized by the compiler most of the time.

See http://www.gamedev.net/topic/651404-shaders-glsl-in-one-file-is-it-practical/

Answer 8

A suggestion:

In your program, put the shader in:

const char shader_code = {
#include "shader_code.data"
, 0x00};

In shader_code.data there should be the shader source code as a list o hex numbers separated by commas. These files should be created before compilation using your shader code written normally in a file. In Linux I would put instructions at Makefile to run the code:

cat shader_code.glsl | xxd -i > shader_code.data

Answer 9

I don`t know if that will work, but you could embed the .vs file into your executable with binutils like program like g2bin, and you can declare your shader programs as externals then you access them as normal resources embedded in the executable. See qrc in Qt, or you can view my small program for embedding stuff in executables here: https://github.com/heatblazer/binutil which is invoked as pre-build command to the IDE.

Answer 10

Another alternative to storing glsl text files or precompiled glsl files is a shader generator, which takes a shade tree as input and outputs glsl (or hlsl, ...) code, that is then compiled and linked at runtime... Following this approach you can more easily adapt to whatever capabilities the gfx hardware has. You can also support hlsl, if you have lots of time, no need for the cg shading language. If you think about glsl/hlsl deeply enough you will see, that transforming shade trees into source code was at the back of the language designers minds.

Answer 11

In C99/C11 you can do it in 2 simple steps.

## Bash build script:

## STEP #1: Convert your [C99/C11] code to GLSL by quoting it:

    awk 'NF { print "\""$0"\\""n""\""}' GLSL_AS_C99.C11 > QUOTED.TXT

## STEP #2: Compile your project:

    gcc -x c -c MY_PROJECT_FILE.C11 -o object_file.o -std=c11 -m64
    gcc -o EXE.exe object_file.o
    rm object_file.o
    ./EXE.exe
    rm EXE.exe

Yes. There is more to it. You have to write your C99 in a generic style that will also compile as GLSL. For example:

#ifdef THIS_IS_BEING_COMPILED_AS_OPEN_GL_SHADER_CODE
    #define I32 int
#endif
#ifdef THIS_IS_BEING_COMPILED_AS_C99_CODE
    #define I32 uint32_t
#endif

Code written in such a way in C99 can be cut and pasted into GLSL shader code with no problems. It also allows you to unit test your GLSL shader code. To include the code that was stringified by the AWK command do something like this:

//:AAC2020_PAINT5D_DEFAULT_001:==============================://
const char* AAC2020_PAINT5D_DEFAULT_001=( //:////////////////://
//://////////////////////////////////////////////////////////://
"#version 330 core                           \n"//://////////://
"#define AAC2020_MACRO_THIS_IS_OPEN_GL (1)   \n"//://////////://
//://////////////////////////////////////////////////////////://
//|Everything Below Is Cut+Pasted From       |////://////////://
//|The C99 File: P5D_OGL._                   |////://////////://
//://////////////////////////////////////////////////////////://

    #include "../QUOTED.TXT"
                                                
); //:///////////////////////////////////////////////////////://
//:==============================:AAC2020_PAINT5D_DEFAULT_001://

If you are familiar with C code and bash script. That should be enough to explain it. But if you require more explanation I filmed a 30 minute demonstration and explanation video on youtube.

https://www.youtube.com/watch?v=kQfSL4kv5k0&list=PLN4rUakF78aCdRxjMU8_JBGAKIrtt_7N5&index=115

If you would prefer WORKING CODE... Here is my game engine build that uses this system. Open up the AAC2020.SH build script to find the "awk" command and work backwards from there.

https://github.com/KanjiCoder/AAC2020

Other files of specific interest to the problem at hand are:

1. P5D1OGL.FRA._ <-- The C99 source code that can be ran as GLSL and unit tested
2. P5D1OGL.FRA.STRING._ <-- P5D1OGL.FRA._ quoted by the awk command.
3. P5D_001._ <-- P5D1OGL.FRA.STRING._ #included into a const char* to embed into exe
4. POLYOGL.D._ <-- OpenGL Polyfills .D (data / structs)
5. POLYOGL.F._ <-- OpenGL Polyfills .F (functions)

Alternatively, if you tune into my twitch stream and request an in-person overview of how I do this, I can give you a live demo and further explanation.

www.twitch.com/kanjicoder

You can also email me at: HeavyMetalCookies@Gmail.com I am currently 35 and the year is 2021. If I don't reply it means I am either dead , too famous to answer, or both. I'll leave it as an exercise to the reader to figure out which one of those it is.

-John Mark