1

I am trying to pass a large amount of data to my fragment shader, however when I try and compile and link the shader, glLinkProgram stalls and hangs forever (no errors are raised, and nothing happens except a black screen) The fragment shader successfully compiles, and gets attached successfully too. Stepping through with my debugger it hits the glLinkProgram call and then just stops, as if it is waiting for it to finish. If I set a smaller size for the arrays inside the structs, the program runs without hanging. I couldn't find anything in the docs around a hard internal structure array size limit so I'm curious what could be causing this?

shader.frag

#version 460 core
#extension GL_NV_gpu_shader5 : enable

// Program hanging is caused by this size
const int MAX_NODE_POOLS = 0xFFFF;

struct Node
{
    int data;
};

struct NodeMask
{
    uint8_t leaf;
    uint8_t nonLeaf;
};

struct NodePool
{
    Node mNodes[8];
};

struct Block
{
    uint16_t ID;
    NodePool mNodePools[MAX_NODE_POOLS];
    NodeMask mNodeMasks[MAX_NODE_POOLS];
};

layout (std430, binding=2) buffer octreeData
{
    Block blocks[];
};
...

shader.vert

#version 460 core
layout (location = 0) in vec2 position;

void main()
{
    gl_Position = vec4(position, 0.0, 1.0);
}

and my shader compilation code

bool Shader::loadShaders(const char *vertexPath, const char *fragmentPath)
{
    // 1. retrieve the vertex/fragment source code from filePath
    std::string vertexCode;
    std::string fragmentCode;
    std::ifstream vShaderFile;
    std::ifstream fShaderFile;
    bool err = false;

    vShaderFile.open(vertexPath);
    fShaderFile.open(fragmentPath);
    if(!vShaderFile.is_open())
    {
        fprintf(stderr, "Error opening vertex shader: %s\n", vertexPath);
        err = true;
    }
    if(!fShaderFile.is_open())
    {
        fprintf(stderr, "Error opening fragment shader: %s\n", fragmentPath);
        err = true;
    }
    

    std::stringstream vShaderStream, fShaderStream;
    // read file's buffer contents into streams
    vShaderStream << vShaderFile.rdbuf();
    fShaderStream << fShaderFile.rdbuf();
    // close file handlers
    vShaderFile.close();
    fShaderFile.close();
    // convert stream into string
    vertexCode = vShaderStream.str();
    fragmentCode = fShaderStream.str();

    const char* vShaderCode = vertexCode.c_str();
    const char * fShaderCode = fragmentCode.c_str();

    // 2. compile shaders
    unsigned int vertex, fragment;
    // vertex shader
    vertex = glCreateShader(GL_VERTEX_SHADER);
    glShaderSource(vertex, 1, &vShaderCode, NULL);
    glCompileShader(vertex);
    if(!checkCompileErrors(vertex, "VERTEX", std::string()))
        err = true;
    // fragment Shader
    fragment = glCreateShader(GL_FRAGMENT_SHADER);
    glShaderSource(fragment, 1, &fShaderCode, NULL);
    glCompileShader(fragment);
    if(!checkCompileErrors(fragment, "FRAGMENT", std::string()))
        err = true;
    // shader Program
    ID = glCreateProgram();
    glAttachShader(ID, vertex);
    glAttachShader(ID, fragment);
/*** Program hangs forever here ***/
    glLinkProgram(ID);
    if(!checkCompileErrors(ID, "PROGRAM", std::string()))
        err = true;
    // delete the shaders as they're linked into our program now and no longer necessery
    glDeleteShader(vertex);
    glDeleteShader(fragment);

    if(err)
        return false;

    return true;
}

Adding in my checkCompileErrors function to show we aren't accidentally missing linking/compilation errors.

GLint Shader::checkCompileErrors(GLuint shader, const std::string &type, const std::string &name)
{
    GLint success;
    GLchar infoLog[1024];
    if (type != "PROGRAM")
    {
        glGetShaderiv(shader, GL_COMPILE_STATUS, &success);
        if (!success)
        {
            glGetShaderInfoLog(shader, 1024, NULL, infoLog);
            std::cerr << name << std::endl;
            fprintf(stderr, "ERROR::SHADER_COMPILATION_ERROR of type: %s\n%s -- --------------------------------------------------- -- \n", type.c_str(), infoLog);
        }
    }
    else
    {
        glGetProgramiv(shader, GL_LINK_STATUS, &success);
        if (!success)
        {
            std::cerr << name << std::endl;
            glGetProgramInfoLog(shader, 1024, NULL, infoLog);
            fprintf(stderr, "ERROR::PROGRAM_LINKING_ERROR of type: %s\n%s -- --------------------------------------------------- -- \n", type.c_str(), infoLog);
        }
    }
    return success;     
}

Also querying my GL_MAX_SHADER_STORAGE_BLOCK_SIZE gives me a size of: 2147483647 bytes (2.15 GB). The data I'm trying to send over is 131.748 MB in size

What is the issue here? Obviously it is due to the large array size, but how can I work around this?

0x003
  • 65
  • 9
  • I can get it to compile completely fine on my 5700-xt with no linking errors or freezing with `MAX_NODE_POOLS` going all the way up to `0x7fffffff`, and it seems like from this [post](https://stackoverflow.com/questions/54526461/operations-on-shader-program-freezes-gl-context-when-using-big-enough-ssbo) that it might be a driver issue. – rdtsc Feb 03 '22 at 10:37
  • Far out that is *incredibly* annoying, thanks for the link – 0x003 Feb 03 '22 at 10:39
  • I've tried you code on a RTX3070 and I the following error through the debug callback: `Unknown internal debug message. The NVIDIA OpenGL driver has encountered an out of memory error. This application might behave inconsistently and fail.` I can see that the driver tried to allocate at least 6 GB of RAM. It doesn't freeze though, the whole run until the exception happens only takes a few seconds. – BDL Feb 03 '22 at 10:52
  • That is an absurd amount of RAM... this seems like a bug in the driver. I'm on linux FWIW, tried with official nvidia drivers 470 and just upgraded to 495 and get the same hanging issue. How do I go about reporting this? – 0x003 Feb 03 '22 at 10:55
  • You might be out of luck given NVIDIA's approach to drivers on Linux, but there is a forum post about how to report Linux problems [here](https://forums.developer.nvidia.com/t/if-you-have-a-problem-please-read-this-first/27131) – rdtsc Feb 03 '22 at 11:07
  • Interestingly if I set the MAX_NODE_POOLS size to 32768 it took 60 seconds to compile the shader on first run, then second run it took 30 milliseconds (code working as expected). I think this is definitely a driver bug. I am going to test out compiling with it at 65535 and see how long it takes. Interesting the difference between first time and second time compilation, do openGL shaders ever get cached after compilation? – 0x003 Feb 03 '22 at 11:40
  • It took 376324 milliseconds (376 seconds, or 6.2 minutes...) for it to compile on first execution with MAX_NODE_POOLS=0xFFFF. Quitting, and relaunching the application it took 63 milliseconds to compile. WTF! – 0x003 Feb 03 '22 at 11:50
  • 1
    Things like caching that don't affect the execution output of OpenGL programs are up to the driver vendor and in your case implemented, and the caching implementation may be what's causing the lag. It would be worth checking if disabling caching has any affect on the compile time. – rdtsc Feb 03 '22 at 13:47
  • 1
    Maybe the driver attempts to unroll some loops in your fragment shader which... would be unfortunate with that many items. – LJᛃ Feb 04 '22 at 14:24

0 Answers0