illegal instruction on aws batch

Question

I'm trying to submit a demo job to batch compute that tests if I can properly use a python module "cppyy".

However, I received "illegal instruction" core dump error. Interestingly, I didn't receive any error message if I run the job at a container (same image) run on a local ec2 instance.

The following the script, test.py used for the entry point, python3 test.py

import cppyy
  
cppyy.cppdef(r'''
#include <cmath>
#include <vector>
#include <memory>
#include <iostream>


using namespace std;


namespace Test {
    void test() {
        cout << "test-yoyo-5" << endl;

        unsigned long long timestamp = 1629072000081;

        vector<shared_ptr<unsigned long long>> events;
        events.push_back(make_shared<unsigned long long>(1629072000081));

        for (auto &event: events) {
            cout << "static_cast<double>(event->timestamp - timestamp): " << static_cast<double>(*event - timestamp) << endl;
        }
    }
}
''')

from cppyy.gbl import Test

Test.test()

The core dump error follows

*** Break *** illegal instruction
test-yoyo-5
 Generating stack trace...
 0x00007f4812ecaf43 in <unknown> from /usr/local/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so
 0x00007f48109bae39 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098b879 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098d249 in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) at /tmp/pip-install-vu_2b778/cpycppyy_c9bff3ea1a5d49d695ef46619c091741/src/CPPMethod.cxx:892 from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098782f in CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) at /tmp/pip-install-vu_2b778/cpycppyy_c9bff3ea1a5d49d695ef46619c091741/src/CPPFunction.cxx:90 from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f4810992fd1 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x000055e4b7135a2a in _PyEval_EvalFrameDefault + 0x6b8a from python3
 0x000055e4b71f6184 in <unknown> from python3
 0x000055e4b71f64aa in PyEval_EvalCode + 0x3a from python3
 0x000055e4b72365ac in <unknown> from python3
 0x000055e4b72385f0 in PyRun_SimpleFileExFlags + 0x1a0 from python3
 0x000055e4b7138bef in <unknown> from python3
 0x000055e4b7139290 in Py_BytesMain + 0x70 from python3
 0x00007f4817d45d90 in <unknown> from /lib/x86_64-linux-gnu/libc.so.6
 0x00007f4817d45e40 in __libc_start_main + 0x80 from /lib/x86_64-linux-gnu/libc.so.6
 0x000055e4b7137e75 in _start + 0x25 from python3
 *** Break *** illegal instruction
 Generating stack trace...
 0x00007f4812ecaf43 in <unknown> from /usr/local/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so
 0x00007f48109bae39 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098b879 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098d249 in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) at /tmp/pip-install-vu_2b778/cpycppyy_c9bff3ea1a5d49d695ef46619c091741/src/CPPMethod.cxx:892 from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f481098782f in CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) at /tmp/pip-install-vu_2b778/cpycppyy_c9bff3ea1a5d49d695ef46619c091741/src/CPPFunction.cxx:90 from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x00007f4810992fd1 in <unknown> from /usr/local/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so
 0x000055e4b7135a2a in _PyEval_EvalFrameDefault + 0x6b8a from python3
 0x000055e4b71f6184 in <unknown> from python3
 0x000055e4b71f64aa in PyEval_EvalCode + 0x3a from python3
 0x000055e4b72365ac in <unknown> from python3
 0x000055e4b72385f0 in PyRun_SimpleFileExFlags + 0x1a0 from python3
 0x000055e4b7138bef in <unknown> from python3
 0x000055e4b7139290 in Py_BytesMain + 0x70 from python3
 0x00007f4817d45d90 in <unknown> from /lib/x86_64-linux-gnu/libc.so.6
 0x00007f4817d45e40 in __libc_start_main + 0x80 from /lib/x86_64-linux-gnu/libc.so.6
 0x000055e4b7137e75 in _start + 0x25 from python3
static_cast<double>(event->timestamp - timestamp):

Its emitting an instruction not supported by the cpu its running on. — Taekahn, Sep 26 '22 at 03:36
@Taekahn Thanks for replying. Why `cppyy` would lead to cpu-unsupported instructions? I download python, C++ compiler all via legit ways. — Max Wong, Sep 26 '22 at 20:05
Without knowing what the instruction is, I can only speculate. Either it’s too new of an instruction, too old of an instruction, or it’s a company specific instruction (intel vs amd) — Taekahn, Sep 26 '22 at 22:10
@Taekahn Yes. I just ran cat `/proc/cpuinfo | grep "model name"` on my local instance, and it reports `model name : Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz`. Whereas on batch, it reports `model name : AMD EPYC 7R13 Processor` — Max Wong, Sep 27 '22 at 22:00

score 2 · Accepted Answer · answered Sep 29 '22 at 01:00

You're not specifying the version of cppyy, but most likely the cause is the precompiled header (PCH). cppyy by default uses the -march=native flag, which enables (if the host supports it) such things as AVX. If you then ship an instance with the PCH to a machine that doesn't support it, any JITted code may still use AVX b/c a compile-time variable __AVX__ will be baked into it.

(Older versions of cppyy explicitly use -mavx if available on the host.)

When running in a heterogeneous environment, you can set CLING_REBUILD_PCH=1 to get around this, or even CLING_STANDARD_PCH=none to have no PCH altogether as a PCH only makes sense when cppyy is imported multiple times from different processes (it only saves on startup time).

illegal instruction on aws batch

1 Answers1