356

While running a C program, It says "(core dumped)" but I can't see any files under the current path.

I have set and verified the ulimit:

ulimit -c unlimited 
ulimit -a

I also tried to find a file named "core", but didn't get the core dumped file?
Any help, where is my core file?

kenorb
  • 155,785
  • 88
  • 678
  • 743
webminal.org
  • 44,948
  • 37
  • 94
  • 125

15 Answers15

285

Read /usr/src/linux/Documentation/sysctl/kernel.txt.

core_pattern is used to specify a core dumpfile pattern name.

  • If the first character of the pattern is a '|', the kernel will treat the rest of the pattern as a command to run. The core dump will be written to the standard input of that program instead of to a file.

Instead of writing the core dump to disk, your system is configured to send it to the abrt (meaning: Automated Bug Reporting Tool, not "abort") program instead. Automated Bug Reporting Tool is possibly not as documented as it should be...

In any case, the quick answer is that you should be able to find your core file in /var/cache/abrt, where abrt stores it after being invoked. Similarly, other systems using Apport may squirrel away cores in /var/crash, and so on.

Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265
ephemient
  • 198,619
  • 38
  • 280
  • 391
  • 41
    yes,I have edited core_pattern with following content echo "core.%e.%p" > /proc/sys/kernel/core_pattern ..now it creates the core dump file in current directory itself. with name "core.giis.12344" etc. Thank you all for your answers/comments/hints . – webminal.org Jan 15 '10 at 06:57
  • 24
    Just to note that fedora 18 abrt program is storing core dumps in `/var/spool/abrt/` instead of `/var/cache/abrt` – Nelson Mar 06 '13 at 09:31
  • 4
    Is there a way to allow users to configure this for themselves, rather than everyone having to use the system configuration? – allyourcode Oct 17 '13 at 19:57
  • When this command is run and process is invoked, are stdout and stderr not opened by default? I see very strange things happenning. when i use dup2 of stderr and stdout into my custom file(applog.txt), Data which i am writing into other file(mycore.BIN) is being redirected to file which i have used to catch stdout & stderr(applog.txt). Is there a good read about this one? Please suggest. Thank you – Sandeep Jun 02 '14 at 07:53
  • 32
    systemd in archlinux store coredumps in `/var/lib/systemd/coredump/` – Francois Jun 21 '16 at 08:46
  • systemd in debian as well uses `/var/lib/systemd/coredump/` – Alex Jul 28 '18 at 19:24
266

On recent Ubuntu (12.04 in my case), it's possible for "Segmentation fault (core dumped)" to be printed, but no core file produced where you might expect one (for instance for a locally compiled program).

This can happen if you have a core file size ulimit of 0 (you haven't done ulimit -c unlimited) -- this is the default on Ubuntu. Normally that would suppress the "(core dumped)", cluing you into your mistake, but on Ubuntu, corefiles are piped to Apport (Ubuntu's crash reporting system) via /proc/sys/kernel/core_pattern, and this seems to cause the misleading message.

If Apport discovers that the program in question is not one it should be reporting crashes for (which you can see happening in /var/log/apport.log), it falls back to simulating the default kernel behaviour of putting a core file in the cwd (this is done in the script /usr/share/apport/apport). This includes honouring ulimit, in which case it does nothing. But (I assume) as far as the kernel is concerned, a corefile was generated (and piped to apport), hence the message "Segmentation fault (core dumped)".

Ultimately PEBKAC for forgetting to set ulimit, but the misleading message had me thinking I was going mad for a while, wondering what was eating my corefiles.

(Also, in general, the core(5) manual page -- man 5 core -- is a good reference for where your core file ends up and reasons it might not be written.)

Marcin Orlowski
  • 72,056
  • 11
  • 123
  • 141
jtn
  • 2,961
  • 1
  • 12
  • 3
  • 4
    Thank you so much -- I ran into the very same problem. Btw, Ubuntu 14.04 exhibits exactly the same behavior as the one you described. – Malte Skoruppa Jul 22 '15 at 10:19
  • 8
    On my Ubuntu install (modified 14.04), there's an easy temporary workaround for this by running `sudo service apport stop` --- after I ran that, it changed `/proc/sys/kernel/core_pattern` from the apport pipe to just `core`. Apport is smart enough to fix up the `core_pattern` temporarily, I suppose. – Patrick Collins Feb 22 '16 at 18:50
  • 8
    "ulimit -c unlimited" is exactly what I needed - Thank you! – Dave C Mar 20 '16 at 18:07
  • Recently, ubuntu writes the core to `~/temp/core`, after you have raised `ulimit`. Check the log on `var/log/apport.log` (yes, you need to have su powers to read that file). – Karl Dec 18 '16 at 12:34
  • 6
    I have tried every applicable answer, and every location in every comment here after doing the ulimit command, but still can't find a core file anywhere on my Ubuntu 16.04 LTS... – Nagev Apr 05 '17 at 13:39
  • 1
    @Nagev, I am facing the same issue with Ubuntu 14.04. Have you ever found a solution? – nico Aug 16 '17 at 15:19
  • 7
    @ElectricGoat No I haven't. It's poor UX design, IMO. The system should just print the path, or not print a message at all if it's not being created. I'll add my own answer, when or if I get a chance to investigate this further. – Nagev Aug 17 '17 at 07:29
  • thanks, solved it this way : echo core | sudo tee /proc/sys/kernel/core_pattern – Willy Tarreau Apr 15 '18 at 20:26
  • 1
    Brilliant! On Ubuntu 16.04, `/var/log/apport.log` contained `host pid [...] crashed in a separate mount namespace, ignoring` (a quick search reveals nothing useful regarding this message, nor how to work around it), and stopping the Apport service temporarily with `systemctl stop apport` allowed the core dump to be written to its expected location. – Ben Johnson May 11 '18 at 14:48
  • I'm surprised about the many up votes: The OP said core size is unlimited. Besides of that the is **no* message `core dumped` if no core was dumped! Most likely today is that `systemd` interfered with core dump (redirecting the core file). Easiest way to find out is reading the syslog (which is always a good idea after problems, BTW). – U. Windl Jan 05 '22 at 10:24
100

With the launch of systemd, there's another scenario as well. By default systemd will store core dumps in its journal, being accessible with the systemd-coredumpctl command. Defined in the core_pattern-file:

$ cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e

Easiest way to check for stored core dumps is via coredumpctl list (older core dumps may have been removed automatically). This behaviour can be disabled with a simple "hack":

$ ln -s /dev/null /etc/sysctl.d/50-coredump.conf
$ sysctl -w kernel.core_pattern=core      # or just reboot

As always, the size of core dumps has to be equal or higher than the size of the core that is being dumped, as done by for example ulimit -c unlimited.

U. Windl
  • 3,480
  • 26
  • 54
timss
  • 9,982
  • 4
  • 34
  • 56
  • That "hack" didn't work for me (even after a restart). I'm running Arch Linux and I've already run `ulimit -c unlimited`. – gsgx Oct 16 '13 at 05:15
  • @gsingh2011 It might be outdated. I don't run Arch anymore so I can't say if it should work these days. If you figure it out feel free to update me/us with a new comment. – timss Oct 16 '13 at 10:58
  • 6
    @gsingh2011 Try `50-coredump.conf` instead of `coredump.conf`. This should override `/lib/sysctl.d/50-coredump.conf`. The default can be restored with `sysctl -w kernel.core_pattern=core` – Lekensteyn Nov 15 '13 at 10:07
  • I had to turn off appport for this to work ```sudo service apport stop``` – rahul003 Feb 02 '18 at 00:06
  • On Ubuntu 21.10 I had to install `systemd-coredump` first, before I could get the `coredumpctl` command to work. After that it was easy ;) – oligofren Apr 27 '22 at 10:48
94

Writing instructions to get a core dump under Ubuntu 16.04 LTS:

  1. As @jtn has mentioned in his answer, Ubuntu delegates the display of crashes to apport, which in turn refuses to write the dump because the program is not an installed package. Before making changes

  2. To remedy the problem, we need to make sure apport writes core dump files for non-package programs as well. To do so, create a file named ~/.config/apport/settings with the following contents:
    [main] unpackaged=true

  3. Now crash your program again, and see your crash files being generated within folder: /var/crash with names like *.1000.crash. Note that these files cannot be read by gdb directly. After making changes

  4. [Optional] To make the dumps readble by gdb, run the following command:

    apport-unpack <location_of_report> <target_directory>

References: Core_dump – Oracle VM VirtualBox

ankurrc
  • 1,072
  • 9
  • 12
13

I could think of two following possibilities:

  1. As others have already pointed out, the program might chdir(). Is the user running the program allowed to write into the directory it chdir()'ed to? If not, it cannot create the core dump.

  2. For some weird reason the core dump isn't named core.* You can check /proc/sys/kernel/core_pattern for that. Also, the find command you named wouldn't find a typical core dump. You should use find / -name "*core.*", as the typical name of the coredump is core.$PID

timss
  • 9,982
  • 4
  • 34
  • 56
ahans
  • 1,697
  • 12
  • 19
  • here is my pattern - does this mean core file is named something like"PID.signal.userid" instead of core.pid ??? $cat /proc/sys/kernel/core_pattern /usr/lib/hookCCpp /var/char/abrt %p %s %u – webminal.org Jan 14 '10 at 17:55
  • 1
    If contents of virtual file `/proc/sys/kernel/core_pattern` starts with a pipe symbol `|` the file will not be written to file defined by `core_pattern` but the *command* defined after the the pipe symbol is launched and the core file will be written to stdin of that program. It's 100% up to the program to handle the core file contents after that. – Mikko Rantalainen Sep 20 '21 at 16:53
  • On item 1: No, the program won't core dump; it would see an `EPERM`. Only if the program fails to handle the error correctly it may be forced to terminate (to prevent even more nonsense to happen). – U. Windl Jan 05 '22 at 10:28
11

In Ubuntu18.04, the most easist way to get a core file is inputing the command below to stop the apport service.

sudo service apport stop

Then rerun the application, you will get dump file in current directory.

Nicolas Gong
  • 520
  • 5
  • 9
  • I tried this and got `/etc/init.d/apport: 68: /etc/init.d/apport: cannot create /proc/sys/fs/suid_dumpable: Operation not permitted` still cannot find the crash files. I have already set `ulimit -c unlimited`. Somebody mentioned the `/proc/sys/kernel/core_pattern`, I can even not see this file. What else should be done? – doraemon Nov 07 '20 at 03:15
  • sorry, since I have not encountered the problem, I cannot give a good answer for you. However, I think you can learn the knowledge of apport service, then you can know where the core dump file. Here is the document of apport: `https://wiki.ubuntu.com/Apport`. – Nicolas Gong Nov 16 '20 at 03:43
9

If you're missing core dumps for binaries on RHEL and when using abrt, make sure that /etc/abrt/abrt-action-save-package-data.conf

contains 

ProcessUnpackaged = yes

This enables the creation of crash reports (including core dumps) for binaries which are not part of installed packages (e.g. locally built).

MRalwasser
  • 15,605
  • 15
  • 101
  • 147
  • You miss the fact that when "(core dumped)" is output, a core was dumped; if no core was dumped, you do not see that message. – U. Windl Jan 05 '22 at 10:29
8

For fedora25, I could find core file at 

/var/spool/abrt/ccpp-2017-02-16-16:36:51-2974/coredump

where ccpp-2017-02-16-16:36:51-2974" is pattern "%s %c %p %u %g %t %P % as per `/proc/sys/kernel/core_pattern'

Mahendra Gunawardena
  • 1,956
  • 5
  • 26
  • 45
mahaveer darade
  • 81
  • 1
  • 1
6

In my case, the reason is that the ulimit command only effect the current terminal.

If I set ulimit -c unlimited on the first terminal. Then I start a new terminal to run the program. It will not generate the core file when core dumped.

You have to confirm the core size of the terminal which runs your program.

The following steps work on ubuntu 20.04 and ubuntu 21.04:

  1. stop apport service
sudo service apport stop
  1. set core size of the terminal which ready to run your program
ulimit -c unlimited
lylhw13
  • 101
  • 2
  • 6
5

My efforts in WSL have been unsuccessful.

For those running on Windows Subsystem for Linux (WSL) there seems to be an open issue at this time for missing core dump files.

The comments indicate that

This is a known issue that we are aware of, it is something we are investigating.

Github issue

Windows Developer Feedback

Brandon Culley
  • 5,219
  • 1
  • 28
  • 28
5

I'm on Linux Mint 19 (Ubuntu 18 based). I wanted to have coredump files in current folder. I had to do two things:

  1. Change /proc/sys/kernel/core_pattern (by # echo "core.%p.%s.%c.%d.%P > /proc/sys/kernel/core_pattern or by # sysctl -w kernel.core_pattern=core.%p.%s.%c.%d.%P)
  2. Raising limit for core file size by $ ulimit -c unlimited

That was written already in the answers, but I wrote to summarize succinctly. Interestingly changing limit did not require root privileges (as per https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user non-root can only lower the limit, so that was unexpected - comments about it are welcome).

Marisha
  • 816
  • 9
  • 14
5

I found core files of my Ubuntu 20.04 system at;

/var/lib/apport/coredump
Hwanyong Lee
  • 51
  • 1
  • 1
3

ulimit -c unlimited made the core file correctly appear in the current directory after a "core dumped".

Nicolas VERHELST
  • 398
  • 3
  • 13
3

If you use Fedora, in order to generate core dump file in the same directory of binary file:

echo "core.%e.%p" > /proc/sys/kernel/core_pattern

And

ulimit -c unlimited
Massimiliano Di Cavio
  • 109
  • 1
  • 11
  • This is fine in itself, but if you want to work _with_ the existing system setup, instead of against it, try to figure out how it is setup and inspect the docs for whatever command is capturing the dumps to get to them. See https://stackoverflow.com/a/14082174/200987 for details. – oligofren Apr 27 '22 at 10:51
0

A one-liner to get the latest core dump path:

ls -t $(cat /proc/sys/kernel/core_pattern | awk -F% '{print $1"*"}') 2>/dev/null | head -1

You can of course modify the last -1 on that line to e.g. -4 to get the last 4 core dumps.

Note: That's not expected to work e.g. in case the path pattern uses variables before the last / or when non core dump files are on that dir.

elad4
  • 1
  • 1