5

Ok so have a fairly huge in house system. We have a nice friendly host who generally work with us keeping things working if something ever goes wrong, or at least points us in the right direction.

A while ago we had to take a copy of the code and host it on a clients external dedicated server, hosted with Rackspace. Considering everything was working fine for a number of months, all of a sudden the server kept throwing hissy-fits and segmentation faults eg

[notice] child pid 8628 exit signal Segmentation fault (11)

Even though no code had been changed, the hosts have said its probably a PHP issue and the devs (us) would have to look into it. I haven't even seen a problem, just that at certain times when the apache logs rotate (late at night) this is causing it to fall over.

I got them to enable core dumps, and I am armed with a bunch of them, and Fixing Apache Segmentation Faults Caused by PHP, but rackspace seem to have moved all the php and apache source dirs to god knows where, I cant find .gdbinit anywhere on the server using 'find', and about all I can get out of gdb is

Program terminated with signal 11, Segmentation fault.
#0  0x00002b8d6fc36c60 in ?? ()

...which couldnt be less helpful if it tried!!!

Any ideas on where to start with tracking down this supposed problem in my code?

Horse
  • 3,023
  • 5
  • 38
  • 65
  • 1
    We had this same issue. Apache tries to do a graceful restart when rotating logs. It doesn't always come back cleanly. I've installed cronolog so that Apache doesn't have to shut down just to rotate the logs. This might be a PHP issue, but it works right up until Apache restarts... so... – nwalke Oct 18 '12 at 15:51
  • 1
    Take a look at this answer: http://stackoverflow.com/questions/7745578/notice-child-pid-3580-exit-signal-segmentation-fault-11-in-apache-error-l – Udan Oct 18 '12 at 15:52
  • @tubaguy50035 ... soooo its rackspaces problem you mean? – Horse Oct 18 '12 at 16:07
  • I don't want to say it's anyone's fault. But whoever is hosting Apache should consider settings things up so that it doesn't have to restart. Our issue didn't happen everyday, it was completely random, but it was always after Apache tried to restart gracefully. Turned off logrotate, no more problems. – nwalke Oct 18 '12 at 16:10
  • @udan just looking now, the problem is it either crashes fairly infrequently, or a few times in a short span, but I cant recreate it myself. also as there are so many PID, which one do I attach gdb too? – Horse Oct 18 '12 at 16:10
  • @tubaguy50035 yeah it does seem odd, i would of thought they would suggest turning that off first or something rather than making me go through all this pain for a problem I am convinced is nothing to do with me :( – Horse Oct 18 '12 at 16:12
  • You say it was a client's external dedicated server. That tells me that someone other than Rackspace set it up. Is that correct? I highly doubt Rackspace would have a silly configuration where Apache restarts everyday. – nwalke Oct 18 '12 at 16:13
  • 1
    Segmentation fault is an internal PHP and sometimes Apache errors. But without propper logs it would be useless to even fill a bugreport – Udan Oct 18 '12 at 16:13
  • 1
    Udan is right. Without logging everything under the sun, there's no way to tell what's happening. It sounds like your issue is very similar to what I went through. I don't think it's worth your time debugging. – nwalke Oct 18 '12 at 16:16
  • No its setup by rackspace, I'm the only technical person who has access to it, but I am more dev than server admin. Rackspace just dont seem very helpful and forthcoming with support and seem keen to avoid having to do anything, completely the opposite to my host of choice – Horse Oct 18 '12 at 16:19

2 Answers2

1

You can start by adding PID to apache logs and see which one dies after what... You will see if its the same request atleast.

And maybe you can see what was updated just before crashing started and maybe you can rollback to previous configuration if some modules where updated and can't live without them.

Udan
  • 5,429
  • 2
  • 28
  • 34
  • Well I noticed they upgraded PHP about a week before we started noticing problems, something which I pointed out to them! But they just responded with 'A PHP upgrade that causes segfaults would be more immediate than a week after the fact. There is unfortunately no way to confirm unless you have a test case that can replicate the issue.' :( sigh – Horse Oct 18 '12 at 16:15
1

If you know how to debug application cores using GDB, here are the source RPMs and RPMs.

I'd also suggest you run "strings" on the application core. This will not give you exact problem but in many cases, it gives something useful.

HTH

Nehal Dattani
  • 933
  • 6
  • 15