4

A while back we removed two damaged OSDs from our Ceph cluster, osd.0 and osd.8. They are now gone from most Ceph commands, but are still showing up in the CRUSH map with weird device names:

# devices
device 0 device0  <-----
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 device8  <-----
device 9 osd.9

Can someone please explain why device0 and device8 are still there, if they have any affect on the cluster, and whether or not we should remove them?

device0 and device8 do not show up anywhere else in the CRUSH map.

We used the procedure from the web site here:

http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-osds/#removing-osds-manual

Basically:

ceph osd crush remove 8
ceph auth del osd.8
ceph osd rm 8

I am mainly asking because we are dealing with some stuck PGs (incomplete) which are still referencing id "8" in various places. Wondering if this is related?

Otherwise, "ceph osd tree" looks how I would expect (no osd.8 and no osd.0):

djakubiec@dev:~$ ceph osd tree
ID WEIGHT   TYPE NAME       UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 58.19960 root default
-2  7.27489     host node24
 1  7.27489         osd.1        up  1.00000          1.00000
-3  7.27489     host node25
 2  7.27489         osd.2        up  1.00000          1.00000
-4  7.27489     host node26
 3  7.27489         osd.3        up  1.00000          1.00000
-5  7.27489     host node27
 4  7.27489         osd.4        up  1.00000          1.00000
-6  7.27489     host node28
 5  7.27489         osd.5        up  1.00000          1.00000
-7  7.27489     host node29
 6  7.27489         osd.6        up  1.00000          1.00000
-8  7.27539     host node30
 9  7.27539         osd.9        up  1.00000          1.00000
-9  7.27489     host node31
 7  7.27489         osd.7        up  1.00000          1.00000

Thanks,

-- Dan

DanJ
  • 1,654
  • 1
  • 16
  • 23
  • What was the procedure you used to remove the osds from the crush map when they failed? – b0bu Sep 06 '16 at 13:59
  • Added the procedure above. – DanJ Sep 07 '16 at 01:19
  • Not sure if it's related yet but I always follow these blogs and this procedure always works for me https://www.sebastien-han.fr/blog/2015/12/11/ceph-properly-remove-an-osd/ – b0bu Sep 07 '16 at 16:00

1 Answers1

0

I had the same problem after a node failure and solved it by manually removing the extra devices from the crush map. I had already removed the osds and the failed node using the standard procedures, but for some reason, I had ghost devices left in my crush map.

Export the crush map and edit it:

~# ceph osd getcrushmap -o /tmp/crushmap
~# crushtool -d /tmp/crushmap -o crush_map
~# vi crush_map

This is what my crush map's devices section looked like before:

# devices
device 0 osd.0
device 1 device1
device 2 osd.2
device 3 osd.3
device 4 device4
device 5 osd.5
device 6 osd.6
device 7 osd.7

I changed it to this - note I had to renumber, not just remove the extra lines.

# devices
device 0 osd.0
device 1 osd.2
device 2 osd.3
device 3 osd.5
device 4 osd.6
device 5 osd.7

Then, recompile the crush map and apply it:

~# crushtool -c crush_map -o /tmp/crushmap
~# ceph osd setcrushmap -i /tmp/crushmap

This kicked off the recovery process again and the ghost devices are now gone.

cscracker
  • 365
  • 1
  • 3
  • 8
  • Editing map like so will screw your osd tree. It is OK for device numbering to have gaps. In fact, "osd.$id" is just a convention and device 42 may certainly map to "osd.777" or "osdiculous" or something. But be sure to know what you are doing: this can (and will) be totally confusing to discover that "osd.6" refers to device with id 4. – Alex Offshore Apr 23 '19 at 12:19
  • I tried simply removing the lines first without renumbering and that didn't work, it re-added the ghost devices on its own. I had to renumber to get them to go away. – cscracker Feb 18 '21 at 16:28