Very strange permission denied for file checkout

Question

I am not a ClearCase newbie, I already did setup some CC-Servers in very well organized networks managed by LDAP. But this time it is a bit different in a non-LDAP managed network… I am facing a real strange checkout issue and I am stuck… To mention it directly at the beginning, this is not a standard permission issue (according to my knowledge).

When I try to checkout a file on my client (clearcase-client) as vobowner (ccadm - I haven't got any other user so far) I get following error:

cleartool: Error: Checked out version, but could not copy data to "testfile.txt" in view: Keine Berechtigung.
Correct the condition, then uncheckout and re-checkout the element.
cleartool: Error: Unable to check out "testfile.txt".

But I am able create Vobs, Views AND view private files. I am even able to checkout folders, but I am not able to checkout a simple text file.

ccadm@clearcase-client:~$ ct mkvob -tag /vobs/test -nc -stgloc vobstore
Created versioned object base.
Host-local path: clearcase-vm:/cc_storage/vobstore/test.vbs
Global path:     /net/clearcase-vm/cc_storage/vobstore/test.vbs
VOB schema:        80
VOB feature level: 8
VOB ownership:
  owner ccadm
  group ccusers

VOBs have special data backup considerations.  For more information on how to
back up your VOB properly, see the documentation for administering ClearCase.
If the backups aren't done properly, you are putting your data at risk!
ccadm@clearcase-client:~$ ct mkview -tag ccadm.dv -stgloc viewstore
Created view.
Host-local path: clearcase-vm:/cc_storage/viewstore/ccadm/ccadm.dv.vws
Global path:     /net/clearcase-vm/cc_storage/viewstore/ccadm/ccadm.dv.vws
It has the following rights:
User : ccadm    : rwx
Group: ccusers  : rwx
Other:          : r-x
ccadm@clearcase-client:~$ ct mount /vobs/test
ccadm@clearcase-client:~$ ct setview ccadm.dv
ccadm@clearcase-client:~$ cd /vobs/test
ccadm@clearcase-client:/vobs/test$ ct co -nc .
Checked out "." from version "/main/0".
ccadm@clearcase-client:/vobs/test$ touch testfile.txt
ccadm@clearcase-client:/vobs/test$ ct mkelem -nc -ci testfile.txt
Created element "testfile.txt" (type "text_file").
Checked in "testfile.txt" version "/main/1".
ccadm@clearcase-client:/vobs/test$ ct ci -nc .
Checked in "." version "/main/1".
ccadm@clearcase-client:/vobs/test$ ct co -nc testfile.txt
cleartool: Error: Checked out version, but could not copy data to "testfile.txt" in view: Keine Berechtigung.
Correct the condition, then uncheckout and re-checkout the element.
cleartool: Error: Unable to check out "testfile.txt".

This is what I do see in mvfs_log:

56fe658a mvfs: Error: cleartext open failed view=ccadm.dv vob=/vobs/test dbid=0x80000003 - Permission denied
56fe658a mvfs: Error: cleartext pname= /net/clearcase-vm/cc_storage/viewstore/ccadm/ccadm.dv.vws/.s/00048/8000000356fe6566testfile.txt
56fe658a mvfs: Error: cleartext create view=ccadm.dv vob=/vobs/test dbid=0x80000003 - Permission denied
56fe658a mvfs: Error: cleartext pname= /net/clearcase-vm/cc_storage/viewstore/ccadm/ccadm.dv.vws/.s/00048/8000000356fe6567testfile.txt.checkedout

Of course I can see “Permission denied” but permissions should be fine (see below)…

Environment description: Network is currently a private test network without LDAP. So UIDs and GIDs are set the same on server and client. I don't know any other prerequisites for using CC in a non-LDAP network. Both are configured for autofs using /net. Both machines are currently virtual machines for testing (I had exactly the same issue on non-VMs… That's why I did setup the VMs for testing)

CC-Server: clearcase-vm

OS: Ubuntu 14.04

Linux clearcase-vm 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Purpose: View-, Vob-, Registryserver

Local user: ccadm

ccadm@clearcase-vm:~$ id -a
uid=2000(ccadm) gid=2000(ccusers) Gruppen=2000(ccusers)

Local vobstorage permissions:

ccadm@clearcase-vm:~$ ls -la /cc_storage/vobstore/test.vbs/
insgesamt 52
drwxr-xr-x 8 ccadm ccusers 4096 Apr  1 14:09 .
drwxrwxrwx 4 root  root    4096 Apr  1 14:09 ..
drwxr-xr-x 2 ccadm ccusers 4096 Apr  1 14:09 admin
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 c
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 d
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 db
-r--r--r-- 1 ccadm ccusers   13 Apr  1 14:09 .hostname
drwx------ 2 ccadm ccusers 4096 Apr  1 14:09 .identity
-rw-r--r-- 1 ccadm ccusers    7 Apr  1 14:09 .pid
-r--r--r-- 1 ccadm ccusers   41 Apr  1 14:09 replica_uuid
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 s
-r--r--r-- 1 ccadm ccusers   41 Apr  1 14:09 vob_oid
-rw-r--r-- 1 ccadm ccusers  625 Apr  1 14:09 vob_server.conf

Local viewstorage permissions:

ccadm@clearcase-vm:~$ ls -la /cc_storage/viewstore/ccadm/
insgesamt 16
drwxr-xr-x 4 ccadm ccusers 4096 Apr  1 14:10 .
drwxrwxrwx 3 root  root    4096 Mär 30 16:43 ..
drwxrwxr-x 5 ccadm ccusers 4096 Apr  4 11:39 ccadm.dv.vws

ClearCase version:

ccadm@clearcase-vm:~$ ct -ver
ClearCase version 8.0.1.1 (Fri Sep 20 16:09:14 EDT 2013) (8.0.1.01.00_2013C.FCS)
ClearCase version 8.0.1.2 (Wed Dec 11 16:09:14 EDT 2013) (8.0.1.02.00_2013D.FCS)
ClearCase version 8.0.1.3 (Wed Mar 19 00:31:17 EST 2014) (8.0.1.03.00_2014A.FCS)
ClearCase version 8.0.1.3-iFix01 (Tue Apr 22 18:14:02 EDT 2014) (8.0.1.03.01_2014A.1.FCS)
ClearCase version 8.0.1.04 (Thu Jun 24 22:31:13 EDT 2014) (8.0.1.04.00_2014B.FCS)
ClearCase version 8.0.1.05 (Wed Sep 17 15:09:27 EDT 2014) (8.0.1.05.00_2014C.FCS)
ClearCase version 8.0.1.06 (Tue Dec 11 22:31:02 EST 2014) (8.0.1.06.00_2014D.FCS)
ClearCase version 8.0.1.07 (Wed Mar 18 00:31:02 EST 2015) (8.0.1.07.00_2015A.FCS)
ClearCase version 8.0.1.08 (Tue Jun 24 22:30:54 EDT 2015) (8.0.1.08.00_2015B.FCS)
ClearCase version 8.0.1.09 (Wed Aug 26 22:16:02 EDT 2015) (8.0.1.09.00_2015C.D150826)
@(#) MVFS version 8.0.1.9 (Fri Aug 21 22:52:34 2015) built at $Date: 2016-03-30.13:14:43 (UTC) $
cleartool                         8.0.1.9 (Thu Aug 20 22:50:12 2015)
db_server                         8.0.1.9 (Mon Aug 17 11:37:37 2015)
VOB database schema versions: 54, 80

CC-Client: clearcase-client

OS: Ubuntu 14.04

Linux clearcase-client 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Purpose: ClearCase client

Local user: ccadm

ccadm@clearcase-client:~$ id -a
uid=2000(ccadm) gid=2000(ccusers) Gruppen=2000(ccusers)

Remote vobstorage permissions on clearcase-vm:

ccadm@clearcase-client:~$ ls -la /net/clearcase-vm/cc_storage/vobstore/test.vbs/
insgesamt 52
drwxr-xr-x 8 ccadm ccusers 4096 Apr  1 14:09 .
drwxrwxrwx 4 root  root    4096 Apr  1 14:09 ..
drwxr-xr-x 3 ccadm ccusers 4096 Apr  3 00:30 admin
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 c
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 d
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 db
-r--r--r-- 1 ccadm ccusers   13 Apr  1 14:09 .hostname
drwx------ 2 ccadm ccusers 4096 Apr  1 14:09 .identity
-rw-r--r-- 1 ccadm ccusers    7 Apr  1 14:09 .pid
-r--r--r-- 1 ccadm ccusers   41 Apr  1 14:09 replica_uuid
drwxr-xr-x 3 ccadm ccusers 4096 Apr  1 14:09 s
-r--r--r-- 1 ccadm ccusers   41 Apr  1 14:09 vob_oid
-rw-r--r-- 1 ccadm ccusers  625 Apr  1 14:09 vob_server.conf

Remote viewstorage permissions on clearcase-vm:

ccadm@clearcase-client:~$ ls -la /net/clearcase-vm/cc_storage/viewstore/ccadm/
insgesamt 16
drwxr-xr-x 4 ccadm ccusers 4096 Apr  1 14:10 .
drwxrwxrwx 3 root  root    4096 Mär 30 16:43 ..
drwxrwxr-x 5 ccadm ccusers 4096 Apr  4 11:39 ccadm.dv.vws

ClearCase version:

ccadm@clearcase-client:~$ ct -ver
ClearCase version 8.0.1.1 (Fri Sep 20 16:09:14 EDT 2013) (8.0.1.01.00_2013C.FCS)
ClearCase version 8.0.1.2 (Wed Dec 11 16:09:14 EDT 2013) (8.0.1.02.00_2013D.FCS)
ClearCase version 8.0.1.3 (Wed Mar 19 00:31:17 EST 2014) (8.0.1.03.00_2014A.FCS)
ClearCase version 8.0.1.3-iFix01 (Tue Apr 22 18:14:02 EDT 2014) (8.0.1.03.01_2014A.1.FCS)
ClearCase version 8.0.1.04 (Thu Jun 24 22:31:13 EDT 2014) (8.0.1.04.00_2014B.FCS)
ClearCase version 8.0.1.05 (Wed Sep 17 15:09:27 EDT 2014) (8.0.1.05.00_2014C.FCS)
ClearCase version 8.0.1.06 (Tue Dec 11 22:31:02 EST 2014) (8.0.1.06.00_2014D.FCS)
ClearCase version 8.0.1.07 (Wed Mar 18 00:31:02 EST 2015) (8.0.1.07.00_2015A.FCS)
ClearCase version 8.0.1.08 (Tue Jun 24 22:30:54 EDT 2015) (8.0.1.08.00_2015B.FCS)
ClearCase version 8.0.1.09 (Wed Aug 26 22:16:02 EDT 2015) (8.0.1.09.00_2015C.D150826)
@(#) MVFS version 8.0.1.9 (Fri Aug 21 22:52:34 2015) built at $Date: 2016-04-01.11:18:08 (UTC) $
cleartool                         8.0.1.9 (Thu Aug 20 22:50:12 2015)
db_server                         8.0.1.9 (Mon Aug 17 11:37:37 2015)
VOB database schema versions: 54, 80

If anybody is wondering what happens when I do checkout a file as ccadm directly working on the CC-Server (clearcase-vm) using the same vob and view, it works as it should... So checkout of files is possible.

Has anybody got an idea what could cause that permission issue or could give me a hint what to do? Maybe I did miss something when setting up and configuring the server/client, but I don't think so…

Is there any process keeping an handle to the resource you are trying to checkout, which would explain the error message? — VonC, Apr 05 '16 at 12:52
Using lsof or fuser on the file itself results in "No such file or directory" bcuz after checking out testfile.txt the local file system does not see it anymore. Using "lsof | grep testfile.txt" or "ps -ef | grep testfile.txt" does not give any result, so I guess no process is keeping an handle on it... Any other idea how to check it reliable? — Jonas, Apr 05 '16 at 13:33
Is the issue reproducible for any file that you are trying to checkout? — VonC, Apr 05 '16 at 13:34
Yes it is. There is currently just one vob with one file, but I just added another one to source control (same issue). Moreover it was the same issue with productive vobs/files that had been replicated to the non-VMs. After that I started to test on VMs — Jonas, Apr 05 '16 at 13:42
Genrally, I start by a fix_prot on the view and/or on the vob, to make sure there is no permission issue. — VonC, Apr 05 '16 at 13:52
And check the logs (vob_server log for instance) and fix_prot the vob: http://stackoverflow.com/a/29856493/6309 — VonC, Apr 05 '16 at 18:40
I never thought about to do fix_prot since the view and the vob are just created and the output looks good according to permissions. BUT I did a fix_prot on the view now... nothing has changed. After that I did a fix_prot on the vob AND... I am able to checkout ONE time! After checking in the file, I am not able to checkout the file again (same issue as before). Means I have to do a fix_prot everytime before checking out ;) — Jonas, Apr 06 '16 at 08:51
BTW If I do fix_prot on the vob, checkout the file BUT unco after that, I am able to checkout again. I can do that until I checkin the file, after checking in I have to do fix_prot again. But many thanks for that idea to do fix_prot! Now I have got a point where I can go on analyzing the issue — Jonas, Apr 06 '16 at 08:54
Is there any clues in the logs (both client and ClearCase server logs) once you do a checkout? — VonC, Apr 06 '16 at 09:06

score 0 · Answer 1 · answered Apr 06 '16 at 00:23

0

Start with undoing the checkout, and then try to cat the file. If it fails, and you dont see cleartext construction errors in the view log, check to see if you can cat the cleartext from the client.

If it fails, check a network trace. I have a sneaking feeling that something is screwing up NFS. You may be defaulting to nfsv4 and it may be trying to do something other than AUTH_SYS.

This would be an interesting PMR.

answered Apr 06 '16 at 00:23

Brian Cowan

1,048
6
7

Many thanks for that hint! Since NFS is working fine for standard file operations without ClearCase involved, I did not expect any NFS issue... But I am not an NFS expert. Since I experienced that fix_prot issue (explained above), I think about a possible NFS issue too. I will take a look at that together with an NFS expert. I will definitively come back with a solution or with a request for more help/hints :) – Jonas Apr 06 '16 at 08:58

score 0 · Answer 2 · answered Apr 07 '16 at 19:38

As an FYI, I built a pair of Ubuntu 14.04.0 VM's and updated them. They are running this kernel release:

Linux test-ub-vbox 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I executed this script to ensure that the UIDs were the same on both sides of the link:

 sudo useradd -u 50126 -g users -m -s /bin/bash -d /home/vobadm vobadm
 sudo useradd -u 50128 -g users -m -s /bin/bash -d /home/user1 user1
 sudo passwd user1
 sudo passwd vobadm

I set /net automount map to execute the provided /etc/auto.net file (the default is to have no /net @ all).

I exported the server's storage as *(rw,no_root_squash).

I created the VOB on the server as "vobadm" and mounted it on the client as vobadm. (appropriate directories created)

I'm not able to make the issue occur. My ubuntu 14.04 setup seems to work fine.

You do need to use ubuntu 14.04.0 from archive.ubuntu.com and not anything later. The rest can take you down the primrose path as some of them started using Ubuntu 15.10 kernels...

And I'm using 8.0.1.10...

If you hit something. Please open a PMR, mention my name, share the link to this thread, and get the network traces. We can then compare notes/traces.

Pity I can't edit the answer... Just reproduced. In my "works fine" case I add an EXISTING file to source control. In the failing case, I add a non-existant file to source control and it fails. — Brian Cowan, Apr 07 '16 at 19:48
@Jonas please open a PMR. This issue seems to only happen when adding a non-existant file to ClearCase. If I create a file and mkelem THAT, it works. I get your exact error, and it may be related to APAR PI22822, which was fixed back in 8.0.1.5. — Brian Cowan, Apr 07 '16 at 19:59
Note: Files with CONTENTS work fine... 0 byte files don't. So, if you do "echo foo>bar" and then mkelem bar it works... — Brian Cowan, Apr 07 '16 at 20:21
Another "belay my last". @Jonas. I need a PMR opened. it turns out that even with contents, once I add them to source I can't check them out. It looks like an odd MVFS interaction with NFSv4. — Brian Cowan, Apr 08 '16 at 17:52

Very strange permission denied for file checkout

2 Answers2