You need to setup ~gitolite/.ssh/authorized_keys with a line like
command="/home/gitolite/bin/gl-auth-command <USERNAME>",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAA...KEY.HERE...ZZZZ== user@label
A random URL with info relating to this (see the bottom of the page)
http://www.geekgumbo.com/2011/10/18/ssh-and-the-gitolite-installation-part-2/
Ensure to change ownership of ~gitolite/.ssh/authorized_keys as per SSH requirements with:
chown gitolite: ~gitolite/.ssh/authorized_keys
chmod go-w ~gitolite/.ssh/authorized_keys
EDIT: to reflect your edits changing 'git' to 'gitolite' system username.
Test your access from the client with:
ssh -l gitolite -i <file_id_rsa_foobar> -v -o PasswordAuthentication=no -T <host>
Added -T seems needed on my local system to get banner (typed in excuse mistakes):
....
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/username/.ssh/id_rsa_foobar
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /home/username/.ssh/id_dsa_foobar
debug1: Remote: Forced command: /home/gitolite/bin/gl-auth/command <username>
....
hello <username> this is gitolite vX.X.XX-g0123abcd running on git X.X.X
the gitolite config gives you the following access:
R W mydir/project1
....