Notes, errors, solutions...

Additional notes, setups and howtos for gitolite and gerrit and their possible errors.

Where to Look

Most of the operations here described should be run as the git user. All directories of the gitolite and gerrit installations are owned by the git user.

gitolite

Gitolite stores logs and SSH keys in a hiddend directory at this path: /home/git/.gitolite.
That same directory has been moved here: /srv/gitolite, and symlinked back at its original position.

Inside there is a logs directory, the first place to look at when something is not working with gitolite.

gerrit

Everything is stored under: /srv/gerrit.
Repositories are stored in the same path as gitolite: /srv/repositories.

LDAP Integration

SSH Keys and Users

gitolite does not have out of the box LDAP integration, but it is possible to extract SSH keys for LDAP and have gitolite adds user based on their SSH keys.

There is a custom script that query the LDAP database and does exactly that:

The keys are stored with the following pattern:

name.surname@key_N.pub

Where N is an index, starting from 0, to indicate multiple user keys.

For more info on why it has been done like that, see gitolite documentation here.

Possible Errors

There is a cron job that extract the keys from LDAP. If it finds an error, usually it is of the form:

Error executing the following command: /home/git/bin/gitolite trigger SSH_AUTHKEYS
Command stderr:
line 1 too long: MIIEpAIBAAKCAQEAwNaMzqN+IDv4SK/r1XkSY8u3...
FATAL: fingerprinting failed for 'keydir/name.surname@key_0.pub'

This kind of error happens usually because the key uploaded on Linaro Login has not been copied correctly:

  • empty spaces in the key
  • the key is divided up in multiple lines

What to do:

  • Take note of the name.surname

  • On the server, as the git user, remove the corrupted key:

    rm /srv/gitolite/keydir/name.surname@key_0.pub
  • Manually run the following gitolite command (always as the git user):

    gitolite trigger SSH_AUTHKEYS
  • The user should be warned that there is something wrong with their key, and they need to fix it

At the next run of the cron, the key, if it has been updated, will be recreated.

LDAP/Crowd Groups

With gitolite it is possible to get groups information from LDAP. Since with LDAP we cannot extract nested groups, or it will take forever to do it with a recursive query, we can use Crowd for that.

For gitolite this does not change, since all that it does is call an external program/script, with the only requirement that the results of this program have to be printed on stdout as a space separated list. For more info, refer to the gitolite documentation here.

The script that does this is here:

This will be reflected in the gitolite-admin repository, where groups defined in Crowd/LDAP can be used to grant access to the repositories.

For more info on gitolite groups support, see here.

Server Side Git Hooks - Only for gitolite

It is possible to install server side git hooks also with gitolite, but it can be done by who has direct SSH access to the server, and access to the gitolite-admin repository.

gitolite has two directories where it is possible to install hooks:

  • /srv/gitolite/local/hooks/common

  • /srv/gitolite/local/hooks/repo-specific

The first path is for hooks that need to be common to all repositories. It is seldomly used.
The second path is where special hooks have to be installed. Once a hook has been installed here, it needs to be enabled for a specific repository from the gitolite-admin repo.

The syntax to enable such hook is as follows:

repo lava/[a-zA-Z0-9].*
    C       = @lava
    RW+     = @lava
    R       = @all
    option hook.post-receive = $HOOK_NAME

where $HOOK_NAME is the name of the script installed in /srv/gitolite/local/hooks/repo-specific.

Installing Hooks

Since it is necessary to have SSH access to the server, to install a server side git hooks a request to IT needs to be filed with at least the following information:

  • The hook script to be installed
  • Which default git hooks it needs to be applied
  • The repositories that will use the hook

Removing Hooks

Once installed and enabled for a repository, if the hook is not needed anymore the first thing to do is to edit the gitolite-admin repository and remove the line added.
From:

repo lava/[a-zA-Z0-9].*
    C       = @lava
    RW+     = @lava
    R       = @all
    option hook.post-receive = $HOOK_NAME

To:

repo lava/[a-zA-Z0-9].*
    C       = @lava
    RW+     = @lava
    R       = @all

This is not sufficient though to remove the hook for that repository: it is also necessary to remove the symlink gitolite creates on the server! gitolite does not take care of cleaning it up!

Post-Receive Hook and Syncing - Only for Staging

During the testing period of gitolite+gitweb on the staging instance, we had the necessity to sync repositories on git.linaro.org in a mirror-like way. To do this, we used an rsync script that hourly synced a predefined directory from the staging instance into the production one.

Another solution, to have an immediate propagation of the changes pushed to a repository, is to create a custom post-receive hook, that will run a similar rsync script.
The custom script is locate at /home/ubuntu/sync-1repo-to-git.sh and is supposed to run as the ubuntu user.

Since git is not in the sudoers list, the following has to be done:

  • Add the file /etc/sudoers.d/git with the following content:

    git ALL=(ALL) NOPASSWD: /home/ubuntu/sync-1repo-to-git.sh

There is a wrapper script for the call with sudo at /home/git/.gitolite/local/hooks/repo-specific/lava-sync, and for git push, the gitolite-admin repository can be configured as follows:

repo lava/[a-zA-Z0-9].*
    C       = @lava-team
    RW+     = @lava-team
    R       = @all
    option hook.post-receive = lava-sync

Gitolite errors, users requests, user repo removals...

Users cannot delete repositories

In order to delete a repository with gitolite, a user needs to be the owner of such repository. gitolite stores ownership in a special file called gl-creator inside the repository directory.

During the migration from the old server to the new one, some repositories have been setup with a general creator. In such cases:

  • Modify the repository gl-creator file with the new user name.

  • Remove from the server the repository and then run, as the git user: gitolite setup.

Users cannot delete remote branches

In order for someone to delete remote branches, it is necessary to have the RW+ permission on the repository. Make sure the gitolite-admin repository has the correct configuration.

Users asks for copying a repository

Users can "copy" repositories themselves. There is a gitolite command called "fork" to be used for that. They must have the "C" (create) permission on the destination where they want to "copy" the repositories.

Users cannot find their own repositories

When we moved the repositories from the "old" server into the new one, we checked which repositories belonged to whom: some of this repos might have been left out due to lack of information.

But not all is lost! We still have the repositories in the "old" server.
There is a script in /home/ubuntu called prod2staging-sync.sh: open it and change the OLD_PATH and NEW_PATH variables in order to point to the path on the old server and on the new one.

The script should also take care of everything else, and after its execution the repositories should be available.
In case the repositories do not show up:

  • Make sure directories and files are owned by git.

  • For each repository, make sure they have a gl-creator and git-daemon-export-ok files. The latter is an empty file, the former needs to contain the name of the user (in the form of name.surname) if the repositories are under people/name.surname/, or a generic user name in the other case.

  • Run gitolite setup to make sure gitolite picks up the new repositories.

Moving repositories under a new user/team

To move repositories around, like from one user to another, or from one user to a team, as the git user:

  • Move the repository directory in the new path.
  • Edit the repository gl-creator file to be the name of the new user (if the repository has been moved under a new one), or the name of a person in the team (if it has been moved under a team space).

  • If the repository has a gl-conf file, remove it.

  • Run the command: gitolite setup.

Gerrit

Problem with duplicated users

  • How to remove duplicated accounts: instructions are for android.git.linaro.org, but the commands are the same.

  • The same result as the commands at the above link, can be achieved using the SSH command line interface for administrators: cmd-set-account

People are not able to approve merge requests

In order for someone to be able to approve a merge request, they need to have two special permissions level set for the repository:

  • Label Code-Review
  • Submit

Usually, this is achieved by granting these permissions to a team they belong to from the access level list of the repository.

Parent repositories for ACLs

Even better would be to have a gerrit only repository (created from within gerrit, only admin can do that), set the new permissions, and make this new gerrit only repository the parent for the access level of the target.

Let's say we have a repository here: infrastructure/linaro-image-tools
In order for all the Infrastructure team to have those permissions on that repository, we would have to modify the the access level for that. But we might have multiple repositories under infrastructure/.

We can create a new project called infrastructure, select the option Only serve as parent for other projects, and modify its access level.
After that, we need to set the parent of all the repositories under infrastructure/*. This is possible only from the command line:

ssh review.linaro.org gerrit set-project-parent --parent PARENT_NAME PROJECT

An example script to do it automatically for a defined path:

parent=$1
for repository in `ssh review.linaro.org gerrit ls-projects | grep people\/ `; do
    ssh -p 29418 review.linaro.org gerrit set-project-parent --parent $parent $repository
done

Group deletion

From gerrit UI it is not possible to delete a group that was mistakenly created. In order to do so, it is necessary to work on the database:

sudo -u postgres psql -d reviewdb

These are the SQL queries that is necessary to run:

DELETE FROM account_groups WHERE name='...';
DELETE FROM account_group_names WHERE group_id NOT IN (SELECT group_id FROM account_groups);
DELETE FROM account_group_members WHERE group_id NOT IN (SELECT group_id FROM account_groups);
DELETE FROM account_group_members_audit WHERE group_id NOT IN (SELECT group_id FROM account_groups);

User deactivation

To deactivate, change the name or remove SSH keys, please refer to Gerrit documentation: https://review.linaro.org/Documentation/cmd-set-account.html

You have to be in the Administrators group in Gerrit to be able to use the above command.

Platform/Systems/GitGerritHowTo (last modified 2014-07-09 13:56:19)