2

I have 2 repositories in sync through a common server.

Repository 1 -- Server -- Repository 2

In each repository, there is a directory called "unique" that can hold the data regarding the setup of the database being used in a virtualenv.

What i would like to achieve is to preserve separately the contents of the "unique" directory for each virtualenv without disturbing the workflow of the python code.

I would appreciate your help.

raratiru
  • 8,748
  • 4
  • 73
  • 113
  • 1
    Would love to know more about this somewhat unique use case... have you looked at Git submodules? – RyPeck Nov 15 '13 at 13:45
  • This actually is a migrations directory of south in Django. It tracks the changes of the database structure and each branch corresponds to a different database. This is probably an exaggerated approach which I could do without it or I can find workarounds outside git. However, I would like to know if there is someway to do this using the configuration settings of git. I have tried git submodules in the past, I find them cumbersome and hard to maintain. If they are the only solution, I would prefer to change the structure of my development setup. – raratiru Nov 15 '13 at 15:22
  • 1
    Is there some reason the two repos need their "unique" directories to have the same name? If you have some other way to distinguish (by your database name, or the host it's running on, for example) you can use that to select the proper directory. If they have different names your problem goes away. – engineerC Nov 15 '13 at 23:01
  • Thank you, CaptainMurphy, for the idea.I will check if I can implement it. I am also planning to use a .local directory with the name of each branch as a subdirectory and use hooks to copy the "unique" files each time I checkout or commit. I will work on it in the afternoon. – raratiru Nov 16 '13 at 09:26

2 Answers2

1

EDIT: What follows is a naive "first approach" which cannot validate as a viable solution.

The issue is far quite more complicated and I will try to refine my approach but this is not a current priority.

The way I am about to tackle with this issue is by soft linking the "unique" file or directory to a file or directory in a .local folder.

The structure of this folder must be created for the first time -at the master branch before creating other branches- with the following command:

cd ./$(git rev-parse --show-cdup) && mkdir -p .local/$(echo "$USER")/$(git symbolic-ref --short -q HEAD)

This command will create -in the root of the repository- the tree: 

.local/<username>/<git branch>.

The "unique" files of the repository have to be moved to a relevant directory under this tree and have to be soft linked.

For example:

The unique file 

./path/to/file/<my_file>

will be manually moved to:

./.local/<username>/<git branch>/path/to/file/<my_file>

while a symlink will be manually created in its old position linking to the new one.

"./path/to/file/<my_file>" -> "./.local/<username>/<git branch>/path/to/file/<my_file>"

The symlink might also be added in the .gitignore file:

path/to/file/<my_file>

If other branches already exist, the .local folder from the master branch has to be checked out and the above procedure has to be repeated manually for the current branch.

From now on two scripts called post-merge and post-checkout in: 

.git/hooks/post-merge
.git/hooks/post-checkout

will hopefully manage to to update the symlinks after each merge if the permissions are set to 755.

During the first pull/merge it will create a local environment where all the "unique" files from the origin repository will be copied and symlinked.

The user has to delete or arrange accordingly the .local subfolders and from then on the links will be automatically updated -when needed- by a script written in bash which mainly uses sed.

I have not thoroughly tested the script due to time constraints. It seems to be working in my fedora 19 environment and I will update the post if changes have to be made.

I a more git oriented solution would still be very welcome.

post-merge:

#! /bin/sh

# "origin": Refers to the existing merged files (soft links) information that will be overriden.
# "local": Refers to the local system information that will replace origin information.

#Get local user name to define the repository directory.
local_user=$(echo "$USER")

#Get local branch name to define the branch directory.
local_branch=$(git symbolic-ref --short -q HEAD)

#Go to the root of the repository
cd ./$(git rev-parse --show-cdup)

#Create local environment
local_environment=$(echo './.local/'"$local_user"'/'"$local_branch"'/')
mkdir -p $local_environment


#Grub and manipulate all soft links.
symlink_no=0
for link in $(find -L ./ -xtype l); 
do
    symlink_no=$[symlink_no+1]
    # Grub information from each symlink
    source_file=$(ls -la $link | sed -n 's/^.*[0-9]\+:[0-9]\+\s\(.*\)\s->.*$/\1/p')
    origin_target_file=$(ls -la $link | sed -n 's/.*>\s\+\(.*\)$/\1/p')
    relative_to_root_origin_target_file=$(ls -la $link | sed -n 's/.*>\s.*\(.\/.local.*\)$/\1/p')

    find_origin_user='s/.*local\/\([a-zA-Z0-9._-]\+\).*/\1/p'
    origin_user=$(echo $origin_target_file | sed -n "$find_origin_user")
    find_origin_branch='s/.*local\/'"$origin_user"'\/\([a-zA-Z0-9._-]\+\).*/\1/p' 
    origin_branch=$(echo $origin_target_file | sed -n "$find_origin_branch")

    local_target_file=$(echo $origin_target_file | sed -n 's/local\/'"$origin_user"'\/'"$origin_branch"'/local\/'"$local_user"'\/'"$local_branch"'/p')
    relative_to_root_local_target_file=$(echo $relative_to_root_origin_target_file | sed -n 's/local\/'"$origin_user"'\/'"$origin_branch"'/local\/'"$local_user"'\/'"$local_branch"'/p')

    remove_file_from_target_path='s/\(.*\)\/.*$/\1/p'
    find_local_target_layout=$(echo $relative_to_root_origin_target_file | sed -n "$remove_file_from_target_path" | sed -n 's/.*'"$origin_branch"'\/\(.*$\)/\1/p')
    local_environment_path=$(echo "$local_environment""$find_local_target_layout")

    #Wording header.
    echo ''
    echo ''
    echo 'Checking for symlink [' "$symlink_no"' ]:' "$source_file"
    echo '------------------------------------------------------'
    echo ''

    # ------------------- Target File Manipulation ---------------------------------
    #Check if target file does not exist in local environment.
    if [ ! -f "$relative_to_root_local_target_file" ]; 
    then
        #If true, verify that the source file exists and copy it.
        if [ -f "$relative_to_root_origin_target_file" ];
        then
            mkdir -p "$local_environment_path"
            rsync -va "$relative_to_root_origin_target_file" "$relative_to_root_local_target_file"
            echo ''
            origin_target_file_exists=1
            local_target_file_exists=1
            echo 'The source file has been created: ' "$relative_to_root_local_target_file"
        else
            echo 'The source file does not exist: ' "$relative_to_root_origin_target_file"
            origin_target_file_exists=0
            local_target_file_exists=0
        fi
    else
        local_target_file_exists=1
        echo 'The local file "'"$relative_to_root_local_target_file" '"  already exists.'
    fi
    source_path=$(echo $source_file | sed -n "$remove_file_from_target_path")
    source_check_target=$(echo "$source_path"'/'"$local_target_file")
    #-------------------- Source File Manipulation ---------------------------------
    # If target file exists:
    if [[ $local_target_file_exists -eq 1 ]];
    then
        #Check the source file if it is already linked.
        if [ "$local_target_file" == "$origin_target_file" ];
        then
            #Is it a correct link?
            if [ -f $source_check_target ];
            then
                echo 'symlink "' "$source_file" '->' "$local_target_file" '" already exists and it is correct.'
            else
                echo '*********** This is a broken link: ' "$source_file"
                echo '           **********************'
            fi
        else
            #Delete existing symlink
            rm "$source_file"

            #Create new symlink according to local environment
            ln -s "$local_target_file" "$source_file"
            echo 'symlink "' "$source_file" '->' "$local_target_file" '" has been created.'
        fi
    else
        echo '*********** This is a broken link: ' "$source_file"
        echo '            **********************'
    fi
done

post-checkout:

#! /bin/sh

# Start from the repository root.
cd ./$(git rev-parse --show-cdup)

# Delete .pyc files and empty directories.
find . -name "*.pyc" -delete
#find . -type d -empty -delete

#----------------Symlink manipulation---------------------------------------
#Get local user name to define the repository directory.
local_user=$(echo "$USER")

#Get local branch name to define the branch directory.
local_branch=$(git symbolic-ref --short -q HEAD)

#Go to the root of the repository
cd ./$(git rev-parse --show-cdup)

#Grub and manipulate all soft links.
symlink_no=0
for link in $(find -L ./ -xtype l); 
do
    symlink_no=$[symlink_no+1]
    # Grub information from each symlink
    source_file=$(ls -la $link | sed -n 's/^.*[0-9]\+:[0-9]\+\s\(.*\)\s->.*$/\1/p')
    origin_target_file=$(ls -la $link | sed -n 's/.*>\s\+\(.*\)$/\1/p')
    relative_to_root_origin_target_file=$(ls -la $link | sed -n 's/.*>\s.*\(.\/.local.*\)$/\1/p')

    find_origin_user='s/.*local\/\([a-zA-Z0-9._-]\+\).*/\1/p'
    origin_user=$(echo $origin_target_file | sed -n "$find_origin_user")
    find_origin_branch='s/.*local\/'"$origin_user"'\/\([a-zA-Z0-9._-]\+\).*/\1/p' 
    origin_branch=$(echo $origin_target_file | sed -n "$find_origin_branch")

    local_target_file=$(echo $origin_target_file | sed -n 's/local\/'"$origin_user"'\/'"$origin_branch"'/local\/'"$local_user"'\/'"$local_branch"'/p')
    relative_to_root_local_target_file=$(echo $relative_to_root_origin_target_file | sed -n 's/local\/'"$origin_user"'\/'"$origin_branch"'/local\/'"$local_user"'\/'"$local_branch"'/p')

    remove_file_from_target_path='s/\(.*\)\/.*$/\1/p'
    find_local_target_layout=$(echo $relative_to_root_origin_target_file | sed -n "$remove_file_from_target_path" | sed -n 's/.*'"$origin_branch"'\/\(.*$\)/\1/p')
    local_environment_path=$(echo "$local_environment""$find_local_target_layout")

    source_path=$(echo $source_file | sed -n "$remove_file_from_target_path")
    source_check_target=$(echo "$source_path"'/'"$local_target_file")

    #Wording header.
    echo ''
    echo ''
    echo 'Checking for symlink [' "$symlink_no"' ]:' "$source_file"
    echo '------------------------------------------------------'
    echo ''

    # ------------------- Target File Manipulation ---------------------------------
    #Check if target file does not exist in local environment.
    if [ -f "$relative_to_root_local_target_file" ]; 
    then
        #Check the source file if it is already linked.
        if [ "$local_target_file" == "$origin_target_file" ];
        then
            #Is it a correct link?
            if [ -f $source_check_target ];
            then
                echo 'symlink "' "$source_file" '->' "$local_target_file" '" already exists and it is correct.'
            else
                echo '*********** This is a broken link: ' "$source_file"
                echo '           **********************'
            fi
        else
            #Delete existing symlink
            rm "$source_file"

            #Create new symlink according to local environment
            ln -s "$local_target_file" "$source_file"
            echo 'symlink "' "$source_file" '->' "$local_target_file" '" has been created.'
        fi
    else
        echo 'The source file does not exist: ' "$relative_to_root_local_target_file"
    fi
done
raratiru
  • 8,748
  • 4
  • 73
  • 113
1

Finally, the solution has been provided by virtualenvs.

A very good post which explains how and why one should use different virtualenvs in django development is Marina Mele's.

unique files correspond to one virtualenv and they have a symlink to the directory .local/path/to/unique/file which is updated every time an environment is activated.

For example, in order to tackle with the different migrations folders that my project would need if I used different databases for development, testing, production, I would do the following:

At ~/.virtualenvs/<myvirtualenv>/bin/activate I would append the commands:

cd /path/to/git/repository/of/my/project && manage_migrations <myvirtualenv>
for item in $(find ./ ! -type f -name "migrations" ! -path "./.git/*" ! -path "./.local/*" ); do touch "$item"/__init__.py; done

Where manage_migrations:

(Script that upon virtualenv activation, it updates the symlink of each migrations folder. e.g.

./app/migrations -> .local/<myvirtualenv>/app/migrations

#!/bin/bash
# vim: filetype=sh

virtualenv_name=$1
repository_root=$(git rev-parse --show-toplevel)
local_environment="$repository_root"/.local/"$virtualenv_name"

mkdir -p "$local_environment"
cd "$repository_root"
for item in $(find ./ ! -type f -name "migrations" ! -path "./.git/*" ! -path "./.local/*" )
do 
    if [ -f $item ]
    then
        echo "$item is a file, this is a plain burdain."
        exit
    elif [ ! -L $item ]
    then
        rsync -vaR --delete "$item" "$local_environment"/
        rm -Rf $item
    fi
    rm "$item" > /dev/null 2>&1
    mkdir -p "$local_environment"/"${item#./}"
    ln -s "$local_environment"/"${item#./}" "$item"
done

I have posted a git-hooks based approach to this issue in the following repository:

Disabled Link

However, this does not seem to be a viable solution.

Using symlinks, Merge Strategies or local config files to override the defaults would be the recommended way to go.

Be aware of this issue before putting any trust to Merge Strategies.

Community
  • 1
  • 1
raratiru
  • 8,748
  • 4
  • 73
  • 113