Is there a general way to add/prepend/remove paths from general environment variables in bash?

Question

I am wondering if anyone has a good way to add/remove paths from environment variables in bash (not just the PATH variable).

My situation is that I am implementing a simple module-like system on a cluster where I do not have admin privileges. I want to be able to write scripts like:

openmpi_module.sh

#!/bin/bash

if [ $1 == "load" ]; then
  path_prepend PATH "$HOME/apps/openmpi-1.8.1-gcc-4.9.0/bin"
  path_prepend CPATH "$HOME/apps/openmpi-1.8.1-gcc-4.9.0/include"
  ...
fi

if [ $1 == "unload" ]; then
  path_remove PATH "$HOME/apps/openmpi-1.8.1-gcc-4.9.0/bin"
  path_remove CPATH "$HOME/apps/openmpi-1.8.1-gcc-4.9.0/include"
  ...
fi

I have found questions dealing with modifying the PATH variable specifically, but not a general environment variable. It seems like a general version of those solutions would be a useful tool to have.

My current solution is to source the following in my .bash_profile, but I am wondering if anyone has a more elegant solution (short of installing a local copy of the actual Modules system). Mostly I feel uncomfortable with the use of so many evals and echos, a practice I prefer to avoid.

#!/bin/bash
# modified from https://stackoverflow.com/questions/370047/

function path_remove () {
eval export $(echo $1=\$\(echo -n $(echo -n "\$$1 | awk -v RS=: -v ORS=: '\$0 != \"'$2'\"' | sed 's/:\$//'")\))
}

function path_append () {
path_remove $1 $2
eval export $1="\$$1:$2"
}

function path_prepend () {
path_remove $1 $2
eval export $1="$2:\$$1"
}

Indeed, `eval`s are evil, but you don't need them for this; see BashFAQ #6, on working with indirect variables: http://mywiki.wooledge.org/BashFAQ/006 — Charles Duffy, Jul 01 '14 at 16:50
BTW, `echo -n` and `echo -E` are among the few places where bash doesn't just extend POSIX, but actually breaks compatibility with the standard. It's very much best to avoid them, and use `printf` whenever you want to do anything fancy. — Charles Duffy, Jul 01 '14 at 17:05

score 3 · Accepted Answer · edited May 23 '17 at 12:05

Here are compact bash functions, which:

avoid eval by using printf -v for setting a variable indirectly, as in @Charles Duffy's excellent answer.
use just parameter expansion to manipulate the path list.
are self-contained (they do not rely on each other - at the expense of duplicating some code).
force pre/appending, as in the OP's functions: i.e., if the specified path entry already exists but in a different position, it is forced into the desired position.

Caveat: Existing duplicate entries are only removed if they're not directly adjacent (see below for an alternative).

#!/usr/bin/env bash

# The functions below operate on PATH-like variables whose fields are separated
# with ':'.
# Note: The *name* of the PATH-style variable must be passed in as the 1st
#       argument and that variable's value is modified *directly*.

# SYNOPSIS: path_prepend varName path
# Note: Forces path into the first position, if already present.
#       Duplicates are removed too, unless they're directly adjacent.
# EXAMPLE: path_prepend PATH /usr/local/bin
path_prepend() {
  local aux=":${!1}:"
  aux=${aux//:$2:/:}; aux=${aux#:}; aux=${aux%:}
  printf -v "$1" '%s' "${2}${aux:+:}${aux}"  
}

# SYNOPSIS: path_append varName path
# Note: Forces path into the last position, if already present.
#       Duplicates are removed too, unless they're directly adjacent.
# EXAMPLE: path_append PATH /usr/local/bin
path_append() {
  local aux=":${!1}:"
  aux=${aux//:$2:/:}; aux=${aux#:}; aux=${aux%:}
  printf -v "$1" '%s' "${aux}${aux:+:}${2}"
}

# SYNOPSIS: path_remove varName path
# Note: Duplicates are removed too, unless they're directly adjacent.
# EXAMPLE: path_remove PATH /usr/local/bin
path_remove() {
  local aux=":${!1}:"
  aux=${aux//:$2:/:}; aux=${aux#:}; aux=${aux%:}
  printf -v "$1" '%s' "$aux"
}

If you do need to deal with directly adjacent duplicates and/or want to be able to specify a different field separator, here are more elaborate functions that use the array technique from @konsolebox' helpful answer.

# SYNOPSIS: field_prepend varName fieldVal [sep]
#   SEP defaults to ':'
# Note: Forces fieldVal into the first position, if already present.
#       Duplicates are removed, too.
# EXAMPLE: field_prepend PATH /usr/local/bin
field_prepend() {
    local varName=$1 fieldVal=$2 IFS=${3:-':'} auxArr
    read -ra auxArr <<< "${!varName}"
    for i in "${!auxArr[@]}"; do
        [[ ${auxArr[i]} == "$fieldVal" ]] && unset auxArr[i]
    done
    auxArr=("$fieldVal" "${auxArr[@]}")
    printf -v "$varName" '%s' "${auxArr[*]}"
}

# SYNOPSIS: field_append varName fieldVal [sep]
#   SEP defaults to ':'
# Note: Forces fieldVal into the last position, if already present.
#       Duplicates are removed, too.
# EXAMPLE: field_append PATH /usr/local/bin
field_append() {
    local varName=$1 fieldVal=$2 IFS=${3:-':'} auxArr
    read -ra auxArr <<< "${!varName}"
    for i in "${!auxArr[@]}"; do
        [[ ${auxArr[i]} == "$fieldVal" ]] && unset auxArr[i]
    done
    auxArr+=("$fieldVal")
    printf -v "$varName" '%s' "${auxArr[*]}"
}

# SYNOPSIS: field_remove varName fieldVal [sep]
#   SEP defaults to ':'
# Note: Duplicates are removed, too.
# EXAMPLE: field_remove PATH /usr/local/bin
field_remove() {
    local varName=$1 fieldVal=$2 IFS=${3:-':'} auxArr
    read -ra auxArr <<< "${!varName}"
    for i in "${!auxArr[@]}"; do
        [[ ${auxArr[i]} == "$fieldVal" ]] && unset auxArr[i]
    done
    printf -v "$varName" '%s' "${auxArr[*]}"
}

@konsolebox: Re splitting idea: Yes, it's yours, which is why I'm giving you credit in my answer - I like your answer in principle, but the things I don't like about it I tried to correct in my answer: (a) use of `printf -v` rather than `eval`, (b) use of lowercase variable names, (c) better handling of `$IFS`. As I commented, I also don't like the global `shopt -s extglob`, but that problem goes away with switching from `eval` to `printf -v`. If it weren't for these issues, I'd up-vote your answer. — mklement0, Jul 02 '14 at 04:37

Charles Duffy · Answer 2 · 2014-07-01T17:08:39.480

2

export sets a flag specifying that a variable should be in the environment. If it's already there, though, updates will always be passed through to the environment; you don't need to do anything else.

Thus:

PATH=/new/value:$PATH

or

PATH=$PATH:/new/value

...is entirely sufficient, unless you want to add your own logic (as around deduplication).

If you want to act only if no duplicate values exist, you might write something like the following:

prepend() {
  local var=$1
  local val=$2
  local sep=${3:-":"}
  [[ ${!var} =~ (^|"$sep")"$val"($|"$sep") ]] && return # already present
  [[ ${!var} ]] || { printf -v "$var" '%s' "$val" && return; } # empty
  printf -v "$var" '%s%s%s' "$val" "$sep" "${!var}" # prepend
}

append() {
  local var=$1
  local val=$2
  local sep=${3:-":"}
  [[ ${!var} =~ (^|"$sep")"$val"($|"$sep") ]] && return # already present
  [[ ${!var} ]] || { printf -v "$var" '%s' "$val" && return; } # empty
  printf -v "$var" '%s%s%s' "${!var}" "$sep" "${val}" # append
}

remove() {
  local var=$1
  local val=$2
  local sep=${3:-":"}
  while [[ ${!var} =~ (^|.*"$sep")"$val"($|"$sep".*) ]]; do
    if [[ ${BASH_REMATCH[1]} && ${BASH_REMATCH[2]} ]]; then
      # match is between both leading and trailing content
      printf -v "$var" '%s%s' "${BASH_REMATCH[1]%$sep}" "${BASH_REMATCH[2]}"
    elif [[ ${BASH_REMATCH[1]} ]]; then
      # match is at the end
      printf -v "$var" "${BASH_REMATCH[1]%$sep}"
    else
      # match is at the beginning
      printf -v "$var" "${BASH_REMATCH[2]#$sep}"
    fi
  done
}

...used as:

prepend PATH /usr/local/bin
remove PATH /usr/local/bin

Note that:

There's no need for the function keyword, which breaks POSIX compatibility for no good reason. (Other things we do break POSIX, but actually add value in some way).
Our regular expression quotes things which should be interpreted literally, and leaves things which should be treated as regex characters unquoted. Note that this behavior is only entirely consistent in versions of bash after 3.2; if you need compatibility with older releases, some updates are called for.

edited Jul 01 '14 at 17:08

answered Jul 01 '14 at 16:43

Charles Duffy

280,126
43
390
441

There's no need to go for POSIX if your script targets Bash. People may prefer using `function` to explicitly show that the script could only run with it. Some may also find `()` in the original sh syntax not really appropriate for its means. It would actually be better to suggest not using `()` when the function is already being declared with `function`. – konsolebox Jul 01 '14 at 19:42
2

@konsolebox, I tend to consider making deviations from POSIX sh knowing and intentional, rather than habit, a beneficial behavior in and of itself. So -- if someone wants to use `function` because they think it's more readable, more power to them. If they use `function` because they think it's the canonical Right Way to define functions in bash, I think that's considerably more questionable. – Charles Duffy Jul 01 '14 at 19:51
2

+1 for a solid answer with robust, generic functions to boot. Quibble: The names `prepend()` and `append()` suggest _unconditional_ actions, which is not the case - perhaps something like `prependUnlessPresent()`? Conversely, sometimes you want to _force_ prepending/appending (i.e., remove existing, if present, then prepend/append) - indeed, it seems that that's what the OP wants. – mklement0 Jul 01 '14 at 20:10
@CharlesDuffy I'm not implying that you enumerated them well however I don't see how any of those preferences could be wrong or choosing to use them is wrong. If it's only about POSIX then it's nothing. Well you have your own preferences. If you opt to keep the unnecessary compatibility then you're free to have it yourself. Just a note: if a script really needs compatibility it can't just follow POSIX. POSIX is a standard that could never make itself enough due to the fact the old syntax shells are still active these days. – konsolebox Jul 01 '14 at 20:19
@mklement0 Yes and all these heavy use of repeating regex tests and redundant looping printf's just to avoid `eval` is just way too inefficient for a conservative solution which won't even give a safety difference. – konsolebox Jul 01 '14 at 20:25
@konsolebox, re: "won't even give a safety difference", if you can point to any vulnerability here, I'd love to hear it. As for the looping, the number of iterations is tied to the number of matches; in the common case (only one instance of the path to be removed), it's constant-time. – Charles Duffy Jul 01 '14 at 20:38
@CharlesDuffy I didn't say that it has a vulnerability. But neither a solution that uses `eval` properly could. – konsolebox Jul 01 '14 at 20:41
1

Sure, it's possible to use eval safely. It's also very, very easy to use eval unsafely. I know which kind of practice I want to promote. – Charles Duffy Jul 01 '14 at 20:41
@konsolebox, ...as for the "redundant, looping `printf`s", they're a 1:1 replacement for the way you're using `eval`, and the consensus best-practice within freenode's #bash channel (and the FAQ and wiki maintained thereby). – Charles Duffy Jul 01 '14 at 20:46
Not promoting `eval` is one's option. However telling people to completely avoid it for it being evil without grounds is a different thing. FAQs and Wikis don't apply to me as I never relied on them. Would I imagine them not promoting `eval` in exchange of very inefficient algorithms. Sure nevermind. The community is never always right - especially those that learned scripting late and perhaps even gathered their ideas from those posts of pioneers they see in forums. Funny people could even promote use of `declare` as a general tool to assign variables. Well good thing there's `-n` already. – konsolebox Jul 01 '14 at 21:21
@konsolebox, again, it's a 1:1 replacement. Each use of `eval` replaces exactly one use of `printf -v`, and the inverse, so your arguments about efficiency have zero merit. If you want to argue that I'm being inefficient in other respects you're welcome to do so, but there's no reasonable argument to be made that the effort I take to avoid using `eval` is contributing to that inefficiency. – Charles Duffy Jul 01 '14 at 23:24
@CharlesDuffy Repeatedly altering a string over an array is inefficient in itself. That's what I'm pointing about your use with `printf` and `BASH_REMATCH`'es there. – konsolebox Jul 02 '14 at 04:21
Yes I guess I'd admit this time that using `eval` doesn't really give much difference - for what turned out. – konsolebox Jul 02 '14 at 05:45
@konsolebox, sure, but it's only "repeatedly" during a removal operation when the path being removed exists more than once. That's a tiny enough corner case that I don't care. If it iterated over all colon-separated names in the array, you'd be right to call it crazy, but that's not what it's actually doing. – Charles Duffy Jul 02 '14 at 12:50

konsolebox · Answer 3 · 2014-07-01T22:01:42.920

0

The easiest way and probably most consistent is to split those strings first into arrays:

IFS=: read -ra T <<< "$PATH"

Then add elements to those arrays:

# Append

T+=("$SOMETHING")

# Prepend or Insert

T=("$SOMETHING" "${T[@]}")

# Remove

for I in "${!T[@]}"; do
    if [[ ${T[I]} == "$SOMETHING" ]]; then
        unset "T[$I]"
        break  ## You can skip breaking if you want to remove all matches not just the first one.
    fi
done

After that you can put it back with a safe eval:

IFS=: eval 'PATH="${T[*]}"'

Actually if you're a bit "conservative", you can save IFS first:

OLD_IFS=$IFS; IFS=:
PATH="${T[*]}"
IFS=$OLD_IFS

Functions:

shopt -s extglob

function path_append {
    local VAR=$1 ELEM=$2 T
    IFS=: read -ra T <<< "${!VAR}"
    T+=("$ELEM")
    [[ $VAR == [[:alpha:]_]*([[:alnum:]_]) ]] && IFS=: eval "$VAR=\${T[*]}"
}

function path_prepend {
    local VAR=$1 ELEM=$2 T
    IFS=: read -ra T <<< "${!VAR}"
    T=("$ELEM" "${T[@]}")
    [[ $VAR == [[:alpha:]_]*([[:alnum:]_]) ]] && IFS=: eval "$VAR=\${T[*]}"
}

function path_remove {
    local VAR=$1 ELEM=$2 T
    IFS=: read -ra T <<< "${!VAR}"
    for I in "${!T[@]}"; do
        [[ ${T[I]} == "$ELEM" ]] && unset "T[$I]"
    done
    [[ $VAR == [[:alpha:]_]*([[:alnum:]_]) ]] && IFS=: eval "$VAR=\${T[*]}"
}

edited Jul 01 '14 at 22:01

answered Jul 01 '14 at 19:16

konsolebox

72,135
12
99
105

Your extended globs for checking for valid variable names are too strict: they won't recognize single-character variable names such as `v`. Changing global state with `shopt -s extglob` (rather than localizing it inside the functions) is problematic. It's better not use all-uppercase variable names (even if local), to avoid conflicts with environment variables. – mklement0 Jul 01 '14 at 20:53
@mklement0 Changing global state with `extglob` can never be problematic as it's been somewhat the default behaviour already inside `[[ ]]` starting 4.1 and is now only about pathname expansions. Using them inside functions would be redundant. And I'm not sure if the target forms of variables is something to worry about especially that the OP wanted to change PATH and CPATH. – konsolebox Jul 01 '14 at 21:25
Re variable names: The OP merely says `not just the PATH variable`, which makes no assumptions about variable names. Given that your solution is _so close_ to recognizing _all_ valid variable names, why not just replace the 2nd `+` with `*` and thus cover single-character variable names, too? – mklement0 Jul 01 '14 at 21:34
Re global `extglob` state: subsequent filename globs not meant to be extglobs could then unexpectedly be interpreted as such. Also, relying on global state modified elsewhere makes your functions non-self-contained. Simply using `=~ ^[[:alpha:]_][[:alnum:]_]*$` instead would solve both problems (and would also work down to at least bash 3.2). – mklement0 Jul 01 '14 at 21:39
Well yeah that's one thing consider although that's unlikely to cause problems especially if you quote your variables properly. I could have opted to just use regex however I already started it with extended globs and I keep forgetting the troubles of Bash's regex before 4.0. Oh yes you could just use `*`. The better one is to no longer use `+()` in the first char like in my original solution: [`[[:alpha:]_]*([[:alnum:]_])`](http://sourceforge.net/p/playshell/code/ci/master/tree/source/patterns.sh). Just was lazy to look at it again. – konsolebox Jul 01 '14 at 21:50
1

Need I point out that using `printf -v` would moot the need to validate or sanitize destination variable names? `printf -v "$VAR" '%s' "${T[*]}"` is a fairly clean drop-in replacement. – Charles Duffy Jul 01 '14 at 23:29
@CharlesDuffy Where's the part where you altered the IFS? That would join with spaces. – konsolebox Jul 02 '14 at 04:19
@konsolebox: Just use a single `local IFS=:` at the top of your function. – mklement0 Jul 02 '14 at 04:31
@mklement0 Yes I saw your version. Using `printf` came to my mind but I never considered separating IFS from the line. Luckily `IFS` doesn't have to be modified more than once in this case and could be done similar to my `utils_merge` function. I started doing stuffs like this in 3.0 or even 2.05b so `printf` never really came as a natural practice to me for saving value to referred parameters, considering also that you can't do it with arrays - unless you use a loop. – konsolebox Jul 02 '14 at 05:37
@konsolebox, I proposed a replacement for the `eval`. That it doesn't replace other parts of your code as well seems... less than relevant. – Charles Duffy Jul 02 '14 at 12:48
1

Yes, it's unfortunate that you can't use `printf -v` with _array_ elements; in that case, you can use `IFS=$'\n' read -r -d '' "$varName"[ndx] <<<...`. You mentioned `declare -n` elsewhere, which solves the problem more elegantly for both array and non-array variables, but it requires bash 4.3+. What `utils_merge` function? – mklement0 Jul 02 '14 at 13:18
@mklement0 Sorry I was actually talking about `utils_implode`. Back then it had the form of `function utils_implode { local IFS=${2:-$' \t'}(NL)eval "$1=\"\${*:3}\""; }`. It was still a very wrong version. – konsolebox Jul 02 '14 at 15:08

Is there a general way to add/prepend/remove paths from general environment variables in bash?

openmpi_module.sh

3 Answers3