features-and-bugs

ShellCheck

The ShellCheck project identifies common bugs and warnings for your shell scripts. It is recommended for all scripts, large or small.

Command Substitution

Use $(command) instead of backticks.

Nested backticks require escaping the inner ones with \ . The $(command) format doesn't change when nested and is easier to read.

Example:

# This is preferred:
var="$(command "$(command1)")"

# This is not:
var="`command \`command1\``"

Test, [ … ], and [[ … ]]

[[ … ]] is preferred over [ … ], test and /usr/bin/[.

[[ … ]] reduces errors as no pathname expansion or word splitting takes place between [[ and ]]. In addition, [[ … ]] allows for regular expression matching, while [ … ] does not.

# This ensures the string on the left is made up of characters in
# the alnum character class followed by the string name.
# Note that the RHS should not be quoted here.
if [[ "filename" =~ ^[[:alnum:]]+name ]]; then
  echo "Match"
fi

# This matches the exact pattern "f*" (Does not match in this case)
if [[ "filename" == "f*" ]]; then
  echo "Match"
fi

# This gives a "too many arguments" error as f* is expanded to the
# contents of the current directory
if [ "filename" == f* ]; then
  echo "Match"
fi

For the gory details, see E14 at http://tiswww.case.edu/php/chet/bash/FAQ

Testing Strings

Use quotes rather than filler characters where possible.

Bash is smart enough to deal with an empty string in a test. So, given that the code is much easier to read, use tests for empty/non-empty strings or empty strings rather than filler characters.

# Do this:
if [[ "${my_var}" == "some_string" ]]; then
  do_something
fi

# -z (string length is zero) and -n (string length is not zero) are
# preferred over testing for an empty string
if [[ -z "${my_var}" ]]; then
  do_something
fi

# This is OK (ensure quotes on the empty side), but not preferred:
if [[ "${my_var}" == "" ]]; then
  do_something
fi

# Not this:
if [[ "${my_var}X" == "some_stringX" ]]; then
  do_something
fi

To avoid confusion about what you're testing for, explicitly use -z or -n.

# Use this
if [[ -n "${my_var}" ]]; then
  do_something
fi

# Instead of this
if [[ "${my_var}" ]]; then
  do_something
fi

For clarity, use == for equality rather than = even though both work. The former encourages the use of [[ and the latter can be confused with an assignment. However, be careful when using < and > in [[ … ]] which performs a lexicographical comparison. Use (( … )) or -lt and -gt for numerical comparison.

# Use this
if [[ "${my_var}" == "val" ]]; then
  do_something
fi

if (( my_var > 3 )); then
  do_something
fi

if [[ "${my_var}" -gt 3 ]]; then
  do_something
fi

# Instead of this
if [[ "${my_var}" = "val" ]]; then
  do_something
fi

# Probably unintended lexicographical comparison.
if [[ "${my_var}" > 3 ]]; then
  # True for 4, false for 22.
  do_something
fi

Wildcard Expansion of Filenames

Use an explicit path when doing wildcard expansion of filenames.

As filenames can begin with a -, it's a lot safer to expand wildcards with ./* instead of *.

# Here's the contents of the directory:
# -f  -r  somedir  somefile

# Incorrectly deletes almost everything in the directory by force
psa@bilby$ rm -v *
removed directory: `somedir'
removed `somefile'

# As opposed to:
psa@bilby$ rm -v ./*
removed `./-f'
removed `./-r'
rm: cannot remove `./somedir': Is a directory
removed `./somefile'

Eval

eval should be avoided.

Eval munges the input when used for assignment to variables and can set variables without making it possible to check what those variables were.

# What does this set?
# Did it succeed? In part or whole?
eval $(set_my_variables)

# What happens if one of the returned values has a space in it?
variable="$(eval some_function)"

Arrays

Bash arrays should be used to store lists of elements, to avoid quoting complications. This particularly applies to argument lists. Arrays should not be used to facilitate more complex data structures (see When to use Shell above).

Arrays store an ordered collection of strings, and can be safely expanded into individual elements for a command or loop.

Using a single string for multiple command arguments should be avoided, as it inevitably leads to authors using eval or trying to nest quotes inside the string, which does not give reliable or readable results and leads to needless complexity.

# An array is assigned using parentheses, and can be appended to
# with +=( … ).
declare -a flags
flags=(--foo --bar='baz')
flags+=(--greeting="Hello ${name}")
mybinary "${flags[@]}"

# Don’t use strings for sequences.
flags='--foo --bar=baz'
flags+=' --greeting="Hello world"'  # This won’t work as intended.
mybinary ${flags}

# Command expansions return single strings, not arrays. Avoid
# unquoted expansion in array assignments because it won’t
# work correctly if the command output contains special
# characters or whitespace.

# This expands the listing output into a string, then does special keyword
# expansion, and then whitespace splitting.  Only then is it turned into a
# list of words.  The ls command may also change behavior based on the user's
# active environment!
declare -a files=($(ls /directory))

# The get_arguments writes everything to STDOUT, but then goes through the
# same expansion process above before turning into a list of arguments.
mybinary $(get_arguments)

Arrays Pros

• Using Arrays allows lists of things without confusing quoting semantics. Conversely, not using arrays leads to misguided attempts to nest quoting inside a string.
• Arrays make it possible to safely store sequences/lists of arbitrary strings, including strings containing whitespace.

Arrays Cons

Using arrays can risk a script’s complexity growing.

Arrays Decision

Arrays should be used to safely create and pass around lists. In particular, when building a set of command arguments, use arrays to avoid confusing quoting issues. Use quoted expansion – "${array[@]}" – to access arrays. However, if more advanced data manipulation is required, shell scripting should be avoided altogether; see above.

Pipes to While

Use process substitution or the readarray builtin (bash4+) in preference to piping to while. Pipes create a subshell, so any variables modified within a pipeline do not propagate to the parent shell.

The implicit subshell in a pipe to while can introduce subtle bugs that are hard to track down.

last_line='NULL'
your_command | while read -r line; do
  if [[ -n "${line}" ]]; then
    last_line="${line}"
  fi
done

# This will always output 'NULL'!
echo "${last_line}"

Using process substitution also creates a subshell. However, it allows redirecting from a subshell to a while without putting the while (or any other command) in a subshell.

last_line='NULL'
while read line; do
  if [[ -n "${line}" ]]; then
    last_line="${line}"
  fi
done < <(your_command)

# This will output the last non-empty line from your_command
echo "${last_line}"

Alternatively, use the readarray builtin to read the file into an array, then loop over the array's contents. Notice that (for the same reason as above) you need to use a process substitution with readarray rather than a pipe, but with the advantage that the input generation for the loop is located before it, rather than after.

last_line='NULL'
readarray -t lines < <(your_command)
for line in "${lines[@]}"; do
  if [[ -n "${line}" ]]; then
    last_line="${line}"
  fi
done
echo "${last_line}"

Note: Be cautious using a for-loop to iterate over output, as in for var in $(...), as the output is split by whitespace, not by line. Sometimes you will know this is safe because the output can't contain any unexpected whitespace, but where this isn't obvious or doesn't improve readability (such as a long command inside $(...)), a while read loop or readarray is often safer and clearer.

Arithmetic

Always use (( … )) or $(( … )) rather than let or $[ … ] or expr.

Never use the $[ … ] syntax, the expr command, or the let built-in.

< and > don't perform numerical comparison inside [[ … ]] expressions (they perform lexicographical comparisons instead; see Testing Strings). For preference, don't use [[ … ]] at all for numeric comparisons, use (( … )) instead.

It is recommended to avoid using (( … )) as a standalone statement, and otherwise be wary of its expression evaluating to zero

• particularly with set -e enabled. For example, set -e; i=0; (( i++ )) will cause the shell to exit.

# Simple calculation used as text - note the use of $(( … )) within
# a string.
echo "$(( 2 + 2 )) is 4"

# When performing arithmetic comparisons for testing
if (( a < b )); then
  …
fi

# Some calculation assigned to a variable.
(( i = 10 * j + 400 ))

# This form is non-portable and deprecated
i=$[2 * 10]

# Despite appearances, 'let' isn't one of the declarative keywords,
# so unquoted assignments are subject to globbing wordsplitting.
# For the sake of simplicity, avoid 'let' and use (( … ))
let i="2 + 2"

# The expr utility is an external program and not a shell builtin.
i=$( expr 4 + 4 )

# Quoting can be error prone when using expr too.
i=$( expr 4 '*' 4 )

Stylistic considerations aside, the shell's built-in arithmetic is many times faster than expr.

When using variables, the ${var} (and $var) forms are not required within $(( … )). The shell knows to look up var for you, and omitting the ${…} leads to cleaner code. This is slightly contrary to the previous rule about always using braces, so this is a recommendation only.

# N.B.: Remember to declare your variables as integers when
# possible, and to prefer local variables over globals.
local -i hundred=$(( 10 * 10 ))
declare -i five=$(( 10 / 2 ))

# Increment the variable "i" by three.
# Note that:
#  - We do not write ${i} or $i.
#  - We put a space after the (( and before the )).
(( i += 3 ))

# To decrement the variable "i" by five:
(( i -= 5 ))

# Do some complicated computations.
# Note that normal arithmetic operator precedence is observed.
hr=2
min=5
sec=30
echo $(( hr * 3600 + min * 60 + sec )) # prints 7530 as expected