Parameter Expansion

String Operators

String interpolation, ie inserting text stored in variables into a template which could be JSON or HTML, is something I do a lot and only recently discovered Bash has an embedded domain specific language, which the manual calls parameter expansion and my 1992 O’Reilly book calls “string operators”.

I stumbled on these courtesy shellcheck when I was making the common newby mistake of using sed to make changes to strings stored in bash variables, sending them as here strings for tasks like, say trimming whitespace.

#!/bin/bash

str=" 1234"
sed 's/^[[:space:]]*//' <<< "$str"

Running my short script through shellcheck produces

In ws_trim_sed.sh line 4:
sed 's/^[[:space:]]*//' <<< "$str"
^-- SC2001 (style): See if you can use ${variable//search/replace} instead.

For more information:
  https://www.shellcheck.net/wiki/SC2001 -- See if you can use ${variable//se...

The shellcheck wiki entry SC2001 explains that if the text is already stored in a bash variable, transforming it with parameter expansion is less overhead than filtering through sed.

Its suggestion ${parameter//pattern/string} isn’t particularly helpful since what’s better here is using the form ${parameter#word}, but at least it pointed me in a new, better direction.

${… DSL goes here …}

While I was aware that if I wanted to insert the content of a bash variable somewhere where it might be seen as part of a longer string, ie no separating delimeter, instead of $varname I could use the slightly more verbose ${varname}. What I only recently learnt was the curly brackets let me accomplish all kinds of text transformations that I’d been using grep, sed, tr and awk for, assuming I knew how.

As is unfortunately common, the options are bewildering — involving metacharacters #,%,:,!,/,@,* to name a few — and illustrative examples few, which is why I’m keeping notes as I discover them.

${!varname}

Placing an exclamation mark before the varname causes indirect expansion. I’ve found 2 common uses for this. Firstly, to get the key when iterating through associative arrays.

In my examples, I used for ks in "${!arr[@]}"; do...done and for ks in "${!arr[*]}"; do...done just to find out that case using @ or * make no difference. My old O’Reilly book came in handy to explain the very “subtle but important” difference between @ and *.

“$*” is a single string that consists of all the positional parameters, separated by the first character in the environment variable IFS (internal field separator), which is a space, TAB and NEWLINE by default.

So one use of * would be IFS=',' echo "$*" to change space separated arguments to comma separated arguments.

“$@” is equal to N separate double quoated strings separated by spaces, ie “$1” “$2” “$3” … “$N”.

The second use of ${!varname} wasn’t obvious to me, and it took me a while to figure out it was the way to write bash functions that altered the content of variables passed by name.

# zatime 'arr["\"startDate\""]'
function zatime {
  eval "$1=$(TZ=Africa/Johannesburg date -d "${!1}" +%Y-%m-%dT%H:%M:%S+02:00)"
}

It seemed a bit counter-intuitive to me that when I want to get the value of a variable passed by name to a function, I use the ! prefix, but that’s how it works.

$

Extracting text

Default values

Substring extraction

#!/usr/bin/bash

ExampleGroup 'doing grep -o with string operators'

  Example 'extract '
    str='/home/roblaing/webapps2/joeblog/content/events/tamara-dey-the-maslow-hotel-lacuna-bistro-202402141830/schema.json'
    IFS='/' read -ra arr <<< "$str"
    When call echo "${arr[7]}"
    The output should eq 'tamara-dey-the-maslow-hotel-lacuna-bistro-202402141830'
  End

End

Transformations

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

End

Globs

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion
# https://www.cyberciti.biz/tips/bash-shell-parameter-substitution-2.html

declare -Ag arr=(
    [\"@context\"]="https://schema.org"
    [\"@type\"]="MusicEvent"
    [\"location\",\"@type\"]="MusicVenue"
    [\"location\",\"name\"]="Chicago Symphony Center"
    [\"location\",\"address\"]="220 S. Michigan Ave, Chicago, Illinois, USA"
    [\"name\"]="Shostakovich Leningrad"
    [\"offers\",\"@type\"]="Offer"
    [\"offers\",\"url\"]="/examples/ticket/12341234"
    [\"offers\",\"price\"]="40"
    [\"offers\",\"priceCurrency\"]="USD"
    [\"offers\",\"availability\"]="https://schema.org/InStock"
    [\"performer\",0,\"@type\"]="MusicGroup"
    [\"performer\",0,\"name\"]="Chicago Symphony Orchestra"
    [\"performer\",0,\"sameAs\",0]="http://cso.org/"
    [\"performer\",0,\"sameAs\",1]="http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
    [\"performer\",1,\"@type\"]="Person"
    [\"performer\",1,\"image\"]="/examples/jvanzweden_s.jpg"
    [\"performer\",1,\"name\"]="Jaap van Zweden"
    [\"performer\",1,\"sameAs\"]="http://www.jaapvanzweden.com/"
    [\"startDate\"]="2014-05-23T20:00"
    [\"workPerformed\",0,\"@type\"]="CreativeWork"
    [\"workPerformed\",0,\"name\"]="Britten Four Sea Interludes and Passacaglia from Peter Grimes"
    [\"workPerformed\",0,\"sameAs\"]="http://en.wikipedia.org/wiki/Peter_Grimes"
    [\"workPerformed\",1,\"@type\"]="CreativeWork"
    [\"workPerformed\",1,\"name\"]="Shostakovich Symphony No. 7 (Leningrad)"
    [\"workPerformed\",1,\"sameAs\"]="http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
)

  Example 'Basic form ${parameter}'
    foo="bar"
    When call echo "${foo}" # brackets are unecessary in this example
    The output should eq "bar"
  End

  Example 'default value of unset variable is empty string'
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'Basic array lookup'
    When call echo "${arr["\"performer\",1,\"sameAs\""]}"
    The output should eq "http://www.jaapvanzweden.com/"
  End

#  Example 'Basic array lookup, no surrounding quotes in key, works but messes up syntax highlighting'
#    When call echo "${arr[\"performer\",1,\"sameAs\"]}"
#    The output should eq "http://www.jaapvanzweden.com/"
#  End

# Set default values if variable unset
  Example 'default value is substituted with ${parameter:-default}'
    When call echo "${foo:-default value}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    echo "${foo:-default value}"
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'default value is assigned with ${parameter:=default}'
    echo "${foo:=default value}"
    When call echo "${foo}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    foo="bar"
    When call echo "${foo:-default value}"
    The output should eq "bar"
  End

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

# TODO substitution examples

End

trimming

After trying to figure things out myself, I worked through the examples in the Pure Bash Bible where I encountered lots of syntax I’d never seen before.

#!/bin/bash

trim_string() {
    # Usage: trim_string "   example   string    "
    : "${1#"${1%%[![:space:]]*}"}"
    : "${_%"${_##*[![:space:]]}"}"
    printf '%s\n' "$_"
}


trim_string "    Hello,  World    "

First off, there’s the leading colon. An answer to What is the purpose of the : (colon) GNU Bash builtin?

A useful application for : is if you’re only interested in using parameter expansions for their side-effects rather than actually passing their result to a command.

defaults

Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries.

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

  Example 'Basic form ${parameter}'
    foo="bar"
    When call echo "${foo}" # brackets are unecessary in this example
    The output should eq "bar"
  End

  Example 'default value of unset variable is empty string'
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'default value is substituted with ${parameter:-default}'
    When call echo "${foo:-default value}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    foo="bar"
    When call echo "${foo:-default value}"
    The output should eq "bar"
  End

  Example ':- doesn`t alter the variable`s value'
    echo "${foo:-default value}"
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'Whereas := does'
    echo "${foo:=default value}"
    When call echo "${foo}"
    The output should eq "default value"
  End


End

slicing

Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation.

The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is:

${parameter:offset}
${parameter:offset:length}

String slices

Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart.

Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript. Negative lengths are actually offsets from the end.

transforming

I only discovered courtesy of ostechnix’s tutorial that Bash offers two ways of doing case conversion:

Lowercasing

${varname@L}

or

${varname,,}

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example 'lower to upper ${parameter@U}'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example 'lower to upper ${parameter^^}'
    foo="bar"
    When call echo "${foo^^}"
    The output should eq "BAR"
  End


  Example 'Title lower case ${parameter@u}'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example 'Title lower case ${parameter^}'
    foo="bar"
    When call echo "${foo^}"
    The output should eq "Bar"
  End

  Example 'upper to lower ${parameter@L}'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'upper to lower ${parameter,,}'
    foo="BAR"
    When call echo "${foo,,}"
    The output should eq "bar"
  End

  Example 'First letter to lower ${parameter,}'
    foo="BAR"
    When call echo "${foo,}"
    The output should eq "bAR"
  End

  Example 'Transpose case ${parameter~~}'
    foo="bAR"
    When call echo "${foo~~}"
    The output should eq "Bar"
  End

  Example 'Transpose first letter ${parameter~}'
    foo="bar"
    When call echo "${foo~}"
    The output should eq "Bar"
  End


  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

End