Parameter Expansion

String Operators

String interpolation, ie inserting text stored in variables into a template which could be JSON or HTML, is something I do a lot and only recently discovered Bash has an embedded domain specific language, which the manual calls parameter expansion and my 1992 O’Reilly book calls “string operators”.

I stumbled on these courtesy shellcheck when I was making the common newby mistake of using sed to make changes to strings stored in bash variables, sending them as here strings for tasks like, say trimming whitespace.

#!/bin/bash

str=" 1234"
sed 's/^[[:space:]]*//' <<< "$str"

Running my short script through shellcheck produces

In ws_trim_sed.sh line 4:
sed 's/^[[:space:]]*//' <<< "$str"
^-- SC2001 (style): See if you can use ${variable//search/replace} instead.

For more information:
  https://www.shellcheck.net/wiki/SC2001 -- See if you can use ${variable//se...

The shellcheck wiki entry SC2001 explains that if the text is already stored in a bash variable, transforming it with parameter expansion is less overhead than filtering through sed.

Its suggestion ${parameter//pattern/string} isn’t particularly helpful since what’s better here is using the form ${parameter#word}, but at least it pointed me in a new, better direction.

${… DSL goes here …}

While I was aware that if I wanted to insert the content of a bash variable somewhere where it might be seen as part of a longer string, ie no separating delimeter, instead of $varname I could use the slightly more verbose ${varname}. What I only recently learnt was the curly brackets let me accomplish all kinds of text transformations that I’d been using grep, sed, tr and awk for, assuming I knew how.

As is unfortunately common, the options are bewildering — involving metacharacters #,%,:,!,/,@,* to name a few — and illustrative examples few, which is why I’m keeping notes as I discover them.

${!varname}

Placing an exclamation mark before the varname causes indirect expansion. I’ve found 2 common uses for this. Firstly, to get the key when iterating through associative arrays.

In my examples, I used for ks in "${!arr[@]}"; do...done and for ks in "${!arr[*]}"; do...done just to find out that case using @ or * make no difference. My old O’Reilly book came in handy to explain the very “subtle but important” difference between @ and *.

“$*” is a single string that consists of all the positional parameters, separated by the first character in the environment variable IFS (internal field separator), which is a space, TAB and NEWLINE by default.

So one use of * would be IFS=',' echo "$*" to change space separated arguments to comma separated arguments.

“$@” is equal to N separate double quoated strings separated by spaces, ie “$1” “$2” “$3” … “$N”.

The second use of ${!varname} wasn’t obvious to me, and it took me a while to figure out it was the way to write bash functions that altered the content of variables passed by name.

# zatime 'arr["\"startDate\""]'
function zatime {
  eval "$1=$(TZ=Africa/Johannesburg date -d "${!1}" +%Y-%m-%dT%H:%M:%S+02:00)"
}

It seemed a bit counter-intuitive to me that when I want to get the value of a variable passed by name to a function, I use the ! prefix, but that’s how it works.

$

Extracting text

Default values

Substring extraction

#!/usr/bin/bash

ExampleGroup 'doing grep -o with string operators'

  Example 'extract '
    str='/home/roblaing/webapps2/joeblog/content/events/tamara-dey-the-maslow-hotel-lacuna-bistro-202402141830/schema.json'
    IFS='/' read -ra arr <<< "$str"
    When call echo "${arr[7]}"
    The output should eq 'tamara-dey-the-maslow-hotel-lacuna-bistro-202402141830'
  End

End

Transformations

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

End

Globs

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion
# https://www.cyberciti.biz/tips/bash-shell-parameter-substitution-2.html

declare -Ag arr=(
    [\"@context\"]="https://schema.org"
    [\"@type\"]="MusicEvent"
    [\"location\",\"@type\"]="MusicVenue"
    [\"location\",\"name\"]="Chicago Symphony Center"
    [\"location\",\"address\"]="220 S. Michigan Ave, Chicago, Illinois, USA"
    [\"name\"]="Shostakovich Leningrad"
    [\"offers\",\"@type\"]="Offer"
    [\"offers\",\"url\"]="/examples/ticket/12341234"
    [\"offers\",\"price\"]="40"
    [\"offers\",\"priceCurrency\"]="USD"
    [\"offers\",\"availability\"]="https://schema.org/InStock"
    [\"performer\",0,\"@type\"]="MusicGroup"
    [\"performer\",0,\"name\"]="Chicago Symphony Orchestra"
    [\"performer\",0,\"sameAs\",0]="http://cso.org/"
    [\"performer\",0,\"sameAs\",1]="http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
    [\"performer\",1,\"@type\"]="Person"
    [\"performer\",1,\"image\"]="/examples/jvanzweden_s.jpg"
    [\"performer\",1,\"name\"]="Jaap van Zweden"
    [\"performer\",1,\"sameAs\"]="http://www.jaapvanzweden.com/"
    [\"startDate\"]="2014-05-23T20:00"
    [\"workPerformed\",0,\"@type\"]="CreativeWork"
    [\"workPerformed\",0,\"name\"]="Britten Four Sea Interludes and Passacaglia from Peter Grimes"
    [\"workPerformed\",0,\"sameAs\"]="http://en.wikipedia.org/wiki/Peter_Grimes"
    [\"workPerformed\",1,\"@type\"]="CreativeWork"
    [\"workPerformed\",1,\"name\"]="Shostakovich Symphony No. 7 (Leningrad)"
    [\"workPerformed\",1,\"sameAs\"]="http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
)

  Example 'Basic form ${parameter}'
    foo="bar"
    When call echo "${foo}" # brackets are unecessary in this example
    The output should eq "bar"
  End

  Example 'default value of unset variable is empty string'
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'Basic array lookup'
    When call echo "${arr["\"performer\",1,\"sameAs\""]}"
    The output should eq "http://www.jaapvanzweden.com/"
  End

#  Example 'Basic array lookup, no surrounding quotes in key, works but messes up syntax highlighting'
#    When call echo "${arr[\"performer\",1,\"sameAs\"]}"
#    The output should eq "http://www.jaapvanzweden.com/"
#  End

# Set default values if variable unset
  Example 'default value is substituted with ${parameter:-default}'
    When call echo "${foo:-default value}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    echo "${foo:-default value}"
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'default value is assigned with ${parameter:=default}'
    echo "${foo:=default value}"
    When call echo "${foo}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    foo="bar"
    When call echo "${foo:-default value}"
    The output should eq "bar"
  End

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

# TODO substitution examples

End

defaults

Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries.

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

  Example 'Basic form ${parameter}'
    foo="bar"
    When call echo "${foo}" # brackets are unecessary in this example
    The output should eq "bar"
  End

  Example 'default value of unset variable is empty string'
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'default value is substituted with ${parameter:-default}'
    When call echo "${foo:-default value}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    foo="bar"
    When call echo "${foo:-default value}"
    The output should eq "bar"
  End

  Example ':- doesn`t alter the variable`s value'
    echo "${foo:-default value}"
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'Whereas := does'
    echo "${foo:=default value}"
    When call echo "${foo}"
    The output should eq "default value"
  End


End

slicing

Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation.

The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is:

${parameter:offset}
${parameter:offset:length}

String slices

Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart.

Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript. Negative lengths are actually offsets from the end.

transforming

Lowercasing

${varname@L}

#!/bin/bash

# find_schema 'artists' 'Arno Carstens'
# Returns just the first match
function find_schema {
  local path
  local file
  path="${2@L}"
  path="*${path//[^[:alnum:]]/\*}*"
  path="/usr/local/webapps2/*/content/${1}/${path}/schema.json"
  for file in $path; do
    if [[ -f $file ]]; then
      echo "$file"
      return 0
    fi
  done
}

ExampleGroup 'list artist and venue schema.json files'

  Example 'Find schema.json for "Arno Carstens"'
    When call find_schema 'artists' 'Arno Carstens'
    The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json"
  End

  Example 'Find schema.json for "No File Created"'
    When call find_schema 'artists' 'No File Created'
    The output should eq ''
  End

  Example 'Find schema.json for "Black Cat Bones"'
    When call find_schema 'artists' 'Black Cat Bones'
    The output should eq '/usr/local/webapps2/joeblog/content/artists/blackcatbones/schema.json'
  End

Example 'Find schema.json for "Bailey`s"'
    When call find_schema 'venues' "Bailey's"
    The output should eq '/usr/local/webapps2/joeblog/content/venues/baileys/schema.json'
  End
  

End

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

End

trimming

The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers.

As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples).

We need Bash’s composite patterns which are only available if

shopt -s extglob

is set. Composite patterns allow +(pattern-list) as used below to match possibly more than one space.