Frontier Software

Iteration

Iterating over JSON arrays via Bash associative-arrays requires two more functions to the json-utils to json2array and array2json in the top section.

A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays.

function getkeys {
  local -n arr
  arr=$1
  local keys
  keys=$(for key in "${!arr[@]}"; do
    if [[ $key =~ $2 ]]; then
      echo "$key"
    fi
  done | sort -u)
  if [[ -z $keys ]]; then 
    echo "Error: $2 is not a valid key" >&2
    return 1;
  fi
  echo "$keys"
}

The return 1 part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.

Here are some examples to illustrate how they are used.

#!/bin/bash

source /usr/local/lib/json-utils.sh

declare -A myarr
json2array myarr "$(< ../musicevent.json)"

ExampleGroup 'extract an array from a JSON object'

  Example 'list all elements in the JSON "performer": [...] array'
    When call getkeys myarr '"performer"'
    The output should equal '"performer",0,"name"
"performer",0,"sameAs",0
"performer",0,"sameAs",1
"performer",0,"@type"
"performer",1,"image"
"performer",1,"name"
"performer",1,"sameAs"
"performer",1,"@type"'
    The status should be success
    The error should be blank
  End

  Example 'list all elements in the JSON "offers" object, which is not an array in the given example'
    When call getkeys myarr '"offers"'
    The output should equal '"offers","availability"
"offers","price"
"offers","priceCurrency"
"offers","@type"
"offers","url"'
    The status should be success
    The error should be blank
  End

  Example 'what happens if getkeys receives a typo key?'
    When call getkeys myarr '"performers"'
    The stderr should equal 'Error: "performers" is not a valid key'
    The status should be failure
  End

End

I then built on my getkeys function to write a function arraylength needed in the next section appending. This function represents my first attempt at throwing errors in bash functions. It checks the given key is indeed a JSON array.

function arraylength {
  local rows
  local cols
  readarray -t rows <<< "$(getkeys "$1" "$2")"
  if [[ ${rows[*]} ]]; then
    readarray -d ',' -t cols <<< "${rows[-1]}"
    if [[ ! ${cols[1]} =~ ^[0-9]+$ ]]; then
      echo "Error: $2 is not an array" >&2
      return 2
    fi
    echo $(( cols[1] + 1 ))
  else
    return $?
  fi
}

The above exits with 1 with the error is unset key, and 2 if the key is not an array. This is handy for routing different errors to different handlers:

function munge_performer {
  local len
  len="$(arraylength event_arr '"performer"')"
  case $? in
   0) update_performers "$len" ;;
   1) echo "no entry for performer" ;;
   2) object2array event_arr '"performer"' 
      update_performers 1
      ;;
  esac
}

#!/bin/bash

source /usr/local/lib/json-utils.sh

declare -A myarr
json2array myarr "$(< ../musicevent.json)"

ExampleGroup 'extract an array from a JSON object'

  Example 'list all elements in the JSON "performer": [...] array'
    When call getkeys myarr '"performer"'
    The output should equal '"performer",0,"name"
"performer",0,"sameAs",0
"performer",0,"sameAs",1
"performer",0,"@type"
"performer",1,"image"
"performer",1,"name"
"performer",1,"sameAs"
"performer",1,"@type"'
    The status should be success
    The error should be blank
  End

  Example 'list all elements in the JSON "offers" object, which is not an array in the given example'
    When call getkeys myarr '"offers"'
    The output should equal '"offers","availability"
"offers","price"
"offers","priceCurrency"
"offers","@type"
"offers","url"'
    The status should be success
    The error should be blank
  End

  Example 'arraylength uses getkeys to get JSON array length'
    When call arraylength myarr '"performer"'
    The output should equal 2
    The status should be success
    The error should be blank
  End

 Example 'Unset key should throw an error'
    When call arraylength myarr '"performers"'
    The stderr should equal 'Error: "performers" is not a valid key'
    The status should be failure
  End

  Example 'arraylength of "offers" in the given example should throw an error'
    When call arraylength myarr '"offers"'
    The stderr should equal 'Error: "offers" is not an array'
    The status should be failure
  End


End

A reminder of what the JSON array looks like:

{
  "@context": "https://schema.org",
  "@type": "MusicEvent",
  "location": {
    "@type": "MusicVenue",
    "name": "Chicago Symphony Center",
    "address": "220 S. Michigan Ave, Chicago, Illinois, USA"
  },
  "name": "Shostakovich Leningrad",
  "offers": {
    "@type": "Offer",
    "url": "/examples/ticket/12341234",
    "price": "40",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  },
  "performer": [
    {
      "@type": "MusicGroup",
      "name": "Chicago Symphony Orchestra",
      "sameAs": [
        "http://cso.org/",
        "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
      ]
    },
    {
      "@type": "Person",
      "image": "/examples/jvanzweden_s.jpg",
      "name": "Jaap van Zweden",
      "sameAs": "http://www.jaapvanzweden.com/"
    }
  ],
  "startDate": "2014-05-23T20:00",
  "workPerformed": [
    {
      "@type": "CreativeWork",
      "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes",
      "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes"
    },
    {
      "@type": "CreativeWork",
      "name": "Shostakovich Symphony No. 7 (Leningrad)",
      "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
    }
  ]
}

For real events as scrapped by my scripts offers tends to be an array, and this illustrates a common snag of getting an object when I want an array. I wrote this function to transform a given object to an array.

This function moves objects such as “offers” into an array:

# object2array myarr '"offers"'
function object2array {
  declare -n arr
  arr=$1
  local rows
  local cols
  local key
  readarray -t rows <<< "$(getkeys "$1" "$2")"
  for key in "${rows[@]}"; do
    readarray -d ',' -t cols <<< "$key"
    if [[ ${cols[1]} =~ ^[0-9]+$ ]]; then
      echo "Error: $2 is already an array" >&2
      return 1
    fi
    arr["${cols[0]},0,${cols[@]:1}"]="${arr["$key"]}"
    unset arr["$key"]
  done
}

The unset arr["$key"] breaks shellcheck’s rule SC2184, but changing the quotes causes shellspec to fail.

#!/bin/bash

source /usr/local/lib/json-utils.sh

declare -A myarr
json2array myarr "$(< ../musicevent.json)"

ExampleGroup 'place an object in an array'

  Example 'move offers into an array'
    object2array myarr '"offers"'
    When call array2json myarr
    The output should equal '{
  "@context": "https://schema.org",
  "@type": "MusicEvent",
  "location": {
    "@type": "MusicVenue",
    "address": "220 S. Michigan Ave, Chicago, Illinois, USA",
    "name": "Chicago Symphony Center"
  },
  "name": "Shostakovich Leningrad",
  "offers": [
    {
      "@type": "Offer",
      "availability": "https://schema.org/InStock",
      "price": "40",
      "priceCurrency": "USD",
      "url": "/examples/ticket/12341234"
    }
  ],
  "performer": [
    {
      "@type": "MusicGroup",
      "name": "Chicago Symphony Orchestra",
      "sameAs": [
        "http://cso.org/",
        "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
      ]
    },
    {
      "@type": "Person",
      "image": "/examples/jvanzweden_s.jpg",
      "name": "Jaap van Zweden",
      "sameAs": "http://www.jaapvanzweden.com/"
    }
  ],
  "startDate": "2014-05-23T20:00",
  "workPerformed": [
    {
      "@type": "CreativeWork",
      "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes",
      "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes"
    },
    {
      "@type": "CreativeWork",
      "name": "Shostakovich Symphony No. 7 (Leningrad)",
      "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
    }
  ]
}'
    The status should be success
    The error should be blank
  End

  Example 'object2array on an array should throw an error'
    When call object2array myarr '"workPerformed"'
    The stderr should equal 'Error: "workPerformed" is already an array'
    The status should be failure
  End

End