Frontier Software

IBM “THINK” Sign

By Robert Laing

This website in its current form represents years of struggling with literate programming.

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. — Donald Knuth.

Many decades ago, I used Knuth’s TeX typesetting system, or rather Leslie Lamport’s LaTeX derivation, to publish a book for a friend. It’s somewhat ironic that Knuth called his software documentation system (for Pascal, the programing language taught in a computer science course I did way back in the last century) Web, since it’s not web as in WWW friendly. Knuth’s software produces beautifully typeset pages for printing on paper, usually in pdf, whereas I want hypertext.

I’ve dabbled with a lot of web “Content Mangement Systems” over the years both in my job as a newspaper journalist and for my hobby events listing sites joeblog.co.za, seatavern.co.za, and many more, and have settled on Hugo, mainly because of its support for syntax highlighting.

Online learning is something that really excited me when it started, and I did some fantastic courses. One by Gregor Kiczales based on a free online textbook How to Design Programs really stressed the importance of test-driven development. The book’s design recipe is fairly similar to Agile, but places emphasis on documenting before you code.

The Agile cult seems to be anti-documentation, at least that’s the impression I got from a supposed guru in a youtube lecture saying: “People don’t read comments. Computers don’t read comments. So why bother with them?”

Comments and documentation have become intertwined through systems such as JavaScript’s JSDoc. Personally, I want the code embedded in the documentation, not vice-versa, which Hugo lets me do. As someone who jumps between programing languages a lot, I also don’t like having to learn a different documentation system for each language.

Testing to learn

The first thing I look for in documentation is examples. The web is a rich resource of these, and I’ve become a huge fan of a specialist search engine phind.com which trawls stackoverflow etc and somehow seems to find the good stuff.

To help remember what I’ve figured out so far, I’m keeping my shellspec “specification” files in the bash section of my Hugo documentation tree to get rendered on this website.

Testing as documentation

Writing is hard, and learning to write takes practice. No simple rules can ensure that you write good specs. One thing to avoid is using code. Code is a bad medium for helping to understand code. Architects don’t make their blueprints out of bricks. — Wired interview with Leslie Lamport

My theory why Unix took over the world is it what slipped onto the budget of a contract for the US patent office who needed a documentation syste. A side product was “man pages”, ie the system came documented, albeit very tersely.

The manual was intended for reference, not learning. Their gazillian options are listed alphabetically, not by how commonly they are used or importance. But they ultimately got us to the web.

As I’m struggling with now, all these “power tools” are not as simple as would be nice, but as a touch typist, what would take days with a GUI takes milliseconds with a CLI if you know how.

Testing as specification

Programmers who advocate writing tests before writing code often believe those tests can serve as a specification. Writing tests does force us to think, and anything that gets us to think before coding is helpful. However, writing tests in code does not get us thinking above the code level. Who Builds a House Without Drawing Blueprints?, an article written by Leslie Lamport

Behavior-driven development systems such as Jasmine for JavaScript and ShellSpec for Bash prefer the jargon specification to tests. Personally, I use a subdirectory called tests rather than specs for my Jasmine and ShellSpec scripts.

unit

[Unit]

This is common to all unit types. It contains metadata about the service such as a description.

systemd.unit(5)

/usr/lib/systemd/system/nginx.service

[Unit]
Description=nginx web server
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

/usr/lib/systemd/system/postgresql.service

[Unit]
Description=PostgreSQL database server
Documentation=man:postgres(1)
After=network.target network-online.target
Wants=network-online.target

/usr/lib/systemd/system/tmp.mount

[Unit]
Description=Temporary Directory /tmp
Documentation=https://systemd.io/TEMPORARY_DIRECTORIES
Documentation=man:file-hierarchy(7)
Documentation=https://systemd.io/API_FILE_SYSTEMS
ConditionPathIsSymbolicLink=!/tmp
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

grep

grep me no patterns and I’ll tell you no lines. — fortune cooky

manual

wikibooks

As explained in The Unix Programing Environment by Brian Kernighan and Rob Pike, sed can do just about everything grep can.

sed -n '/pattern/p' files

is the same as

grep -h pattern files

Why do we have both sed and grep? After all, grep is just a simple special case of sed. Part of the reason is history — grep came well before sed. But grep survives, indeed thrives, because for the job that they both do, it is significantly easier to use than sed is.

appending

Following from iteration, the next step is to add JSON objects to the version converted to bash.

#!/bin/bash

source /usr/local/lib/json-utils.sh

ExampleGroup 'append new JSON object to an array'

  Example 'add another "workPerformed" object'
    declare -A myarr
    json2array myarr "$(< ../musicevent.json)"
    json1='{
      "@type": "CreativeWork",
      "name": "Sinfonia da Requiem",
      "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem"
    }'
    array_append myarr '"workPerformed"' "$json1"
    When call array2json myarr
    The output should equal '{
  "@context": "https://schema.org",
  "@type": "MusicEvent",
  "location": {
    "@type": "MusicVenue",
    "address": "220 S. Michigan Ave, Chicago, Illinois, USA",
    "name": "Chicago Symphony Center"
  },
  "name": "Shostakovich Leningrad",
  "offers": {
    "@type": "Offer",
    "availability": "https://schema.org/InStock",
    "price": "40",
    "priceCurrency": "USD",
    "url": "/examples/ticket/12341234"
  },
  "performer": [
    {
      "@type": "MusicGroup",
      "name": "Chicago Symphony Orchestra",
      "sameAs": [
        "http://cso.org/",
        "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
      ]
    },
    {
      "@type": "Person",
      "image": "/examples/jvanzweden_s.jpg",
      "name": "Jaap van Zweden",
      "sameAs": "http://www.jaapvanzweden.com/"
    }
  ],
  "startDate": "2014-05-23T20:00",
  "workPerformed": [
    {
      "@type": "CreativeWork",
      "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes",
      "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes"
    },
    {
      "@type": "CreativeWork",
      "name": "Shostakovich Symphony No. 7 (Leningrad)",
      "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
    },
    {
      "@type": "CreativeWork",
      "name": "Sinfonia da Requiem",
      "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem"
    }
  ]
}'
    The status should be success
    The error should be blank
  End


End

Style Guides

Block, Element, Modifier

github

Tutorials

/* Block component */
.btn {}

/* Element that depends upon the block */ 
.btn__price {}

/* Modifier that changes the style of the block */
.btn--orange {} 
.btn--big {}

https://css-tricks.com/bem-101/

https://google.github.io/styleguide/htmlcssguide.html

https://blog.logrocket.com/5-things-to-consider-when-creating-your-css-style-guide-7b85fa70039d/

https://cssguidelin.es/

Templates

Not

By Robert Laing

Lets turn the query “Which students applied for ‘CS’?” around, again using Stanford University dean Jennifer Widom’s basic SQL examples using her college applications data.

Which students have not applied for CS?

A trap many people will fall into is to rewrite the σMajor = ‘CS’(Apply) selection to not equal, editing it into something like σMajor != ‘CS’(Apply).

That would return the set for not ‘CS’ as {123, 234, 345, 678, 765, 876}. Recalling that the students who applied for ‘CS’ were {123, 345, 543, 876, 987}, we see we shouldn’t have {123, 345, 876} in our result. The reason those students match Major <> 'CS' is student 123 also applied for ‘EE’, 345 applied for ‘EE’ and ‘bioengineering’ besides ‘CS’, while 876 applied for ‘biology’ and ‘marine biology’ besides ‘CS’.

Or

By Robert Laing

Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration, modifying the same SQL examples done in conjunction as “which students applied for ‘CS’ and ‘EE’?”.

Which students applied for ‘CS’ or ‘EE’?

As with conjunction’s p ∧ q being a spiky version of set theory’s P ∩ Q, disjunction’s p ∨ q is a spiky version of set theory’s P ∪ Q.

And

By Robert Laing

Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration.

Which students applied for ‘CS’ and ‘EE’?

In what I’ve called copula notation, conjunction is written p ∧ q, which is a handy mnemonic that its set equivalent is P ∩ Q. That logic has a spiky version of set theory’s rounded symbol we’ll also encounter in disjunction and implication.

service

[Service]

systemd.service(5)

/usr/lib/systemd/system/nginx.service

[Service]
Type=forking
PIDFile=/run/nginx.pid
PrivateDevices=yes
PrivateTmp=true
SyslogLevel=err

ExecStart=/usr/bin/nginx
ExecReload=/usr/bin/nginx -s reload
Restart=on-failure
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5

/usr/lib/systemd/system/postgresql.service

[Service]
Type=notify
TimeoutSec=120
User=postgres
Group=postgres

Environment=PGROOT=/var/lib/postgres

SyslogIdentifier=postgres
PIDFile=/var/lib/postgres/data/postmaster.pid
RuntimeDirectory=postgresql
RuntimeDirectoryMode=755

ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data
ExecStart=/usr/bin/postgres -D ${PGROOT}/data
ExecReload=/bin/kill -HUP ${MAINPID}
KillMode=mixed
KillSignal=SIGINT

# Due to PostgreSQL's use of shared memory, OOM killer is often overzealous in
# killing Postgres, so adjust it downward
OOMScoreAdjust=-200

# Additional security-related features
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
NoNewPrivileges=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
PrivateDevices=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native

systemd.exec(5)

sed

Manual

Four types of sed scripts

1. Multiple Edits to the Same File

2. Making Changes Across a Set of Files

3. Extracting Contents of a File

4. Edits To Go

Substitute [address]s/regexp/replacement/[flags]

The ‘s’ command (as in substitute) is probably the most important in ‘sed’ and has a lot of different options. The syntax of the ‘s’ command is ’s/REGEXP/REPLACEMENT/FLAGS'.

The ‘/’ characters may be uniformly replaced by any other single character within any given ‘s’ command.

Difference

By Robert Laing

Lets start by repeating the example in not, however using SQL’s EXCEPT operator, which ties into set theory’s P - Q.

In relational algebra, the query we want is ΠsID(Student) - ΠsIDMajor = ‘CS’(Apply)).

Note that relational algebra involving different tables/sets requires them to be projected into a shared type of compound data structure. Here I’m keeping things simple by only working with their common column sID. In fill values I’ll go into the complexities that negation sometimes causes because of missing columns.

Union

Intersection

By Robert Laing

Which majors intersect?

Lets return to Venn’s 5 relations. Here we want to check A ∩ B ≠ Ø, which is true for any of the first four relations.

venn-relations2.svg

We need a self-join for this query, which is a form of intersection.

SELECT DISTINCT a1.major AS A, a2.major AS B
FROM apply as a1, apply as a2
WHERE a1.sid = a2.sid
ORDER BY A, B;
       a        |       b        
----------------+----------------
 bioengineering | bioengineering
 bioengineering | CS
 bioengineering | EE
 biology        | biology
 biology        | CS
 biology        | marine biology
 CS             | bioengineering
 CS             | biology
 CS             | CS
 CS             | EE
 CS             | marine biology
 EE             | bioengineering
 EE             | CS
 EE             | EE
 history        | history
 history        | psychology
 marine biology | biology
 marine biology | CS
 marine biology | marine biology
 psychology     | history
 psychology     | psychology
(21 rows)

Note everything intersects with itself. To remove all the A ≡ A along with A ∩ B so B ∩ A reciprocal intersections, we could add a a1.major < a2.major test:

Fill Values

By Robert Laing

The problem of how to deal with unknown values for logical variables led to the development of three-valued logic, which SQL implemented with NULL, and Prolog with unset variables.

Which students have not made any college applications yet?

In preparation of outer joins, lets here do the portion called an antijoin

Student ▷ Apply

Our basic goal here is to find the set {456, 567, 654, 789}.

college-classes.svg

In the example so far involving both the Student and Apply tables — “Which students have not applied for CS?” — I sidestepped the problem that the student table has columns sID, sName, GPA and sizeHS while Apply has columns sID, cName, major and decision by projecting both to their single common column, SID.

Outer Join

By Robert Laing

As explained in inner join relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.

P - Q P ∩ Q Q - P P Q

The P ∩ Q portion could be considered inner join P ⋈ Q. A left outer join P ⟕ Q is (P - Q) ∪ (P ∩ Q), a right outer join P ⟖ Q is (P ∩ Q) ∪ (Q - P), and a full outer join P ⟗ Q is (P - Q) ∪ (P ∩ Q) ∪ (Q - P).

Inner Join

By Robert Laing

Relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.

P - Q P ∩ Q Q - P P Q

The above illustrates what relational algebra calls a full outer join which combines three parts, the inner or natural join P ⋈ Q which equates to P ∩ Q, and what this section covers, flanked by left and right antijoins P ▷ Q which equats to P - Q, and Q ▷ P which equats to Q - P.

De Morgan's Law

By Robert Laing

What I took to calling copula notation because I read somewhere that the ∧ and ∨ symbols are called copulas (but I can’t find the reference again) is a handy mnemonic for remembering two substitution rules commonly called De Morgan’s Laws:

  1. ¬(p ∧ q) can be substituted with ¬p ∨ ¬q
  2. ¬(p ∨ q) can be substituted with ¬p ∧ ¬q

There’s an interesting relationship with set theory in that

Sum

By Robert Laing

In 1863 William Stanley Jevons wrote to George Boole that surely Boole’s operation of addition should be replaced by the more natural ‘inclusive or’ (or ‘union’), leading to the law X+X=X. Boole completely rejected this suggestion (it would have destroyed his system based on ordinary algebra) and broke off the correspondence.The Algebra of Logic Tradition
pqp + q
111
101
011
000

As can be seen from the above quote, that 1 + 1 = 1 in logic arithmetic caused rancour between the field’s founding fathers, and does to this day. Jevons used a rotated ÷ symbol, ⋅∣⋅, for or instead of + to sidestep the problem.

Product

By Robert Laing

2 Value Algebra

Since conjunction is closely associated with the word and, it jarred me a bit to discover that it is logic’s equivalent of multiplication, while disjunction — commonly thought of as or — is logic’s addition, the arithmetic operator I associate with and.

Why conjunction equates to multiplication is best illustrated by its truth table:

pqp · q
111
100
010
000

Moving from binary to any number of propositions, the universal quantification symbol ∀(p) tends to be used, as in

cycles-with-tabling

By adding arc(k, e). we alter the previous example in Transitive Closures to this:

tree_cycle.svg

Unless guarded against, this will cause tc(a, X). to never escape looping between k and e. Thanks to a relatively recent addition to SWI Prolog Tabled execution, all that’s needed to avoid this is adding one line of code prolog:- table tc/2. to that previously shown in TransitiveClosures.

:- table tc/2.

arc(a, b).
arc(a, c).
arc(a, d).
arc(b, e).
arc(b, f).
arc(c, g).
arc(c, h).
arc(c, i).
arc(d, j).
arc(e, k).
arc(f, l).
arc(f, m).
arc(h, n).
arc(i, o).
arc(i, p).
arc(j, q).
arc(j, r).
arc(j, s).
arc(k, e).
arc(m, t).

tc(X, Y) :-
    arc(X, Y).

tc(X, Z) :-
    arc(X, Y),
    tc(Y, Z).

Tabling changes the order of traversal returned by tc(a, X) to this:

Parallel

Series

By Robert Laing

Flow Control

Yet another way to think of logic is as electrical switches, where on is true and off is false. Here conjunction is switches in series, and disjunction is switches in parallel.

p q r s

If any switch in series is not on (such as switch r in the above diagram), it breaks the flow of electric current, meaning a light or whatever the circuit powers would be off. If the switches were in parallel, (which I’ll illustrate in disjunction), any on switch would put the circuit on.

iterative-deepening

By Robert Laing

A snag we hit in puzzle solving is the graph isn’t handily prestored in a database for route finding using transitive closures with infinite loop protection thanks to cycles with tabling. We have to turn to what’s sometimes called generative recursion to dynamically build our graph as we explore it. For this, we’ll return to the techniques introduced in BreadthDepthFirst.

We’ll be using iterative deepening depth-first search which gets depth first to mimic breadth first by doing it repeatedly, lengthening its depth-limit one step at a time. The reason is unbounded depth first instead of finding the shortest route, finds the leftmost route. Breadth first does find the shortest route, but even the cheap RAM available on modern computers is often insufficient for it to complete this task.

filter

FILTER can be combined with an aggregate expression.

SELECT city, count(*) FILTER (WHERE temp_lo < 45), max(temp_lo)
FROM weather
GROUP BY city;
SELECT count(*) AS unfiltered, count(*) FILTER (WHERE i < 5) AS filtered
FROM generate_series(1,10) AS s(i);

FILTER is much like WHERE, except that it removes rows only from the input of the particular aggregate function that it is attached to. Here, the count aggregate counts only rows with temp_lo below 45; but the max aggregate is still applied to all rows, so it still finds the reading of 46. — Aggregate Functions

game

Game

{
  "@context": "https://schema.org",
  "@type": "Game",
  "offers":{
    "@type":"Offer",
    "priceCurrency":"$",
    "price":"17.99",
    "availableAtOrFrom":"/monopoly-2/en_US/shop/where-to-buy.cfm?brand_guid=DAD28866-1C43-11DD-BD0B-0800200C9A66&prodName=Monopoly%20Game"
  },
  "audience":{
    "@type":"PeopleAudience",
    "suggestedMinAge":"8"
  },
  "description":"Own it all as a high-flying trader in the fast-paced world of real estate. Tour the city for the hottest properties: sites, stations and utilities are all up for grabs. Invest in houses and hotels, then watch the rent come pouring in! Make deals with other players and look out for bargains at auction. There are many ways to get what you want. For really speedy dealers, use the speed die for a quick and intense game of Monopoly. So get on Go and trade your way to success!",
  "gameItem":["gameboard","8 tokens","Title Deed cards","16 Chance cards", "16 Community Chest cards","money pack","32 houses","12 hotels"],
  "numberOfPlayers":{
    "@type":"QuantitativeValue",
    "minValue":"3",
    "maxValue":"5"
  },
  "copyrightHolder":"Hasbro"
}

set

explain

show

This is a synomym for SELECT current_setting ( setting_name text [, missing_ok boolean ] ).

SHOW

SHOW log_destination;

SHOW logging_collector;

read-write-conflicts

Understanding locks

7 tips for dealing with locks

Something I learnt the hard way by getting the computer to play games over-and-over and then update the average scores each time was it crashed my Postgresql functions with

ERROR:  relation "_states" does not exist
LINE 1: SELECT id FROM _states WHERE state = NULL

The problem was garbage values were getting returned from queries if their table was being updated, and the solution was:

shebang

wikipedia

The first line of shell scripts is most commonly written without a space, ie #!/usr/bash however in the examples of O’Reilly’s sed & awk book it’s written #! /bin/sh

According to this stackoverflow post

The shebang (#!) is a 16-bit kernel-interpreted magic “number” (actually 0x23 0x21). Thereafter, the kernel program loader (following the exec-family call) attempts to execute the remainder of the line to handle the remainder of the containing file’s content. Many, if not most modern kernels disregard preceding spaces to the command that follows the shebang. As such, #! /bin/bash, #! /usr/bin/perl, and the like, should be acceptable with most modern kernels. That said, a shebang with no following space is far more commonly seen.

microdata

deletion

#!/bin/bash

source /usr/local/lib/json-utils.sh

ExampleGroup 'delete objects or array elements'

  Example 'delete "location"'
    declare -A myarr
    json2array myarr "$(< ../musicevent.json)"
    del_key myarr '"location"'
    When call array2json myarr
    The output should equal '{
  "@context": "https://schema.org",
  "@type": "MusicEvent",
  "name": "Shostakovich Leningrad",
  "offers": {
    "@type": "Offer",
    "availability": "https://schema.org/InStock",
    "price": "40",
    "priceCurrency": "USD",
    "url": "/examples/ticket/12341234"
  },
  "performer": [
    {
      "@type": "MusicGroup",
      "name": "Chicago Symphony Orchestra",
      "sameAs": [
        "http://cso.org/",
        "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
      ]
    },
    {
      "@type": "Person",
      "image": "/examples/jvanzweden_s.jpg",
      "name": "Jaap van Zweden",
      "sameAs": "http://www.jaapvanzweden.com/"
    }
  ],
  "startDate": "2014-05-23T20:00",
  "workPerformed": [
    {
      "@type": "CreativeWork",
      "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes",
      "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes"
    },
    {
      "@type": "CreativeWork",
      "name": "Shostakovich Symphony No. 7 (Leningrad)",
      "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
    }
  ]
}'
    The status should be success
    The error should be blank
  End

  Example 'delete "performer",1,'
    declare -A myarr
    json2array myarr "$(< ../musicevent.json)"
    del_key myarr '"performer",1,'
    When call array2json myarr
    The output should equal '{
  "@context": "https://schema.org",
  "@type": "MusicEvent",
  "location": {
    "@type": "MusicVenue",
    "address": "220 S. Michigan Ave, Chicago, Illinois, USA",
    "name": "Chicago Symphony Center"
  },
  "name": "Shostakovich Leningrad",
  "offers": {
    "@type": "Offer",
    "availability": "https://schema.org/InStock",
    "price": "40",
    "priceCurrency": "USD",
    "url": "/examples/ticket/12341234"
  },
  "performer": [
    {
      "@type": "MusicGroup",
      "name": "Chicago Symphony Orchestra",
      "sameAs": [
        "http://cso.org/",
        "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra"
      ]
    }
  ],
  "startDate": "2014-05-23T20:00",
  "workPerformed": [
    {
      "@type": "CreativeWork",
      "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes",
      "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes"
    },
    {
      "@type": "CreativeWork",
      "name": "Shostakovich Symphony No. 7 (Leningrad)",
      "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)"
    }
  ]
}'
    The status should be success
    The error should be blank
  End

End

defaults

Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries.

ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

  Example 'Basic form ${parameter}'
    foo="bar"
    When call echo "${foo}" # brackets are unecessary in this example
    The output should eq "bar"
  End

  Example 'default value of unset variable is empty string'
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'default value is substituted with ${parameter:-default}'
    When call echo "${foo:-default value}"
    The output should eq "default value"
  End

  Example 'default value is substituted with ${parameter:-default}'
    foo="bar"
    When call echo "${foo:-default value}"
    The output should eq "bar"
  End

  Example ':- doesn`t alter the variable`s value'
    echo "${foo:-default value}"
    When call echo "${foo}"
    The output should eq ""
  End

  Example 'Whereas := does'
    echo "${foo:=default value}"
    When call echo "${foo}"
    The output should eq "default value"
  End


End

slicing

Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation.

The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is:

${parameter:offset}
${parameter:offset:length}

String slices

Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart.

Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript. Negative lengths are actually offsets from the end.

transforming

Lowercasing

${varname@L}

#!/bin/bash

# find_schema 'artists' 'Arno Carstens'
# Returns just the first match
function find_schema {
  local path
  local file
  path="${2@L}"
  path="*${path//[^[:alnum:]]/\*}*"
  path="/usr/local/webapps2/*/content/${1}/${path}/schema.json"
  for file in $path; do
    if [[ -f $file ]]; then
      echo "$file"
      return 0
    fi
  done
}

ExampleGroup 'list artist and venue schema.json files'

  Example 'Find schema.json for "Arno Carstens"'
    When call find_schema 'artists' 'Arno Carstens'
    The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json"
  End

  Example 'Find schema.json for "No File Created"'
    When call find_schema 'artists' 'No File Created'
    The output should eq ''
  End

  Example 'Find schema.json for "Black Cat Bones"'
    When call find_schema 'artists' 'Black Cat Bones'
    The output should eq '/usr/local/webapps2/joeblog/content/artists/blackcatbones/schema.json'
  End

Example 'Find schema.json for "Bailey`s"'
    When call find_schema 'venues' "Bailey's"
    The output should eq '/usr/local/webapps2/joeblog/content/venues/baileys/schema.json'
  End
  

End
ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html'
# also https://opensource.com/article/17/6/bash-parameter-expansion

# https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator
  Example '${parameter@operator} U'
    foo="bar"
    When call echo "${foo@U}"
    The output should eq "BAR"
  End

  Example '${parameter@operator} u'
    foo="bar"
    When call echo "${foo@u}"
    The output should eq "Bar"
  End

  Example '${parameter@operator} L'
    foo="BAR"
    When call echo "${foo@L}"
    The output should eq "bar"
  End

  Example 'associative array JSON path'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo}"
    The output should eq "performer,0,sameAs,0"
  End

  # Output to be reused as input of another command
  Example '${parameter@operator} Q'
    foo=""performer",0,"sameAs",0"
    When call echo "${foo@Q}"
    The output should eq "'performer,0,sameAs,0'"
  End

  Example '${parameter@operator} a'
    declare -A arr=(["foo"]="bar")
    When call echo "${arr@a}"
    The output should eq "A"
  End

  Example '${parameter@operator} K'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@K}"
    The output should eq "performer,0,sameAs,0 \"bar\" "
  End

  Example '${parameter@operator} k'
    declare -A arr=([""performer",0,"sameAs",0"]="bar")
    When call echo "${arr[@]@k}"
    The output should eq "performer,0,sameAs,0 bar"
  End

End

indexed-arrays

Something I’ve found a lot harder than it should be is converting JSON arrays to bash “indexed” (as in not associative) arrays.

JSON arrays are comma delimited, similar to csv but with surrounding square brackets. There are many ways to do this. The old way involves temporarily resetting the internal field separator environment variable IFS from its default space, TAB, NEWLINE to a comma, which shellcheck and others frown upon.

The modern way involves using readarray (aka mapfile, the two are synonymous).

iteration

Iterating over JSON arrays via Bash associative-arrays requires two more functions to the json-utils to json2array and array2json in the top section.

A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays.

function getkeys {
  local -n arr
  arr=$1
  local keys
  keys=$(for key in "${!arr[@]}"; do
    if [[ $key =~ $2 ]]; then
      echo "$key"
    fi
  done | sort -u)
  if [[ -z $keys ]]; then 
    echo "Error: $2 is not a valid key" >&2
    return 1;
  fi
  echo "$keys"
}

The return 1 part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.

wolf-goat-cabbage

This was my first attempt at illustrating graph traversal using an animated pgn. Instructions on how I did this below.

The animation is made out of 8 png files generated by graphviz’s dot as in dot -T png -o frame7.png frame7.dot. These then were combined into an animated png file using apngasm.

apngasm frame0.png frame1.png frame2.png frame3.png frame4.png frame5.png frame6.png frame7.png -d 1000 -o cgw.png

My main lesson from creating this diagram was utility values have direction, ie the utility of a node needs to be stored in the graph table with the parent and move, not just with the node itself. This is because the graph is full of cycles, and to avoid the automaton going back, the utility of going back to a node needs to be less than going forward.

buttons-and-lights

This is a simple example puzzle translated from a kif file into SWI Prolog. The full code is on Swish where the query route(Actions) gives two answers

  • Actions = [does(robot, a), does(robot, b), does(robot, c), does(robot, a), does(robot, b), does(robot, a)]
  • Actions = [does(robot, a), does(robot, b), does(robot, a), does(robot, c), does(robot, b), does(robot, a)]

The puzzle consists of three lights labeled p, q, and r, each of which can be on or off, giving us 8 possible states (provided we ignore the state’s step counter). The player moves from state to state by pressing one of three buttons, labelled a, b, or c and the object is to get to the final state with 6 button presses.

trimming

The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers.

As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples).

We need Bash’s composite patterns which are only available if

shopt -s extglob

is set. Composite patterns allow +(pattern-list) as used below to match possibly more than one space.

menus

Directory traversal

I’ve developed my own homegrown way of recursively walking a Hugo site’s content tree as described in this discourse thread since the official way of putting the menu in the site config file isn’t what I want as I add and remove pages.

With more experience, I changed the deprecated template with an inline partial

My code for layouts/partials/menu.html looks like this:

<nav>
{{ partial "inline/walk.html" (dict "dir" .Site.Home.Pages) }}

{{ define "partials/inline/walk.html" }}
  <ol>
  {{ range .dir }}
    {{ if (eq .BundleType "branch") }}
        <li><a href="{{ .RelPermalink }}">{{ .Title }}</a>
        {{ partial "inline/walk.html" (dict "dir" .Pages) }}
        </li>
    {{ else }}
      <li><a href="{{ .RelPermalink }}">{{ .Title }}</a></li>
    {{ end }}
  {{ end }}
  </ol>
{{ end }}
</nav>

Section menu

Hugo’s default menu system requires everything to be manually entered into the system configuration file.

date

The primary reason I’m using bash rather than some programing language is to get easy access to GNU’s powerful date program. It automagically reads most text descriptions of dates and can then translate them to what I want, which is ISO 8601 as specified by schema.org.

Getting dates right has been a constant source of bugs — it took me a while to figure that because my development machine is set to my local time, “Africa/Johannesburg” and my server to “UTC”, joeblog.co.za was showing all events 2 hours early.

tree

man page

Before I discovered the tree command, I wasted huge amounts of time manually making grapic directory listings.

$ tree ~/webapps/frontiersoftware
/home/roblaing/webapps/frontiersoftware
├── archetypes
│   └── default.md
├── assets
├── config
│   └── _default
│       └── hugo.json
├── content
│   ├── bash
│   │   ├── coreutils
│   │   │   ├── _index.md
│   │   │   └── sed
│   │   │       ├── index.md
│   │   │       └── spec
│   │   │           ├── basics_spec.sh
│   │   │           ├── deleting_lines_spec.sh
│   │   │           ├── inserting_lines_spec.sh
│   │   │           ├── printing_spec.sh
│   │   │           ├── spec_helper.sh
│   │   │           └── substitution_spec.sh
│   │   ├── extra
│   │   │   ├── _index.md
│   │   │   └── tree
│   │   │       └── index.md
│   │   ├── _index.md
│   │   └── util-linux
│   │       └── _index.md
│   ├── hugo
│   └── _index.md
├── data
├── i18n
├── layouts
│   ├── _default
│   │   ├── baseof.html
│   │   ├── list.html
│   │   └── single.html
│   └── shortcodes
│       └── insert-code.html
├── static
└── themes

21 directories, 19 files

Proofs

The contrapositive law

(p ⇒ q) ⇔ (¬q ⇒ ¬p)

Proof by Contradiction

(¬p ⇒ 0) ⇔ p

Equivalence to Truth

(p ≡ 1) ≡ p

Deduction

(E1 AND E2 AND · · · AND Ek) ⇒ E

Modus ponens

(p AND (p ⇒ q)) ⇒ q

Resolution

((p + q)(¬p + r)) → (q + r)