Frontier Software

By Robert Laing

This website in its current form represents years of struggling with literate programming.

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. — Donald Knuth.

Many decades ago, I used Knuth’s TeX typesetting system, or rather Leslie Lamport’s LaTeX derivation, to publish a book for a friend. It’s somewhat ironic that Knuth called his software documentation system (for Pascal, the programing language taught in a computer science course I did way back in the last century) Web, since it’s not web as in WWW friendly. Knuth’s software produces beautifully typeset pages for printing on paper, usually in pdf, whereas I want hypertext.

I’ve dabbled with a lot of web “Content Mangement Systems” over the years both in my job as a newspaper journalist and for my hobby events listing sites joeblog.co.za, seatavern.co.za, and many more, and have settled on Hugo, mainly because of its support for syntax highlighting.

Online learning is something that really excited me when it started, and I did some fantastic courses. One by Gregor Kiczales based on a free online textbook How to Design Programs really stressed the importance of test-driven development. The book’s design recipe is fairly similar to Agile, but places emphasis on documenting before you code.

The Agile cult seems to be anti-documentation, at least that’s the impression I got from a supposed guru in a youtube lecture saying: “People don’t read comments. Computers don’t read comments. So why bother with them?”

Comments and documentation have become intertwined through systems such as JavaScript’s JSDoc. Personally, I want the code embedded in the documentation, not vice-versa, which Hugo lets me do. As someone who jumps between programing languages a lot, I also don’t like having to learn a different documentation system for each language.

Testing to learn

The first thing I look for in documentation is examples. The web is a rich resource of these, and I’ve become a huge fan of a specialist search engine phind.com which trawls stackoverflow etc and somehow seems to find the good stuff.

To help remember what I’ve figured out so far, I’m keeping my shellspec “specification” files in the bash section of my Hugo documentation tree to get rendered on this website.

Testing as documentation

Writing is hard, and learning to write takes practice. No simple rules can ensure that you write good specs. One thing to avoid is using code. Code is a bad medium for helping to understand code. Architects don’t make their blueprints out of bricks. — Wired interview with Leslie Lamport

There was a great quote from Lamport I read online but can no longer find (again a reminder why it’s important to to have an easy to use online notetaking system) in which he argued tests are not specifications. I’m not sure what Lamport’s views on tests as documentation would be, but probably that they’re better than nothing.

My theory why Unix took over the world is it what slipped onto the budget of a contract for the US patent office who needed a documentation syste. A side product was “man pages”, ie the system came documented, albeit very tersely.

The manual was intended for reference, not learning. Their gazillian options are listed alphabetically, not by how commonly they are used or importance. But they ultimately got us to the web.

As I’m struggling with now, all these “power tools” are not as simple as would be nice, but as a touch typist, what would take days with a GUI takes milliseconds with a CLI if you know how.

appending

Following from iteration, the next step is to add JSON objects to the version converted to bash. #!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'append new JSON object to an array' Example 'add another "workPerformed" object' declare -A myarr json2array myarr "$(< ../musicevent.json)" json1='{ "@type": "CreativeWork", "name": "Sinfonia da Requiem", "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem" }' array_append myarr '"workPerformed"' "$json1" When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "location": { "@type": "MusicVenue", "address": "220 S.

Color

The arithmetic of colours Colour Wheel Munsell color system hsl(H, S, L[, A]) H (hue) is an angle (degrees by default if no units are given). Red is 0 or 360, green 120, and blue 240. 0° Red 30° Orange 180° Cyan S (saturation) ranges from 100% for fully the hue to 0% for gray. L (lightness) is 50% for “normal”. 100% is black and 0% is white. A (alpha) can be a number between 0 and 1, or a percentage, where the number 1 corresponds to 100% (full opacity).

Style Guides

Block, Element, Modifier github Tutorials /* Block component */ .btn {} /* Element that depends upon the block */ .btn__price {} /* Modifier that changes the style of the block */ .btn--orange {} .btn--big {} https://css-tricks.com/bem-101/ https://google.github.io/styleguide/htmlcssguide.html https://blog.logrocket.com/5-things-to-consider-when-creating-your-css-style-guide-7b85fa70039d/ https://cssguidelin.es/ Templates

Not

By Robert Laing Lets turn the query “Which students applied for ‘CS’?” around, again using Stanford University dean Jennifer Widom’s basic SQL examples using her college applications data. Which students have not applied for CS? A trap many people will fall into is to rewrite the σMajor = ‘CS’(Apply) selection to not equal, editing it into something like σMajor != ‘CS’(Apply). That would return the set for not ‘CS’ as {123, 234, 345, 678, 765, 876}.

Or

By Robert Laing Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration, modifying the same SQL examples done in conjunction as “which students applied for ‘CS’ and ‘EE’?”. Which students applied for ‘CS’ or ‘EE’? As with conjunction’s p ∧ q being a spiky version of set theory’s P ∩ Q, disjunction’s p ∨ q is a spiky version of set theory’s P ∪ Q.

And

By Robert Laing Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration. Which students applied for ‘CS’ and ‘EE’? In what I’ve called copula notation, conjunction is written p ∧ q, which is a handy mnemonic that its set equivalent is P ∩ Q. That logic has a spiky version of set theory’s rounded symbol we’ll also encounter in disjunction and implication.

Difference

By Robert Laing Lets start by repeating the example in not, however using SQL’s EXCEPT operator, which ties into set theory’s P - Q. In relational algebra, the query we want is ΠsID(Student) - ΠsID(σMajor = ‘CS’(Apply)). Note that relational algebra involving different tables/sets requires them to be projected into a shared type of compound data structure. Here I’m keeping things simple by only working with their common column sID. In fill values I’ll go into the complexities that negation sometimes causes because of missing columns.

Union

Intersection

Fill Values

By Robert Laing The problem of how to deal with unknown values for logical variables led to the development of three-valued logic, which SQL implemented with NULL, and Prolog with unset variables. Which students have not made any college applications yet? In preparation of outer joins, lets here do the portion called an antijoin Student ▷ Apply Our basic goal here is to find the set {456, 567, 654, 789}.

Outer Join

By Robert Laing As explained in inner join relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram. P - Q P ∩ Q Q - P P ⟗ Q The P ∩ Q portion could be considered inner join P ⋈ Q. A left outer join P ⟕ Q is (P - Q) ∪ (P ∩ Q), a right outer join P ⟖ Q is (P ∩ Q) ∪ (Q - P), and a full outer join P ⟗ Q is (P - Q) ∪ (P ∩ Q) ∪ (Q - P).

Inner Join

By Robert Laing Relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram. P - Q P ∩ Q Q - P P ⟗ Q The above illustrates what relational algebra calls a full outer join which combines three parts, the inner or natural join P ⋈ Q which equates to P ∩ Q, and what this section covers, flanked by left and right antijoins P ▷ Q which equats to P - Q, and Q ▷ P which equats to Q - P.

De Morgan's Law

By Robert Laing What I took to calling copula notation because I read somewhere that the ∧ and ∨ symbols are called copulas (but I can’t find the reference again) is a handy mnemonic for remembering two substitution rules commonly called De Morgan’s Laws: ¬(p ∧ q) can be substituted with ¬p ∨ ¬q ¬(p ∨ q) can be substituted with ¬p ∧ ¬q There’s an interesting relationship with set theory in that

Sum

By Robert Laing In 1863 William Stanley Jevons wrote to George Boole that surely Boole’s operation of addition should be replaced by the more natural ‘inclusive or’ (or ‘union’), leading to the law X+X=X. Boole completely rejected this suggestion (it would have destroyed his system based on ordinary algebra) and broke off the correspondence. — The Algebra of Logic Tradition pqp + q 111 101 011 000 As can be seen from the above quote, that 1 + 1 = 1 in logic arithmetic caused rancour between the field’s founding fathers, and does to this day.

Product

By Robert Laing 2 Value Algebra Since conjunction is closely associated with the word and, it jarred me a bit to discover that it is logic’s equivalent of multiplication, while disjunction — commonly thought of as or — is logic’s addition, the arithmetic operator I associate with and. Why conjunction equates to multiplication is best illustrated by its truth table: pqp · q 111 100 010 000 Moving from binary to any number of propositions, the universal quantification symbol ∀(p) tends to be used, as in

cycles-with-tabling

By adding arc(k, e). we alter the previous example in Transitive Closures to this: Unless guarded against, this will cause tc(a, X). to never escape looping between k and e. Thanks to a relatively recent addition to SWI Prolog Tabled execution, all that’s needed to avoid this is adding one line of code prolog:- table tc/2. to that previously shown in TransitiveClosures. :- table tc/2. arc(a, b). arc(a, c). arc(a, d).

Parallel

Series

By Robert Laing Flow Control Yet another way to think of logic is as electrical switches, where on is true and off is false. Here conjunction is switches in series, and disjunction is switches in parallel. p q r s If any switch in series is not on (such as switch r in the above diagram), it breaks the flow of electric current, meaning a light or whatever the circuit powers would be off.

iterative-deepening

By Robert Laing A snag we hit in puzzle solving is the graph isn’t handily prestored in a database for route finding using transitive closures with infinite loop protection thanks to cycles with tabling. We have to turn to what’s sometimes called generative recursion to dynamically build our graph as we explore it. For this, we’ll return to the techniques introduced in BreadthDepthFirst. We’ll be using iterative deepening depth-first search which gets depth first to mimic breadth first by doing it repeatedly, lengthening its depth-limit one step at a time.

microdata

deletion

#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'delete objects or array elements' Example 'delete "location"' declare -A myarr json2array myarr "$(< ../musicevent.json)" del_key myarr '"location"' When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] }, { "@type": "Person", "image": "/examples/jvanzweden_s.

defaults

Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries. ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html' # also https://opensource.com/article/17/6/bash-parameter-expansion Example 'Basic form ${parameter}' foo="bar" When call echo "${foo}" # brackets are unecessary in this example The output should eq "bar" End Example 'default value of unset variable is empty string' When call echo "${foo}" The output should eq "" End Example 'default value is substituted with ${parameter:-default}' When call echo "${foo:-default value}" The output should eq "default value" End Example 'default value is substituted with ${parameter:-default}' foo="bar" When call echo "${foo:-default value}" The output should eq "bar" End Example ':- doesn`t alter the variable`s value' echo "${foo:-default value}" When call echo "${foo}" The output should eq "" End Example 'Whereas := does' echo "${foo:=default value}" When call echo "${foo}" The output should eq "default value" End End

slicing

Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation. The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is: ${parameter:offset} ${parameter:offset:length} String slices Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart. Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript.

transforming

Lowercasing ${varname@L} #!/bin/bash # find_schema 'artists' 'Arno Carstens' # Returns just the first match function find_schema { local path local file path="${2@L}" path="*${path//[^[:alnum:]]/\*}*" path="/usr/local/webapps2/*/content/${1}/${path}/schema.json" for file in $path; do if [[ -f $file ]]; then echo "$file" return 0 fi done } ExampleGroup 'list artist and venue schema.json files' Example 'Find schema.json for "Arno Carstens"' When call find_schema 'artists' 'Arno Carstens' The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json" End Example 'Find schema.json for "No File Created"' When call find_schema 'artists' 'No File Created' The output should eq '' End Example 'Find schema.

indexed-arrays

Something I’ve found a lot harder than it should be is converting JSON arrays to bash “indexed” (as in not associative) arrays. JSON arrays are comma delimited, similar to csv but with surrounding square brackets. There are many ways to do this. The old way involves temporarily resetting the internal field separator environment variable IFS from its default space, TAB, NEWLINE to a comma, which shellcheck and others frown upon.

grep

manual wikibooks #!/bin/bash # dictionary_matches $dictionary $text function json_dict_words { arr=() while IFS= read -r line; do arr+=("$line") done <<< "$(grep -Fwio -f "${HOME}/share/dict/${1}" <<< "$2" | sort -u)" if [[ -n ${arr[*]} ]]; then jq -cn '$ARGS.positional' --args "${arr[@]}" fi } ExampleGroup 'Create a JSON array of phrases matching those in a dictionary' Example 'find a word in the text' When call json_dict_words 'actors' 'Appel @ Padstal' The output should equal '["Appel"]' The status should be success The error should be blank End Example 'find a phrase in the text' When call json_dict_words 'actors' 'n Aand saam Jakkie Louw (ten bate van SAACA) @ Die Centurion Teater' The output should equal '["Jakkie Louw"]' The status should be success The error should be blank End Example 'find two phrases in the text' When call json_dict_words 'actors' 'Emile Alexander & Dakes LIVE in Johannesburg at Gatzbys LIVE, Midrand - 10 February 2024' The output should equal '["Dakes","Emile Alexander"]' The status should be success The error should be blank End Example 'no matches' When call json_dict_words 'actors' 'Nothing in dictionary' The output should equal '' The status should be success The error should be blank End End As is common, the above uses IFS= read -r and courtesy of the SC2162 I discovered IFS= clears the internal field separator prevents stripping leand and trailing whitepace, which I actuall usually want.

iteration

Iterating over JSON arrays via Bash associative-arrays requires two more functions to the json-utils to json2array and array2json in the top section. A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays. function getkeys { local -n arr arr=$1 local keys keys=$(for key in "${!arr[@]}"; do if [[ $key =~ $2 ]]; then echo "$key" fi done | sort -u) if [[ -z $keys ]]; then echo "Error: $2 is not a valid key" >&2 return 1; fi echo "$keys" } The return 1 part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.

wolf-goat-cabbage

buttons-and-lights

This is a simple example puzzle translated from a kif file into SWI Prolog. The full code is on Swish where the query route(Actions) gives two answers Actions = [does(robot, a), does(robot, b), does(robot, c), does(robot, a), does(robot, b), does(robot, a)] Actions = [does(robot, a), does(robot, b), does(robot, a), does(robot, c), does(robot, b), does(robot, a)] The puzzle consists of three lights labeled p, q, and r, each of which can be on or off, giving us 8 possible states (provided we ignore the state’s step counter).

trimming

The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers. As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples). We need Bash’s composite patterns which are only available if shopt -s extglob is set. Composite patterns allow +(pattern-list) as used below to match possibly more than one space.

menus

Directory traversal I’ve developed my own homegrown way of recursively walking a Hugo site’s content tree as described in this discourse thread since the official way of putting the menu in the site config file isn’t what I want as I add and remove pages. With more experience, I changed the deprecated template with an inline partial My code for layouts/partials/menu.html looks like this: <nav> {{ partial "inline/walk.html" (dict "dir" .

date

The primary reason I’m using bash rather than some programing language is to get easy access to GNU’s powerful date program. It automagically reads most text descriptions of dates and can then translate them to what I want, which is ISO 8601 as specified by schema.org. Getting dates right has been a constant source of bugs — it took me a while to figure that because my development machine is set to my local time, “Africa/Johannesburg” and my server to “UTC”, joeblog.

sed

Manual Delete lines ‘/REGEXP/d’ Create a newline after each newline as needed by md sed G Insert lines ‘a TEXT’ $ seq 3 | sed '2a hello' Transliterate ‘y/SOURCE-CHARS/DEST-CHARS/’ Transliterate any characters in the pattern space which match any of the SOURCE-CHARS with the corresponding character in DEST-CHARS. See the ‘tr’ command from GNU coreutils for similar functionality. Print (as in grep) ‘/REGEXP/p’ Range addresses To extract YAML between lines starting and ending with — in Hugo index.

tree

man page Before I discovered the tree command, I wasted huge amounts of time manually making grapic directory listings. $ tree ~/webapps/frontiersoftware /home/roblaing/webapps/frontiersoftware ├── archetypes │ └── default.md ├── assets ├── config │ └── _default │ └── hugo.json ├── content │ ├── bash │ │ ├── coreutils │ │ │ ├── _index.md │ │ │ └── sed │ │ │ ├── index.md │ │ │ └── spec │ │ │ ├── basics_spec.

Proofs

The contrapositive law (p ⇒ q) ⇔ (¬q ⇒ ¬p) Proof by Contradiction (¬p ⇒ 0) ⇔ p Equivalence to Truth (p ≡ 1) ≡ p Deduction (E1 AND E2 AND · · · AND Ek) ⇒ E Modus ponens (p AND (p ⇒ q)) ⇒ q Resolution ((p + q)(¬p + r)) → (q + r)