Frontier Software

By Robert Laing

This website in its current form represents years of struggling with literate programming.

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. — Donald Knuth.

Many decades ago, I used Knuth’s TeX typesetting system, or rather Leslie Lamport’s LaTeX derivation, to publish a book for a friend. It’s somewhat ironic that Knuth called his software documentation system (for Pascal, the programing language taught in a computer science course I did way back in the last century) Web, since it’s not web as in WWW friendly. Knuth’s software produces beautifully typeset pages for printing on paper, usually in pdf, whereas I want hypertext.

I’ve dabbled with a lot of web “Content Mangement Systems” over the years both in my job as a newspaper journalist and for my hobby events listing sites joeblog.co.za, seatavern.co.za, and many more, and have settled on Hugo, mainly because of its support for syntax highlighting.

Online learning is something that really excited me when it started, and I did some fantastic courses. One by Gregor Kiczales based on a free online textbook How to Design Programs really stressed the importance of test-driven development. The book’s design recipe is fairly similar to Agile, but places emphasis on documenting before you code.

The Agile cult seems to be anti-documentation, at least that’s the impression I got from a supposed guru in a youtube lecture saying: “People don’t read comments. Computers don’t read comments. So why bother with them?”

Comments and documentation have become intertwined through systems such as JavaScript’s JSDoc. Personally, I want the code embedded in the documentation, not vice-versa, which Hugo lets me do. As someone who jumps between programing languages a lot, I also don’t like having to learn a different documentation system for each language.

Testing to learn

The first thing I look for in documentation is examples. The web is a rich resource of these, and I’ve become a huge fan of a specialist search engine phind.com which trawls stackoverflow etc and somehow seems to find the good stuff.

To help remember what I’ve figured out so far, I’m keeping my shellspec “specification” files in the bash section of my Hugo documentation tree to get rendered on this website.

Testing as documentation

Writing is hard, and learning to write takes practice. No simple rules can ensure that you write good specs. One thing to avoid is using code. Code is a bad medium for helping to understand code. Architects don’t make their blueprints out of bricks. — Wired interview with Leslie Lamport

There was a great quote from Lamport I read online but can no longer find (again a reminder why it’s important to to have an easy to use online notetaking system) in which he argued tests are not specifications. I’m not sure what Lamport’s views on tests as documentation would be, but probably that they’re better than nothing.

My theory why Unix took over the world is it what slipped onto the budget of a contract for the US patent office who needed a documentation syste. A side product was “man pages”, ie the system came documented, albeit very tersely.

The manual was intended for reference, not learning. Their gazillian options are listed alphabetically, not by how commonly they are used or importance. But they ultimately got us to the web.

As I’m struggling with now, all these “power tools” are not as simple as would be nice, but as a touch typist, what would take days with a GUI takes milliseconds with a CLI if you know how.

unit

[Unit] This is common to all unit types. It contains metadata about the service such as a description. systemd.unit(5) /usr/lib/systemd/system/nginx.service [Unit] Description=nginx web server After=network-online.target remote-fs.target nss-lookup.target Wants=network-online.target /usr/lib/systemd/system/postgresql.service [Unit] Description=PostgreSQL database server Documentation=man:postgres(1) After=network.target network-online.target Wants=network-online.target /usr/lib/systemd/system/tmp.mount [Unit] Description=Temporary Directory /tmp Documentation=https://systemd.io/TEMPORARY_DIRECTORIES Documentation=man:file-hierarchy(7) Documentation=https://systemd.io/API_FILE_SYSTEMS ConditionPathIsSymbolicLink=!/tmp DefaultDependencies=no Conflicts=umount.target Before=local-fs.target umount.target After=swap.target

grep

grep me no patterns and I’ll tell you no lines. — fortune cooky manual wikibooks As explained in The Unix Programing Environment by Brian Kernighan and Rob Pike, sed can do just about everything grep can. sed -n '/pattern/p' files is the same as grep -h pattern files Why do we have both sed and grep? After all, grep is just a simple special case of sed. Part of the reason is history — grep came well before sed.

appending

Following from iteration, the next step is to add JSON objects to the version converted to bash. #!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'append new JSON object to an array' Example 'add another "workPerformed" object' declare -A myarr json2array myarr "$(< ../musicevent.json)" json1='{ "@type": "CreativeWork", "name": "Sinfonia da Requiem", "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem" }' array_append myarr '"workPerformed"' "$json1" When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "location": { "@type": "MusicVenue", "address": "220 S.

Color

The arithmetic of colours Colour Wheel Munsell color system hsl(H, S, L[, A]) H (hue) is an angle (degrees by default if no units are given). Red is 0 or 360, green 120, and blue 240. 0° Red 30° Orange 180° Cyan S (saturation) ranges from 100% for fully the hue to 0% for gray. L (lightness) is 50% for “normal”. 100% is black and 0% is white. A (alpha) can be a number between 0 and 1, or a percentage, where the number 1 corresponds to 100% (full opacity).

Style Guides

Block, Element, Modifier github Tutorials /* Block component */ .btn {} /* Element that depends upon the block */ .btn__price {} /* Modifier that changes the style of the block */ .btn--orange {} .btn--big {} https://css-tricks.com/bem-101/ https://google.github.io/styleguide/htmlcssguide.html https://blog.logrocket.com/5-things-to-consider-when-creating-your-css-style-guide-7b85fa70039d/ https://cssguidelin.es/ Templates

Not

By Robert Laing Lets turn the query “Which students applied for ‘CS’?” around, again using Stanford University dean Jennifer Widom’s basic SQL examples using her college applications data. Which students have not applied for CS? A trap many people will fall into is to rewrite the σMajor = ‘CS’(Apply) selection to not equal, editing it into something like σMajor != ‘CS’(Apply). That would return the set for not ‘CS’ as {123, 234, 345, 678, 765, 876}.

Or

By Robert Laing Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration, modifying the same SQL examples done in conjunction as “which students applied for ‘CS’ and ‘EE’?”. Which students applied for ‘CS’ or ‘EE’? As with conjunction’s p ∧ q being a spiky version of set theory’s P ∩ Q, disjunction’s p ∨ q is a spiky version of set theory’s P ∪ Q.

And

By Robert Laing Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration. Which students applied for ‘CS’ and ‘EE’? In what I’ve called copula notation, conjunction is written p ∧ q, which is a handy mnemonic that its set equivalent is P ∩ Q. That logic has a spiky version of set theory’s rounded symbol we’ll also encounter in disjunction and implication.

service

[Service] systemd.service(5) /usr/lib/systemd/system/nginx.service [Service] Type=forking PIDFile=/run/nginx.pid PrivateDevices=yes PrivateTmp=true SyslogLevel=err ExecStart=/usr/bin/nginx ExecReload=/usr/bin/nginx -s reload Restart=on-failure KillMode=mixed KillSignal=SIGQUIT TimeoutStopSec=5 /usr/lib/systemd/system/postgresql.service [Service] Type=notify TimeoutSec=120 User=postgres Group=postgres Environment=PGROOT=/var/lib/postgres SyslogIdentifier=postgres PIDFile=/var/lib/postgres/data/postmaster.pid RuntimeDirectory=postgresql RuntimeDirectoryMode=755 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data ExecStart=/usr/bin/postgres -D ${PGROOT}/data ExecReload=/bin/kill -HUP ${MAINPID} KillMode=mixed KillSignal=SIGINT # Due to PostgreSQL's use of shared memory, OOM killer is often overzealous in # killing Postgres, so adjust it downward OOMScoreAdjust=-200 # Additional security-related features PrivateTmp=true ProtectHome=true ProtectSystem=full NoNewPrivileges=true ProtectControlGroups=true ProtectKernelModules=true ProtectKernelTunables=true PrivateDevices=true RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 RestrictNamespaces=true RestrictRealtime=true SystemCallArchitectures=native systemd.

sed

Manual Four types of sed scripts 1. Multiple Edits to the Same File 2. Making Changes Across a Set of Files 3. Extracting Contents of a File 4. Edits To Go Substitute [address]s/regexp/replacement/[flags] The ‘s’ command (as in substitute) is probably the most important in ‘sed’ and has a lot of different options. The syntax of the ‘s’ command is ’s/REGEXP/REPLACEMENT/FLAGS'. The ‘/’ characters may be uniformly replaced by any other single character within any given ‘s’ command.

Difference

By Robert Laing Lets start by repeating the example in not, however using SQL’s EXCEPT operator, which ties into set theory’s P - Q. In relational algebra, the query we want is ΠsID(Student) - ΠsID(σMajor = ‘CS’(Apply)). Note that relational algebra involving different tables/sets requires them to be projected into a shared type of compound data structure. Here I’m keeping things simple by only working with their common column sID. In fill values I’ll go into the complexities that negation sometimes causes because of missing columns.

Union

Intersection

Fill Values

By Robert Laing The problem of how to deal with unknown values for logical variables led to the development of three-valued logic, which SQL implemented with NULL, and Prolog with unset variables. Which students have not made any college applications yet? In preparation of outer joins, lets here do the portion called an antijoin Student ▷ Apply Our basic goal here is to find the set {456, 567, 654, 789}.

Outer Join

By Robert Laing As explained in inner join relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram. P - Q P ∩ Q Q - P P ⟗ Q The P ∩ Q portion could be considered inner join P ⋈ Q. A left outer join P ⟕ Q is (P - Q) ∪ (P ∩ Q), a right outer join P ⟖ Q is (P ∩ Q) ∪ (Q - P), and a full outer join P ⟗ Q is (P - Q) ∪ (P ∩ Q) ∪ (Q - P).

Inner Join

By Robert Laing Relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram. P - Q P ∩ Q Q - P P ⟗ Q The above illustrates what relational algebra calls a full outer join which combines three parts, the inner or natural join P ⋈ Q which equates to P ∩ Q, and what this section covers, flanked by left and right antijoins P ▷ Q which equats to P - Q, and Q ▷ P which equats to Q - P.

De Morgan's Law

By Robert Laing What I took to calling copula notation because I read somewhere that the ∧ and ∨ symbols are called copulas (but I can’t find the reference again) is a handy mnemonic for remembering two substitution rules commonly called De Morgan’s Laws: ¬(p ∧ q) can be substituted with ¬p ∨ ¬q ¬(p ∨ q) can be substituted with ¬p ∧ ¬q There’s an interesting relationship with set theory in that

Sum

By Robert Laing In 1863 William Stanley Jevons wrote to George Boole that surely Boole’s operation of addition should be replaced by the more natural ‘inclusive or’ (or ‘union’), leading to the law X+X=X. Boole completely rejected this suggestion (it would have destroyed his system based on ordinary algebra) and broke off the correspondence. — The Algebra of Logic Tradition pqp + q 111 101 011 000 As can be seen from the above quote, that 1 + 1 = 1 in logic arithmetic caused rancour between the field’s founding fathers, and does to this day.

Product

By Robert Laing 2 Value Algebra Since conjunction is closely associated with the word and, it jarred me a bit to discover that it is logic’s equivalent of multiplication, while disjunction — commonly thought of as or — is logic’s addition, the arithmetic operator I associate with and. Why conjunction equates to multiplication is best illustrated by its truth table: pqp · q 111 100 010 000 Moving from binary to any number of propositions, the universal quantification symbol ∀(p) tends to be used, as in

cycles-with-tabling

By adding arc(k, e). we alter the previous example in Transitive Closures to this: Unless guarded against, this will cause tc(a, X). to never escape looping between k and e. Thanks to a relatively recent addition to SWI Prolog Tabled execution, all that’s needed to avoid this is adding one line of code prolog:- table tc/2. to that previously shown in TransitiveClosures. :- table tc/2. arc(a, b). arc(a, c). arc(a, d).

Parallel

Series

By Robert Laing Flow Control Yet another way to think of logic is as electrical switches, where on is true and off is false. Here conjunction is switches in series, and disjunction is switches in parallel. p q r s If any switch in series is not on (such as switch r in the above diagram), it breaks the flow of electric current, meaning a light or whatever the circuit powers would be off.

iterative-deepening

By Robert Laing A snag we hit in puzzle solving is the graph isn’t handily prestored in a database for route finding using transitive closures with infinite loop protection thanks to cycles with tabling. We have to turn to what’s sometimes called generative recursion to dynamically build our graph as we explore it. For this, we’ll return to the techniques introduced in BreadthDepthFirst. We’ll be using iterative deepening depth-first search which gets depth first to mimic breadth first by doing it repeatedly, lengthening its depth-limit one step at a time.

shebang

wikipedia The first line of shell scripts is most commonly written without a space, ie #!/usr/bash however in the examples of O’Reilly’s sed & awk book it’s written #! /bin/sh According to this stackoverflow post The shebang (#!) is a 16-bit kernel-interpreted magic “number” (actually 0x23 0x21). Thereafter, the kernel program loader (following the exec-family call) attempts to execute the remainder of the line to handle the remainder of the containing file’s content.

microdata

deletion

#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'delete objects or array elements' Example 'delete "location"' declare -A myarr json2array myarr "$(< ../musicevent.json)" del_key myarr '"location"' When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] }, { "@type": "Person", "image": "/examples/jvanzweden_s.

defaults

Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries. ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html' # also https://opensource.com/article/17/6/bash-parameter-expansion Example 'Basic form ${parameter}' foo="bar" When call echo "${foo}" # brackets are unecessary in this example The output should eq "bar" End Example 'default value of unset variable is empty string' When call echo "${foo}" The output should eq "" End Example 'default value is substituted with ${parameter:-default}' When call echo "${foo:-default value}" The output should eq "default value" End Example 'default value is substituted with ${parameter:-default}' foo="bar" When call echo "${foo:-default value}" The output should eq "bar" End Example ':- doesn`t alter the variable`s value' echo "${foo:-default value}" When call echo "${foo}" The output should eq "" End Example 'Whereas := does' echo "${foo:=default value}" When call echo "${foo}" The output should eq "default value" End End

slicing

Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation. The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is: ${parameter:offset} ${parameter:offset:length} String slices Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart. Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript.

transforming

Lowercasing ${varname@L} #!/bin/bash # find_schema 'artists' 'Arno Carstens' # Returns just the first match function find_schema { local path local file path="${2@L}" path="*${path//[^[:alnum:]]/\*}*" path="/usr/local/webapps2/*/content/${1}/${path}/schema.json" for file in $path; do if [[ -f $file ]]; then echo "$file" return 0 fi done } ExampleGroup 'list artist and venue schema.json files' Example 'Find schema.json for "Arno Carstens"' When call find_schema 'artists' 'Arno Carstens' The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json" End Example 'Find schema.json for "No File Created"' When call find_schema 'artists' 'No File Created' The output should eq '' End Example 'Find schema.

indexed-arrays

Something I’ve found a lot harder than it should be is converting JSON arrays to bash “indexed” (as in not associative) arrays. JSON arrays are comma delimited, similar to csv but with surrounding square brackets. There are many ways to do this. The old way involves temporarily resetting the internal field separator environment variable IFS from its default space, TAB, NEWLINE to a comma, which shellcheck and others frown upon.

iteration

Iterating over JSON arrays via Bash associative-arrays requires two more functions to the json-utils to json2array and array2json in the top section. A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays. function getkeys { local -n arr arr=$1 local keys keys=$(for key in "${!arr[@]}"; do if [[ $key =~ $2 ]]; then echo "$key" fi done | sort -u) if [[ -z $keys ]]; then echo "Error: $2 is not a valid key" >&2 return 1; fi echo "$keys" } The return 1 part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.

wolf-goat-cabbage

buttons-and-lights

This is a simple example puzzle translated from a kif file into SWI Prolog. The full code is on Swish where the query route(Actions) gives two answers Actions = [does(robot, a), does(robot, b), does(robot, c), does(robot, a), does(robot, b), does(robot, a)] Actions = [does(robot, a), does(robot, b), does(robot, a), does(robot, c), does(robot, b), does(robot, a)] The puzzle consists of three lights labeled p, q, and r, each of which can be on or off, giving us 8 possible states (provided we ignore the state’s step counter).

trimming

The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers. As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples). We need Bash’s composite patterns which are only available if shopt -s extglob is set. Composite patterns allow +(pattern-list) as used below to match possibly more than one space.

menus

Directory traversal I’ve developed my own homegrown way of recursively walking a Hugo site’s content tree as described in this discourse thread since the official way of putting the menu in the site config file isn’t what I want as I add and remove pages. With more experience, I changed the deprecated template with an inline partial My code for layouts/partials/menu.html looks like this: <nav> {{ partial "inline/walk.html" (dict "dir" .

date

The primary reason I’m using bash rather than some programing language is to get easy access to GNU’s powerful date program. It automagically reads most text descriptions of dates and can then translate them to what I want, which is ISO 8601 as specified by schema.org. Getting dates right has been a constant source of bugs — it took me a while to figure that because my development machine is set to my local time, “Africa/Johannesburg” and my server to “UTC”, joeblog.

tree

man page Before I discovered the tree command, I wasted huge amounts of time manually making grapic directory listings. $ tree ~/webapps/frontiersoftware /home/roblaing/webapps/frontiersoftware ├── archetypes │ └── default.md ├── assets ├── config │ └── _default │ └── hugo.json ├── content │ ├── bash │ │ ├── coreutils │ │ │ ├── _index.md │ │ │ └── sed │ │ │ ├── index.md │ │ │ └── spec │ │ │ ├── basics_spec.

Proofs

The contrapositive law (p ⇒ q) ⇔ (¬q ⇒ ¬p) Proof by Contradiction (¬p ⇒ 0) ⇔ p Equivalence to Truth (p ≡ 1) ≡ p Deduction (E1 AND E2 AND · · · AND Ek) ⇒ E Modus ponens (p AND (p ⇒ q)) ⇒ q Resolution ((p + q)(¬p + r)) → (q + r)