By Robert Laing
This website in its current form represents years of struggling with
literate programming.
Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. — Donald Knuth.
Many decades ago, I used Knuth’s TeX typesetting system, or rather Leslie Lamport’s LaTeX derivation, to publish a book for a friend. It’s somewhat ironic that Knuth called his software documentation system (for Pascal, the programing language taught in a computer science course I did way back in the last century) Web, since it’s not web as in WWW friendly. Knuth’s software produces beautifully typeset pages for printing on paper, usually in pdf, whereas I want hypertext.
I’ve dabbled with a lot of web “Content Mangement Systems” over the years both in my job as a newspaper journalist and for my hobby events listing sites joeblog.co.za, seatavern.co.za, and many more, and have settled on Hugo, mainly because of its support for
syntax highlighting.
Online learning is something that really excited me when it started, and I did some fantastic courses. One by Gregor Kiczales based on a free online textbook How to Design Programs really stressed the importance of test-driven development. The book’s design recipe is fairly similar to Agile, but places emphasis on documenting before you code.
The Agile cult seems to be anti-documentation, at least that’s the impression I got from a supposed guru in a youtube lecture saying: “People don’t read comments. Computers don’t read comments. So why bother with them?”
Comments and documentation have become intertwined through systems such as JavaScript’s JSDoc. Personally, I want the code embedded in the documentation, not vice-versa, which Hugo lets me do. As someone who jumps between programing languages a lot, I also don’t like having to learn a different documentation system for each language.
Testing to learn
The first thing I look for in documentation is examples. The web is a rich resource of these, and I’ve become a huge fan of a specialist search engine phind.com which trawls stackoverflow etc and somehow seems to find the good stuff.
To help remember what I’ve figured out so far, I’m keeping my shellspec “specification” files in the bash section of my Hugo documentation tree to get rendered on this website.
Testing as documentation
Writing is hard, and learning to write takes practice. No simple rules can ensure that you write good specs. One thing to avoid is using code. Code is a bad medium for helping to understand code. Architects don’t make their blueprints out of bricks. — Wired interview with Leslie Lamport
My theory why Unix took over the world is it what slipped onto the budget of a contract for the US patent office who needed a documentation syste. A side product was “man pages”, ie the system came documented, albeit very tersely.
The manual was intended for reference, not learning. Their gazillian options are listed alphabetically, not by how commonly they are used or importance. But they ultimately got us to the web.
As I’m struggling with now, all these “power tools” are not as simple as would be nice, but as a touch typist, what would take days with a GUI takes milliseconds with a CLI if you know how.
Testing as specification
Programmers who advocate writing tests before writing code often believe those tests can serve as a specification. Writing tests does force us to think, and anything that gets us to think before coding is helpful. However, writing tests in code does not get us thinking above the code level. Who Builds a House Without Drawing Blueprints?, an article written by Leslie Lamport
Behavior-driven development systems such as Jasmine for JavaScript and ShellSpec for Bash prefer the jargon specification to tests. Personally, I use a subdirectory called tests rather than specs for my Jasmine and ShellSpec scripts.
[Unit] This is common to all unit types. It contains metadata about the service such as a description.
systemd.unit(5)
/usr/lib/systemd/system/nginx.service
[Unit] Description=nginx web server After=network-online.target remote-fs.target nss-lookup.target Wants=network-online.target /usr/lib/systemd/system/postgresql.service
[Unit] Description=PostgreSQL database server Documentation=man:postgres(1) After=network.target network-online.target Wants=network-online.target /usr/lib/systemd/system/tmp.mount
[Unit] Description=Temporary Directory /tmp Documentation=https://systemd.io/TEMPORARY_DIRECTORIES Documentation=man:file-hierarchy(7) Documentation=https://systemd.io/API_FILE_SYSTEMS ConditionPathIsSymbolicLink=!/tmp DefaultDependencies=no Conflicts=umount.target Before=local-fs.target umount.target After=swap.target
grep me no patterns and I’ll tell you no lines. — fortune cooky
manual
wikibooks
As explained in The Unix Programing Environment by Brian Kernighan and Rob Pike, sed can do just about everything grep can.
sed -n '/pattern/p' files is the same as
grep -h pattern files Why do we have both sed and grep? After all, grep is just a simple special case of sed. Part of the reason is history — grep came well before sed.
Following from iteration, the next step is to add JSON objects to the version converted to bash.
#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'append new JSON object to an array' Example 'add another "workPerformed" object' declare -A myarr json2array myarr "$(< ../musicevent.json)" json1='{ "@type": "CreativeWork", "name": "Sinfonia da Requiem", "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem" }' array_append myarr '"workPerformed"' "$json1" When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "location": { "@type": "MusicVenue", "address": "220 S.
The arithmetic of colours Colour Wheel Munsell color system
hsl(H, S, L[, A])
H (hue) is an angle (degrees by default if no units are given). Red is 0 or 360, green 120, and blue 240.
0° Red
30° Orange
180° Cyan
S (saturation) ranges from 100% for fully the hue to 0% for gray.
L (lightness) is 50% for “normal”. 100% is black and 0% is white.
A (alpha) can be a number between 0 and 1, or a percentage, where the number 1 corresponds to 100% (full opacity).
Block, Element, Modifier github
Tutorials
/* Block component */ .btn {} /* Element that depends upon the block */ .btn__price {} /* Modifier that changes the style of the block */ .btn--orange {} .btn--big {} https://css-tricks.com/bem-101/
https://google.github.io/styleguide/htmlcssguide.html
https://blog.logrocket.com/5-things-to-consider-when-creating-your-css-style-guide-7b85fa70039d/
https://cssguidelin.es/
Templates
By Robert Laing
Lets turn the query “Which students applied for ‘CS’?” around, again using Stanford University dean Jennifer Widom’s basic SQL examples using her college applications data.
Which students have not applied for CS? A trap many people will fall into is to rewrite the σMajor = ‘CS’(Apply) selection to not equal, editing it into something like σMajor != ‘CS’(Apply).
That would return the set for not ‘CS’ as {123, 234, 345, 678, 765, 876}.
By Robert Laing
Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration, modifying the same SQL examples done in conjunction as “which students applied for ‘CS’ and ‘EE’?”.
Which students applied for ‘CS’ or ‘EE’? As with conjunction’s p ∧ q being a spiky version of set theory’s P ∩ Q, disjunction’s p ∨ q is a spiky version of set theory’s P ∪ Q.
By Robert Laing
Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration.
Which students applied for ‘CS’ and ‘EE’? In what I’ve called copula notation, conjunction is written p ∧ q, which is a handy mnemonic that its set equivalent is P ∩ Q. That logic has a spiky version of set theory’s rounded symbol we’ll also encounter in disjunction and implication.
[Service] systemd.service(5)
/usr/lib/systemd/system/nginx.service
[Service] Type=forking PIDFile=/run/nginx.pid PrivateDevices=yes PrivateTmp=true SyslogLevel=err ExecStart=/usr/bin/nginx ExecReload=/usr/bin/nginx -s reload Restart=on-failure KillMode=mixed KillSignal=SIGQUIT TimeoutStopSec=5 /usr/lib/systemd/system/postgresql.service
[Service] Type=notify TimeoutSec=120 User=postgres Group=postgres Environment=PGROOT=/var/lib/postgres SyslogIdentifier=postgres PIDFile=/var/lib/postgres/data/postmaster.pid RuntimeDirectory=postgresql RuntimeDirectoryMode=755 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data ExecStart=/usr/bin/postgres -D ${PGROOT}/data ExecReload=/bin/kill -HUP ${MAINPID} KillMode=mixed KillSignal=SIGINT # Due to PostgreSQL's use of shared memory, OOM killer is often overzealous in # killing Postgres, so adjust it downward OOMScoreAdjust=-200 # Additional security-related features PrivateTmp=true ProtectHome=true ProtectSystem=full NoNewPrivileges=true ProtectControlGroups=true ProtectKernelModules=true ProtectKernelTunables=true PrivateDevices=true RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 RestrictNamespaces=true RestrictRealtime=true SystemCallArchitectures=native systemd.
Manual
Four types of sed scripts 1. Multiple Edits to the Same File 2. Making Changes Across a Set of Files 3. Extracting Contents of a File 4. Edits To Go Substitute [address]s/regexp/replacement/[flags] The ‘s’ command (as in substitute) is probably the most important in ‘sed’ and has a lot of different options. The syntax of the ‘s’ command is ’s/REGEXP/REPLACEMENT/FLAGS'.
The ‘/’ characters may be uniformly replaced by any other single character within any given ‘s’ command.
By Robert Laing
Lets start by repeating the example in not, however using SQL’s EXCEPT operator, which ties into set theory’s P - Q.
In relational algebra, the query we want is ΠsID(Student) - ΠsID(σMajor = ‘CS’(Apply)).
Note that relational algebra involving different tables/sets requires them to be projected into a shared type of compound data structure. Here I’m keeping things simple by only working with their common column sID. In fill values I’ll go into the complexities that negation sometimes causes because of missing columns.
By Robert Laing
Which majors intersect? Lets return to Venn’s 5 relations. Here we want to check A ∩ B ≠ Ø, which is true for any of the first four relations.
We need a self-join for this query, which is a form of intersection.
SELECT DISTINCT a1.major AS A, a2.major AS B FROM apply as a1, apply as a2 WHERE a1.sid = a2.sid ORDER BY A, B; a | b ----------------+----------------
bioengineering | bioengineering
bioengineering | CS
bioengineering | EE
biology | biology
biology | CS
biology | marine biology
CS | bioengineering
CS | biology
CS | CS
CS | EE
CS | marine biology
EE | bioengineering
EE | CS
EE | EE
history | history
history | psychology
marine biology | biology
marine biology | CS
marine biology | marine biology
psychology | history
psychology | psychology
(21 rows) Note everything intersects with itself.
By Robert Laing
The problem of how to deal with unknown values for logical variables led to the development of three-valued logic, which SQL implemented with NULL, and Prolog with unset variables.
Which students have not made any college applications yet? In preparation of outer joins, lets here do the portion called an antijoin
Student ▷ Apply
Our basic goal here is to find the set {456, 567, 654, 789}.
By Robert Laing
As explained in inner join relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.
P - Q
P ∩ Q
Q - P
P
⟗
Q
The P ∩ Q portion could be considered inner join P ⋈ Q. A left outer join P ⟕ Q is (P - Q) ∪ (P ∩ Q), a right outer join P ⟖ Q is (P ∩ Q) ∪ (Q - P), and a full outer join P ⟗ Q is (P - Q) ∪ (P ∩ Q) ∪ (Q - P).
By Robert Laing
Relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.
P - Q
P ∩ Q
Q - P
P
⟗
Q
The above illustrates what relational algebra calls a full outer join which combines three parts, the inner or natural join P ⋈ Q which equates to P ∩ Q, and what this section covers, flanked by left and right antijoins P ▷ Q which equats to P - Q, and Q ▷ P which equats to Q - P.
By Robert Laing
What I took to calling copula notation because I read somewhere that the ∧ and ∨ symbols are called copulas (but I can’t find the reference again) is a handy mnemonic for remembering two substitution rules commonly called De Morgan’s Laws:
¬(p ∧ q) can be substituted with ¬p ∨ ¬q ¬(p ∨ q) can be substituted with ¬p ∧ ¬q There’s an interesting relationship with set theory in that
By Robert Laing
In 1863 William Stanley Jevons wrote to George Boole that surely Boole’s operation of addition should be replaced by the more natural ‘inclusive or’ (or ‘union’), leading to the law X+X=X. Boole completely rejected this suggestion (it would have destroyed his system based on ordinary algebra) and broke off the correspondence. —
The Algebra of Logic Tradition
pqp + q
111
101
011
000
As can be seen from the above quote, that 1 + 1 = 1 in logic arithmetic caused rancour between the field’s founding fathers, and does to this day.
By Robert Laing
2 Value Algebra Since conjunction is closely associated with the word and, it jarred me a bit to discover that it is logic’s equivalent of multiplication, while disjunction — commonly thought of as or — is logic’s addition, the arithmetic operator I associate with and.
Why conjunction equates to multiplication is best illustrated by its truth table:
pqp · q
111
100
010
000
Moving from binary to any number of propositions, the universal quantification symbol ∀(p) tends to be used, as in
By adding arc(k, e). we alter the previous example in Transitive Closures to this:
Unless guarded against, this will cause tc(a, X). to never escape looping between k and e. Thanks to a relatively recent addition to SWI Prolog Tabled execution, all that’s needed to avoid this is adding one line of code prolog:- table tc/2. to that previously shown in TransitiveClosures.
:- table tc/2. arc(a, b). arc(a, c). arc(a, d).
By Robert Laing
Flow Control Yet another way to think of logic is as electrical switches, where on is true and off is false. Here conjunction is switches in series, and disjunction is switches in parallel.
p
q
r
s
If any switch in series is not on (such as switch r in the above diagram), it breaks the flow of electric current, meaning a light or whatever the circuit powers would be off.
By Robert Laing
A snag we hit in puzzle solving is the graph isn’t handily prestored in a database for route finding using transitive closures with infinite loop protection thanks to cycles with tabling. We have to turn to what’s sometimes called generative recursion to dynamically build our graph as we explore it. For this, we’ll return to the techniques introduced in BreadthDepthFirst.
We’ll be using iterative deepening depth-first search which gets depth first to mimic breadth first by doing it repeatedly, lengthening its depth-limit one step at a time.
FILTER can be combined with an aggregate expression.
SELECT city, count(*) FILTER (WHERE temp_lo < 45), max(temp_lo) FROM weather GROUP BY city; SELECT count(*) AS unfiltered, count(*) FILTER (WHERE i < 5) AS filtered FROM generate_series(1,10) AS s(i); FILTER is much like WHERE, except that it removes rows only from the input of the particular aggregate function that it is attached to. Here, the count aggregate counts only rows with temp_lo below 45; but the max aggregate is still applied to all rows, so it still finds the reading of 46.
Game
{ "@context": "https://schema.org", "@type": "Game", "offers":{ "@type":"Offer", "priceCurrency":"$", "price":"17.99", "availableAtOrFrom":"/monopoly-2/en_US/shop/where-to-buy.cfm?brand_guid=DAD28866-1C43-11DD-BD0B-0800200C9A66&prodName=Monopoly%20Game" }, "audience":{ "@type":"PeopleAudience", "suggestedMinAge":"8" }, "description":"Own it all as a high-flying trader in the fast-paced world of real estate. Tour the city for the hottest properties: sites, stations and utilities are all up for grabs. Invest in houses and hotels, then watch the rent come pouring in! Make deals with other players and look out for bargains at auction. There are many ways to get what you want.
This is a synomym for SELECT current_setting ( setting_name text [, missing_ok boolean ] ).
SHOW SHOW log_destination; SHOW logging_collector;
Understanding locks
7 tips for dealing with locks
Something I learnt the hard way by getting the computer to play games over-and-over and then update the average scores each time was it crashed my Postgresql functions with
ERROR: relation "_states" does not exist LINE 1: SELECT id FROM _states WHERE state = NULL The problem was garbage values were getting returned from queries if their table was being updated, and the solution was:
wikipedia
The first line of shell scripts is most commonly written without a space, ie #!/usr/bash however in the examples of O’Reilly’s sed & awk book it’s written #! /bin/sh
According to this stackoverflow post
The shebang (#!) is a 16-bit kernel-interpreted magic “number” (actually 0x23 0x21). Thereafter, the kernel program loader (following the exec-family call) attempts to execute the remainder of the line to handle the remainder of the containing file’s content.
#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'delete objects or array elements' Example 'delete "location"' declare -A myarr json2array myarr "$(< ../musicevent.json)" del_key myarr '"location"' When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] }, { "@type": "Person", "image": "/examples/jvanzweden_s.
Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries.
ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html' # also https://opensource.com/article/17/6/bash-parameter-expansion Example 'Basic form ${parameter}' foo="bar" When call echo "${foo}" # brackets are unecessary in this example The output should eq "bar" End Example 'default value of unset variable is empty string' When call echo "${foo}" The output should eq "" End Example 'default value is substituted with ${parameter:-default}' When call echo "${foo:-default value}" The output should eq "default value" End Example 'default value is substituted with ${parameter:-default}' foo="bar" When call echo "${foo:-default value}" The output should eq "bar" End Example ':- doesn`t alter the variable`s value' echo "${foo:-default value}" When call echo "${foo}" The output should eq "" End Example 'Whereas := does' echo "${foo:=default value}" When call echo "${foo}" The output should eq "default value" End End
Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation.
The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is:
${parameter:offset} ${parameter:offset:length} String slices Bash’s ${parameter:offset:length} differs from Javascript’s slice(indexStart, indexEnd) in that length=indexEnd-indexStart.
Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript.
Lowercasing ${varname@L}
#!/bin/bash # find_schema 'artists' 'Arno Carstens' # Returns just the first match function find_schema { local path local file path="${2@L}" path="*${path//[^[:alnum:]]/\*}*" path="/usr/local/webapps2/*/content/${1}/${path}/schema.json" for file in $path; do if [[ -f $file ]]; then echo "$file" return 0 fi done } ExampleGroup 'list artist and venue schema.json files' Example 'Find schema.json for "Arno Carstens"' When call find_schema 'artists' 'Arno Carstens' The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json" End Example 'Find schema.json for "No File Created"' When call find_schema 'artists' 'No File Created' The output should eq '' End Example 'Find schema.
Something I’ve found a lot harder than it should be is converting JSON arrays to bash “indexed” (as in not associative) arrays.
JSON arrays are comma delimited, similar to csv but with surrounding square brackets. There are many ways to do this. The old way involves temporarily resetting the internal field separator environment variable IFS from its default space, TAB, NEWLINE to a comma, which shellcheck and others frown upon.
Iterating over JSON arrays via Bash associative-arrays requires two more functions to the json-utils to json2array and array2json in the top section.
A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays.
function getkeys { local -n arr arr=$1 local keys keys=$(for key in "${!arr[@]}"; do if [[ $key =~ $2 ]]; then echo "$key" fi done | sort -u) if [[ -z $keys ]]; then echo "Error: $2 is not a valid key" >&2 return 1; fi echo "$keys" } The return 1 part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.
This was my first attempt at illustrating graph traversal using an animated pgn. Instructions on how I did this below.
The animation is made out of 8 png files generated by graphviz’s dot as in dot -T png -o frame7.png frame7.dot. These then were combined into an animated png file using apngasm.
apngasm frame0.png frame1.png frame2.png frame3.png frame4.png frame5.png frame6.png frame7.png -d 1000 -o cgw.png
frame0.dot frame1.dot frame2.dot frame3.dot frame4.dot frame5.
This is a simple example puzzle translated from a kif file into SWI Prolog. The full code is on Swish where the query route(Actions) gives two answers
Actions = [does(robot, a), does(robot, b), does(robot, c), does(robot, a), does(robot, b), does(robot, a)] Actions = [does(robot, a), does(robot, b), does(robot, a), does(robot, c), does(robot, b), does(robot, a)] The puzzle consists of three lights labeled p, q, and r, each of which can be on or off, giving us 8 possible states (provided we ignore the state’s step counter).
The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers.
As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples).
We need Bash’s composite patterns which are only available if
shopt -s extglob is set. Composite patterns allow +(pattern-list) as used below to match possibly more than one space.
Directory traversal I’ve developed my own homegrown way of recursively walking a Hugo site’s content tree as described in this discourse thread since the official way of putting the menu in the site config file isn’t what I want as I add and remove pages.
With more experience, I changed the deprecated template with an inline partial
My code for layouts/partials/menu.html looks like this:
<nav> {{ partial "inline/walk.html" (dict "dir" .
The primary reason I’m using bash rather than some programing language is to get easy access to GNU’s powerful date program. It automagically reads most text descriptions of dates and can then translate them to what I want, which is ISO 8601 as specified by schema.org.
Getting dates right has been a constant source of bugs — it took me a while to figure that because my development machine is set to my local time, “Africa/Johannesburg” and my server to “UTC”, joeblog.
man page
Before I discovered the tree command, I wasted huge amounts of time manually making grapic directory listings.
$ tree ~/webapps/frontiersoftware /home/roblaing/webapps/frontiersoftware ├── archetypes │ └── default.md ├── assets ├── config │ └── _default │ └── hugo.json ├── content │ ├── bash │ │ ├── coreutils │ │ │ ├── _index.md │ │ │ └── sed │ │ │ ├── index.md │ │ │ └── spec │ │ │ ├── basics_spec.
The contrapositive law (p ⇒ q) ⇔ (¬q ⇒ ¬p)
Proof by Contradiction (¬p ⇒ 0) ⇔ p
Equivalence to Truth (p ≡ 1) ≡ p
Deduction (E1 AND E2 AND · · · AND Ek) ⇒ E
Modus ponens (p AND (p ⇒ q)) ⇒ q
Resolution ((p + q)(¬p + r)) → (q + r)