By Robert Laing
This website in its current form represents years of struggling with literate programming.
Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. — Donald Knuth.
Many decades ago, I used Knuth’s TeX typesetting system, or rather Leslie Lamport’s LaTeX derivation, to publish a book for a friend. It’s somewhat ironic that Knuth called his software documentation system (for Pascal, the programing language taught in a computer science course I did way back in the last century) Web, since it’s not web as in WWW friendly. Knuth’s software produces beautifully typeset pages for printing on paper, usually in pdf, whereas I want hypertext.
I’ve dabbled with a lot of web “Content Mangement Systems” over the years both in my job as a newspaper journalist and for my hobby events listing sites joeblog.co.za, seatavern.co.za, and many more, and have settled on Hugo, mainly because of its support for syntax highlighting.
Online learning is something that really excited me when it started, and I did some fantastic courses. One by Gregor Kiczales based on a free online textbook How to Design Programs really stressed the importance of test-driven development. The book’s design recipe is fairly similar to Agile, but places emphasis on documenting before you code.
The Agile cult seems to be anti-documentation, at least that’s the impression I got from a supposed guru in a youtube lecture saying: “People don’t read comments. Computers don’t read comments. So why bother with them?”
Comments and documentation have become intertwined through systems such as JavaScript’s JSDoc. Personally, I want the code embedded in the documentation, not vice-versa, which Hugo lets me do. As someone who jumps between programing languages a lot, I also don’t like having to learn a different documentation system for each language.
Testing to learn
The first thing I look for in documentation is examples. The web is a rich resource of these, and I’ve become a huge fan of a specialist search engine phind.com which trawls stackoverflow etc and somehow seems to find the good stuff.
To help remember what I’ve figured out so far, I’m keeping my shellspec “specification” files in the bash section of my Hugo documentation tree to get rendered on this website.
Testing as documentation
Writing is hard, and learning to write takes practice. No simple rules can ensure that you write good specs. One thing to avoid is using code. Code is a bad medium for helping to understand code. Architects don’t make their blueprints out of bricks. — Wired interview with Leslie Lamport
My theory why Unix took over the world is it what slipped onto the budget of a contract for the US patent office who needed a documentation syste. A side product was “man pages”, ie the system came documented, albeit very tersely.
The manual was intended for reference, not learning. Their gazillian options are listed alphabetically, not by how commonly they are used or importance. But they ultimately got us to the web.
As I’m struggling with now, all these “power tools” are not as simple as would be nice, but as a touch typist, what would take days with a GUI takes milliseconds with a CLI if you know how.
Testing as specification
Programmers who advocate writing tests before writing code often believe those tests can serve as a specification. Writing tests does force us to think, and anything that gets us to think before coding is helpful. However, writing tests in code does not get us thinking above the code level. Who Builds a House Without Drawing Blueprints?, an article written by Leslie Lamport
Behavior-driven development systems such as Jasmine for JavaScript and ShellSpec for Bash prefer the jargon specification to tests. Personally, I use a subdirectory called tests rather than specs for my Jasmine and ShellSpec scripts.
unit
[Unit]
This is common to all unit types. It contains metadata about the service such as a description.
systemd.unit(5)
/usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx web server
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
/usr/lib/systemd/system/postgresql.service
[Unit]
Description=PostgreSQL database server
Documentation=man:postgres(1)
After=network.target network-online.target
Wants=network-online.target
/usr/lib/systemd/system/tmp.mount
[Unit]
Description=Temporary Directory /tmp
Documentation=https://systemd.io/TEMPORARY_DIRECTORIES
Documentation=man:file-hierarchy(7)
Documentation=https://systemd.io/API_FILE_SYSTEMS
ConditionPathIsSymbolicLink=!/tmp
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target
grep
grep me no patterns and I’ll tell you no lines. — fortune cooky
As explained in The Unix Programing Environment by Brian Kernighan and Rob Pike, sed can do just about everything grep can.
sed -n '/pattern/p' files
is the same as
grep -h pattern files
Why do we have both sed and grep? After all, grep is just a simple special case of sed. Part of the reason is history — grep came well before sed. But grep survives, indeed thrives, because for the job that they both do, it is significantly easier to use than sed is.
appending
Following from iteration, the next step is to add JSON objects to the version converted to bash.
#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'append new JSON object to an array' Example 'add another "workPerformed" object' declare -A myarr json2array myarr "$(< ../musicevent.json)" json1='{ "@type": "CreativeWork", "name": "Sinfonia da Requiem", "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem" }' array_append myarr '"workPerformed"' "$json1" When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "location": { "@type": "MusicVenue", "address": "220 S. Michigan Ave, Chicago, Illinois, USA", "name": "Chicago Symphony Center" }, "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] }, { "@type": "Person", "image": "/examples/jvanzweden_s.jpg", "name": "Jaap van Zweden", "sameAs": "http://www.jaapvanzweden.com/" } ], "startDate": "2014-05-23T20:00", "workPerformed": [ { "@type": "CreativeWork", "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes", "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes" }, { "@type": "CreativeWork", "name": "Shostakovich Symphony No. 7 (Leningrad)", "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)" }, { "@type": "CreativeWork", "name": "Sinfonia da Requiem", "sameAs": "https://en.wikipedia.org/wiki/Sinfonia_da_Requiem" } ] }' The status should be success The error should be blank End EndColor
The arithmetic of colours
Colour Wheel
H (hue) is an angle (degrees by default if no units are given). Red is 0 or 360, green 120, and blue 240.
S (saturation) ranges from 100% for fully the hue to 0% for gray.
Style Guides
Block, Element, Modifier
/* Block component */ .btn {} /* Element that depends upon the block */ .btn__price {} /* Modifier that changes the style of the block */ .btn--orange {} .btn--big {}https://css-tricks.com/bem-101/
https://google.github.io/styleguide/htmlcssguide.html
https://blog.logrocket.com/5-things-to-consider-when-creating-your-css-style-guide-7b85fa70039d/
Templates
Not
By Robert Laing
Lets turn the query “Which students applied for ‘CS’?” around, again using Stanford University dean Jennifer Widom’s basic SQL examples using her college applications data.
Which students have not applied for CS?
A trap many people will fall into is to rewrite the σMajor = ‘CS’(Apply) selection to not equal, editing it into something like σMajor != ‘CS’(Apply).
That would return the set for not ‘CS’ as {123, 234, 345, 678, 765, 876}. Recalling that the students who applied for ‘CS’ were {123, 345, 543, 876, 987}, we see we shouldn’t have {123, 345, 876} in our result. The reason those students match
Major <> 'CS'is student 123 also applied for ‘EE’, 345 applied for ‘EE’ and ‘bioengineering’ besides ‘CS’, while 876 applied for ‘biology’ and ‘marine biology’ besides ‘CS’.Or
By Robert Laing
Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration, modifying the same SQL examples done in conjunction as “which students applied for ‘CS’ and ‘EE’?”.
Which students applied for ‘CS’ or ‘EE’?
As with conjunction’s p ∧ q being a spiky version of set theory’s P ∩ Q, disjunction’s p ∨ q is a spiky version of set theory’s P ∪ Q.
And
By Robert Laing
Once again, lets use Stanford University dean Jennifer Widom’s basic SQL examples from her college applications data as an illustration.
Which students applied for ‘CS’ and ‘EE’?
In what I’ve called copula notation, conjunction is written p ∧ q, which is a handy mnemonic that its set equivalent is P ∩ Q. That logic has a spiky version of set theory’s rounded symbol we’ll also encounter in disjunction and implication.
service
[Service]
systemd.service(5)
/usr/lib/systemd/system/nginx.service
[Service] Type=forking PIDFile=/run/nginx.pid PrivateDevices=yes PrivateTmp=true SyslogLevel=err ExecStart=/usr/bin/nginx ExecReload=/usr/bin/nginx -s reload Restart=on-failure KillMode=mixed KillSignal=SIGQUIT TimeoutStopSec=5/usr/lib/systemd/system/postgresql.service
[Service] Type=notify TimeoutSec=120 User=postgres Group=postgres Environment=PGROOT=/var/lib/postgres SyslogIdentifier=postgres PIDFile=/var/lib/postgres/data/postmaster.pid RuntimeDirectory=postgresql RuntimeDirectoryMode=755 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data ExecStart=/usr/bin/postgres -D ${PGROOT}/data ExecReload=/bin/kill -HUP ${MAINPID} KillMode=mixed KillSignal=SIGINT # Due to PostgreSQL's use of shared memory, OOM killer is often overzealous in # killing Postgres, so adjust it downward OOMScoreAdjust=-200 # Additional security-related features PrivateTmp=true ProtectHome=true ProtectSystem=full NoNewPrivileges=true ProtectControlGroups=true ProtectKernelModules=true ProtectKernelTunables=true PrivateDevices=true RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 RestrictNamespaces=true RestrictRealtime=true SystemCallArchitectures=nativesystemd.exec(5)
sed
Four types of sed scripts
1. Multiple Edits to the Same File
2. Making Changes Across a Set of Files
3. Extracting Contents of a File
4. Edits To Go
Substitute [address]s/regexp/replacement/[flags]
The ‘s’ command (as in substitute) is probably the most important in ‘sed’ and has a lot of different options. The syntax of the ‘s’ command is ’s/REGEXP/REPLACEMENT/FLAGS'.
The ‘/’ characters may be uniformly replaced by any other single character within any given ‘s’ command.
Difference
By Robert Laing
Lets start by repeating the example in not, however using SQL’s EXCEPT operator, which ties into set theory’s P - Q.
In relational algebra, the query we want is ΠsID(Student) - ΠsID(σMajor = ‘CS’(Apply)).
Note that relational algebra involving different tables/sets requires them to be projected into a shared type of compound data structure. Here I’m keeping things simple by only working with their common column sID. In fill values I’ll go into the complexities that negation sometimes causes because of missing columns.
Union
Intersection
By Robert Laing
Which majors intersect?
Lets return to Venn’s 5 relations. Here we want to check A ∩ B ≠ Ø, which is true for any of the first four relations.
We need a self-join for this query, which is a form of intersection.
SELECT DISTINCT a1.major AS A, a2.major AS B FROM apply as a1, apply as a2 WHERE a1.sid = a2.sid ORDER BY A, B;a | b ----------------+---------------- bioengineering | bioengineering bioengineering | CS bioengineering | EE biology | biology biology | CS biology | marine biology CS | bioengineering CS | biology CS | CS CS | EE CS | marine biology EE | bioengineering EE | CS EE | EE history | history history | psychology marine biology | biology marine biology | CS marine biology | marine biology psychology | history psychology | psychology (21 rows)Note everything intersects with itself. To remove all the A ≡ A along with A ∩ B so B ∩ A reciprocal intersections, we could add a
a1.major < a2.majortest:Fill Values
By Robert Laing
The problem of how to deal with unknown values for logical variables led to the development of three-valued logic, which SQL implemented with NULL, and Prolog with unset variables.
Which students have not made any college applications yet?
In preparation of outer joins, lets here do the portion called an antijoin
Student ▷ Apply
Our basic goal here is to find the set {456, 567, 654, 789}.
In the example so far involving both the
StudentandApplytables — “Which students have not applied for CS?” — I sidestepped the problem that the student table has columnssID, sName, GPAandsizeHSwhile Apply has columnssID, cName, majoranddecisionby projecting both to their single common column, SID.Outer Join
By Robert Laing
As explained in inner join relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.
The P ∩ Q portion could be considered inner join P ⋈ Q. A left outer join P ⟕ Q is (P - Q) ∪ (P ∩ Q), a right outer join P ⟖ Q is (P ∩ Q) ∪ (Q - P), and a full outer join P ⟗ Q is (P - Q) ∪ (P ∩ Q) ∪ (Q - P).
Inner Join
By Robert Laing
Relational algebra uses various bowtie symbols to describe binary operations on two tables, which are anologous to combining two sets as I’ll try to explain in this diagram.
The above illustrates what relational algebra calls a full outer join which combines three parts, the inner or natural join P ⋈ Q which equates to P ∩ Q, and what this section covers, flanked by left and right antijoins P ▷ Q which equats to P - Q, and Q ▷ P which equats to Q - P.
De Morgan's Law
By Robert Laing
What I took to calling copula notation because I read somewhere that the ∧ and ∨ symbols are called copulas (but I can’t find the reference again) is a handy mnemonic for remembering two substitution rules commonly called De Morgan’s Laws:
- ¬(p ∧ q) can be substituted with ¬p ∨ ¬q
- ¬(p ∨ q) can be substituted with ¬p ∧ ¬q
There’s an interesting relationship with set theory in that
Sum
By Robert Laing
In 1863 William Stanley Jevons wrote to George Boole that surely Boole’s operation of addition should be replaced by the more natural ‘inclusive or’ (or ‘union’), leading to the law X+X=X. Boole completely rejected this suggestion (it would have destroyed his system based on ordinary algebra) and broke off the correspondence.— The Algebra of Logic Tradition
p q p + q 1 1 1 1 0 1 0 1 1 0 0 0 As can be seen from the above quote, that
1 + 1 = 1in logic arithmetic caused rancour between the field’s founding fathers, and does to this day. Jevons used a rotated ÷ symbol, ⋅∣⋅, for or instead of + to sidestep the problem.Product
By Robert Laing
2 Value Algebra
Since conjunction is closely associated with the word and, it jarred me a bit to discover that it is logic’s equivalent of multiplication, while disjunction — commonly thought of as or — is logic’s addition, the arithmetic operator I associate with and.
Why conjunction equates to multiplication is best illustrated by its truth table:
p q p · q 1 1 1 1 0 0 0 1 0 0 0 0 Moving from binary to any number of propositions, the universal quantification symbol ∀(p) tends to be used, as in
cycles-with-tabling
By adding
arc(k, e).we alter the previous example in Transitive Closures to this:
Unless guarded against, this will cause
tc(a, X).to never escape looping between k and e. Thanks to a relatively recent addition to SWI Prolog Tabled execution, all that’s needed to avoid this is adding one line of codeprolog:- table tc/2.to that previously shown in TransitiveClosures.:- table tc/2. arc(a, b). arc(a, c). arc(a, d). arc(b, e). arc(b, f). arc(c, g). arc(c, h). arc(c, i). arc(d, j). arc(e, k). arc(f, l). arc(f, m). arc(h, n). arc(i, o). arc(i, p). arc(j, q). arc(j, r). arc(j, s). arc(k, e). arc(m, t). tc(X, Y) :- arc(X, Y). tc(X, Z) :- arc(X, Y), tc(Y, Z).Tabling changes the order of traversal returned by
tc(a, X)to this:Parallel
Series
By Robert Laing
Flow Control
Yet another way to think of logic is as electrical switches, where on is true and off is false. Here conjunction is switches in series, and disjunction is switches in parallel.
If any switch in series is not on (such as switch r in the above diagram), it breaks the flow of electric current, meaning a light or whatever the circuit powers would be off. If the switches were in parallel, (which I’ll illustrate in disjunction), any on switch would put the circuit on.
iterative-deepening
By Robert Laing
A snag we hit in puzzle solving is the graph isn’t handily prestored in a database for route finding using transitive closures with infinite loop protection thanks to cycles with tabling. We have to turn to what’s sometimes called generative recursion to dynamically build our graph as we explore it. For this, we’ll return to the techniques introduced in BreadthDepthFirst.
We’ll be using iterative deepening depth-first search which gets depth first to mimic breadth first by doing it repeatedly, lengthening its depth-limit one step at a time. The reason is unbounded depth first instead of finding the shortest route, finds the leftmost route. Breadth first does find the shortest route, but even the cheap RAM available on modern computers is often insufficient for it to complete this task.
filter
FILTER can be combined with an aggregate expression.
SELECT city, count(*) FILTER (WHERE temp_lo < 45), max(temp_lo) FROM weather GROUP BY city;SELECT count(*) AS unfiltered, count(*) FILTER (WHERE i < 5) AS filtered FROM generate_series(1,10) AS s(i);FILTER is much like WHERE, except that it removes rows only from the input of the particular aggregate function that it is attached to. Here, the count aggregate counts only rows with temp_lo below 45; but the max aggregate is still applied to all rows, so it still finds the reading of 46. — Aggregate Functions
game
{ "@context": "https://schema.org", "@type": "Game", "offers":{ "@type":"Offer", "priceCurrency":"$", "price":"17.99", "availableAtOrFrom":"/monopoly-2/en_US/shop/where-to-buy.cfm?brand_guid=DAD28866-1C43-11DD-BD0B-0800200C9A66&prodName=Monopoly%20Game" }, "audience":{ "@type":"PeopleAudience", "suggestedMinAge":"8" }, "description":"Own it all as a high-flying trader in the fast-paced world of real estate. Tour the city for the hottest properties: sites, stations and utilities are all up for grabs. Invest in houses and hotels, then watch the rent come pouring in! Make deals with other players and look out for bargains at auction. There are many ways to get what you want. For really speedy dealers, use the speed die for a quick and intense game of Monopoly. So get on Go and trade your way to success!", "gameItem":["gameboard","8 tokens","Title Deed cards","16 Chance cards", "16 Community Chest cards","money pack","32 houses","12 hotels"], "numberOfPlayers":{ "@type":"QuantitativeValue", "minValue":"3", "maxValue":"5" }, "copyrightHolder":"Hasbro" }set
explain
show
This is a synomym for
SELECT current_setting ( setting_name text [, missing_ok boolean ] ).SHOW
SHOW log_destination; SHOW logging_collector;read-write-conflicts
Something I learnt the hard way by getting the computer to play games over-and-over and then update the average scores each time was it crashed my Postgresql functions with
ERROR: relation "_states" does not exist LINE 1: SELECT id FROM _states WHERE state = NULLThe problem was garbage values were getting returned from queries if their table was being updated, and the solution was:
shebang
The first line of shell scripts is most commonly written without a space, ie
#!/usr/bashhowever in the examples of O’Reilly’s sed & awk book it’s written#! /bin/shAccording to this stackoverflow post
The shebang (#!) is a 16-bit kernel-interpreted magic “number” (actually 0x23 0x21). Thereafter, the kernel program loader (following the exec-family call) attempts to execute the remainder of the line to handle the remainder of the containing file’s content. Many, if not most modern kernels disregard preceding spaces to the command that follows the shebang. As such, #! /bin/bash, #! /usr/bin/perl, and the like, should be acceptable with most modern kernels. That said, a shebang with no following space is far more commonly seen.
microdata
deletion
#!/bin/bash source /usr/local/lib/json-utils.sh ExampleGroup 'delete objects or array elements' Example 'delete "location"' declare -A myarr json2array myarr "$(< ../musicevent.json)" del_key myarr '"location"' When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] }, { "@type": "Person", "image": "/examples/jvanzweden_s.jpg", "name": "Jaap van Zweden", "sameAs": "http://www.jaapvanzweden.com/" } ], "startDate": "2014-05-23T20:00", "workPerformed": [ { "@type": "CreativeWork", "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes", "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes" }, { "@type": "CreativeWork", "name": "Shostakovich Symphony No. 7 (Leningrad)", "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)" } ] }' The status should be success The error should be blank End Example 'delete "performer",1,' declare -A myarr json2array myarr "$(< ../musicevent.json)" del_key myarr '"performer",1,' When call array2json myarr The output should equal '{ "@context": "https://schema.org", "@type": "MusicEvent", "location": { "@type": "MusicVenue", "address": "220 S. Michigan Ave, Chicago, Illinois, USA", "name": "Chicago Symphony Center" }, "name": "Shostakovich Leningrad", "offers": { "@type": "Offer", "availability": "https://schema.org/InStock", "price": "40", "priceCurrency": "USD", "url": "/examples/ticket/12341234" }, "performer": [ { "@type": "MusicGroup", "name": "Chicago Symphony Orchestra", "sameAs": [ "http://cso.org/", "http://en.wikipedia.org/wiki/Chicago_Symphony_Orchestra" ] } ], "startDate": "2014-05-23T20:00", "workPerformed": [ { "@type": "CreativeWork", "name": "Britten Four Sea Interludes and Passacaglia from Peter Grimes", "sameAs": "http://en.wikipedia.org/wiki/Peter_Grimes" }, { "@type": "CreativeWork", "name": "Shostakovich Symphony No. 7 (Leningrad)", "sameAs": "http://en.wikipedia.org/wiki/Symphony_No._7_(Shostakovich)" } ] }' The status should be success The error should be blank End Enddefaults
Much of my project involves checking if the provided JSON data has needed key-value pairs, and if missing deciding whether to simply skip (which Bash confused me by calling continue in a loop), or salvaging the entry by figuring out if the required data can be obtained from other entries.
ExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html' # also https://opensource.com/article/17/6/bash-parameter-expansion Example 'Basic form ${parameter}' foo="bar" When call echo "${foo}" # brackets are unecessary in this example The output should eq "bar" End Example 'default value of unset variable is empty string' When call echo "${foo}" The output should eq "" End Example 'default value is substituted with ${parameter:-default}' When call echo "${foo:-default value}" The output should eq "default value" End Example 'default value is substituted with ${parameter:-default}' foo="bar" When call echo "${foo:-default value}" The output should eq "bar" End Example ':- doesn`t alter the variable`s value' echo "${foo:-default value}" When call echo "${foo}" The output should eq "" End Example 'Whereas := does' echo "${foo:=default value}" When call echo "${foo}" The output should eq "default value" End Endslicing
Bash is akin to JavaScript in that it has the equivalent of a string slice and an array slice which use similar notation.
The Bash manual calls these Substring Expansion, but I find JavaScript’s slice more descriptive. The Bash notation is:
${parameter:offset} ${parameter:offset:length}String slices
Bash’s
${parameter:offset:length}differs from Javascript’sslice(indexStart, indexEnd)in thatlength=indexEnd-indexStart.Something I wasn’t aware of was that when using negative offsets and “length”, Bash behaves excactly like Javascript. Negative lengths are actually offsets from the end.
transforming
Lowercasing
${varname@L}#!/bin/bash # find_schema 'artists' 'Arno Carstens' # Returns just the first match function find_schema { local path local file path="${2@L}" path="*${path//[^[:alnum:]]/\*}*" path="/usr/local/webapps2/*/content/${1}/${path}/schema.json" for file in $path; do if [[ -f $file ]]; then echo "$file" return 0 fi done } ExampleGroup 'list artist and venue schema.json files' Example 'Find schema.json for "Arno Carstens"' When call find_schema 'artists' 'Arno Carstens' The output should eq "/usr/local/webapps2/joeblog/content/artists/arno-carstens/schema.json" End Example 'Find schema.json for "No File Created"' When call find_schema 'artists' 'No File Created' The output should eq '' End Example 'Find schema.json for "Black Cat Bones"' When call find_schema 'artists' 'Black Cat Bones' The output should eq '/usr/local/webapps2/joeblog/content/artists/blackcatbones/schema.json' End Example 'Find schema.json for "Bailey`s"' When call find_schema 'venues' "Bailey's" The output should eq '/usr/local/webapps2/joeblog/content/venues/baileys/schema.json' End EndExampleGroup 'from https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html' # also https://opensource.com/article/17/6/bash-parameter-expansion # https://stackoverflow.com/questions/40732193/bash-how-to-use-operator-parameter-expansion-parameteroperator Example '${parameter@operator} U' foo="bar" When call echo "${foo@U}" The output should eq "BAR" End Example '${parameter@operator} u' foo="bar" When call echo "${foo@u}" The output should eq "Bar" End Example '${parameter@operator} L' foo="BAR" When call echo "${foo@L}" The output should eq "bar" End Example 'associative array JSON path' foo=""performer",0,"sameAs",0" When call echo "${foo}" The output should eq "performer,0,sameAs,0" End # Output to be reused as input of another command Example '${parameter@operator} Q' foo=""performer",0,"sameAs",0" When call echo "${foo@Q}" The output should eq "'performer,0,sameAs,0'" End Example '${parameter@operator} a' declare -A arr=(["foo"]="bar") When call echo "${arr@a}" The output should eq "A" End Example '${parameter@operator} K' declare -A arr=([""performer",0,"sameAs",0"]="bar") When call echo "${arr[@]@K}" The output should eq "performer,0,sameAs,0 \"bar\" " End Example '${parameter@operator} k' declare -A arr=([""performer",0,"sameAs",0"]="bar") When call echo "${arr[@]@k}" The output should eq "performer,0,sameAs,0 bar" End Endindexed-arrays
Something I’ve found a lot harder than it should be is converting JSON arrays to bash “indexed” (as in not associative) arrays.
JSON arrays are comma delimited, similar to csv but with surrounding square brackets. There are many ways to do this. The old way involves temporarily resetting the internal field separator environment variable IFS from its default space, TAB, NEWLINE to a comma, which shellcheck and others frown upon.
The modern way involves using readarray (aka mapfile, the two are synonymous).
iteration
Iterating over JSON arrays via Bash associative-arrays requires two more functions to the
json-utilsto json2array and array2json in the top section.A basic form of itereratino is getkeys which returns the keys from any given level of the path. It will work for objects and arrays.
function getkeys { local -n arr arr=$1 local keys keys=$(for key in "${!arr[@]}"; do if [[ $key =~ $2 ]]; then echo "$key" fi done | sort -u) if [[ -z $keys ]]; then echo "Error: $2 is not a valid key" >&2 return 1; fi echo "$keys" }The
return 1part is something I had to add after my initial attempt as I got more familiar with the “logic programing” style of bash, to move to an alternative block of code if the function doesn’t receive a valid key, which could create the key or whatever.wolf-goat-cabbage
This was my first attempt at illustrating graph traversal using an animated pgn. Instructions on how I did this below.
The animation is made out of 8 png files generated by graphviz’s dot as in
dot -T png -o frame7.png frame7.dot. These then were combined into an animated png file using apngasm.
apngasm frame0.png frame1.png frame2.png frame3.png frame4.png frame5.png frame6.png frame7.png -d 1000 -o cgw.pngMy main lesson from creating this diagram was utility values have direction, ie the utility of a node needs to be stored in the graph table with the parent and move, not just with the node itself. This is because the graph is full of cycles, and to avoid the automaton going back, the utility of going back to a node needs to be less than going forward.
buttons-and-lights
This is a simple example puzzle translated from a kif file into SWI Prolog. The full code is on Swish where the query
route(Actions)gives two answers
Actions = [does(robot, a), does(robot, b), does(robot, c), does(robot, a), does(robot, b), does(robot, a)]Actions = [does(robot, a), does(robot, b), does(robot, a), does(robot, c), does(robot, b), does(robot, a)]The puzzle consists of three lights labeled p, q, and r, each of which can be on or off, giving us 8 possible states (provided we ignore the state’s step counter). The player moves from state to state by pressing one of three buttons, labelled a, b, or c and the object is to get to the final state with 6 button presses.
trimming
The symbol for removing from the front is # and from the back is %. A handy mneumonic is number sign # usually precedes numbers while percentage sign % usually follows numbers.
As shown below, these usually need to be doubled to remove the longest match (possibly several whitespaces for my initial examples).
We need Bash’s composite patterns which are only available if
shopt -s extglobis set. Composite patterns allow
+(pattern-list)as used below to match possibly more than one space.menus
Directory traversal
I’ve developed my own homegrown way of recursively walking a Hugo site’s content tree as described in this discourse thread since the official way of putting the menu in the site config file isn’t what I want as I add and remove pages.
With more experience, I changed the deprecated
templatewith an inline partialMy code for
layouts/partials/menu.htmllooks like this:<nav> {{ partial "inline/walk.html" (dict "dir" .Site.Home.Pages) }} {{ define "partials/inline/walk.html" }} <ol> {{ range .dir }} {{ if (eq .BundleType "branch") }} <li><a href="{{ .RelPermalink }}">{{ .Title }}</a> {{ partial "inline/walk.html" (dict "dir" .Pages) }} </li> {{ else }} <li><a href="{{ .RelPermalink }}">{{ .Title }}</a></li> {{ end }} {{ end }} </ol> {{ end }} </nav>Section menu
Hugo’s default menu system requires everything to be manually entered into the system configuration file.
date
The primary reason I’m using bash rather than some programing language is to get easy access to GNU’s powerful date program. It automagically reads most text descriptions of dates and can then translate them to what I want, which is ISO 8601 as specified by schema.org.
Getting dates right has been a constant source of bugs — it took me a while to figure that because my development machine is set to my local time, “Africa/Johannesburg” and my server to “UTC”, joeblog.co.za was showing all events 2 hours early.
tree
Before I discovered the tree command, I wasted huge amounts of time manually making grapic directory listings.
$ tree ~/webapps/frontiersoftware /home/roblaing/webapps/frontiersoftware ├── archetypes │ └── default.md ├── assets ├── config │ └── _default │ └── hugo.json ├── content │ ├── bash │ │ ├── coreutils │ │ │ ├── _index.md │ │ │ └── sed │ │ │ ├── index.md │ │ │ └── spec │ │ │ ├── basics_spec.sh │ │ │ ├── deleting_lines_spec.sh │ │ │ ├── inserting_lines_spec.sh │ │ │ ├── printing_spec.sh │ │ │ ├── spec_helper.sh │ │ │ └── substitution_spec.sh │ │ ├── extra │ │ │ ├── _index.md │ │ │ └── tree │ │ │ └── index.md │ │ ├── _index.md │ │ └── util-linux │ │ └── _index.md │ ├── hugo │ └── _index.md ├── data ├── i18n ├── layouts │ ├── _default │ │ ├── baseof.html │ │ ├── list.html │ │ └── single.html │ └── shortcodes │ └── insert-code.html ├── static └── themes 21 directories, 19 filesProofs
The contrapositive law
(p ⇒ q) ⇔ (¬q ⇒ ¬p)
Proof by Contradiction
(¬p ⇒ 0) ⇔ p
Equivalence to Truth
(p ≡ 1) ≡ p
Deduction
(E1 AND E2 AND · · · AND Ek) ⇒ E
Modus ponens
(p AND (p ⇒ q)) ⇒ q
Resolution
((p + q)(¬p + r)) → (q + r)
