Refactor the build process to use cleopatra the Second

1 files changed, 320 insertions, 0 deletions

diff --git a/site/posts/CleopatraV1.org b/site/posts/CleopatraV1.org
new file mode 100644
index 0000000..349e237
--- /dev/null
+++ b/site/posts/CleopatraV1.org
@@ -0,0 +1,320 @@
+#+BEGIN_EXPORT html
+<h1><strong><code>cleopatra</code></strong>: Bootstrapping an Extensible Toolchain</h1>
+#+END_EXPORT
+
+#+BEGIN_TODO
+You are about to read the first version of *~cleopatra~*, the toolchain
+initially implemented to build the website you are reading. Since then,
+*~cleopatra~* has been completely rewritten as a
+[[https://cleopatra.soap.coffee][independant, more generic command-line
+program]]. That being said, the build process described in this write-up remains
+the one implemented in *~cleopatra~* the Second.
+#+END_TODO
+
+A literate program is a particular type of software program where code is not
+directly written in source files, but rather in text document as code
+snippets. In some sense, literate programming allows for writing in the same
+place both the software program and its technical documentation.
+
+That being said, *~cleopatra~* is a toolchain to build a website before being a
+literate program, and one of its objective is to be /part of this very website
+it is used to generate/. To acheive this, *~cleopatra~* has been written as a
+collection of org files which can be either “tangled” using
+[[https://orgmode.org/worg/org-contrib/babel/][Babel]] or “exported” as a HTML
+document. Tangling here refers to extracted marked code blocks into files.
+
+The page you are currently reading is *~cleopatra~* entry point. Its primilarly
+purpose is to introduce two Makefiles: ~Makefile~ and ~bootstrap.mk~.
+
+#+TOC: headlines 2
+
+~Makefile~ serves two purposes: it initiates a few global variables, and it
+provides a rule to generate ~bootstrap.mk~.  At this point, some readers may
+wonder /why/ we need ~Makefile~ in this context, and the motivation behind this
+choice is really reminescent of a boot sequence. The rationale is that we need a
+“starting point” for *~cleopatra~*. The toolchain cannot live solely inside
+org-files, otherwise there would not have any code to execute the first time we
+tried to generate the website. We need an initial Makefile, one that has little
+chance to change, so that we can almost consider it read-only. Contrary to the
+other Makefiles that we will generate, this one will not be deleted by ~make
+clean~.
+
+This is similar to your computer: it requires a firmware to boot, whose purpose
+—in a nutshell— is to find and load an operating system.
+
+Modifying the content of ~Makefile~ in this document /will/ modify
+~Makefile~. This means one can easily put *~cleopatra~* into an inconsistent
+state, which would prevent further generation. This is why the generated
+~Makefile~ should be versioned, so that you can restore it using ~git~ if you
+made a mistake when you modified it.
+
+For readers interested in using *~cleopatra~* for their own websites, this
+documents tries to highlight the potential modifications they would have to
+make.
+
+* Global Constants and Variables
+
+First, ~Makefile~ defines several global “constants” (although as far as I know
+~make~ does not support true constant values, it is expected further generation
+process will not modify them).
+
+In a nutshell,
+
+- ~ROOT~ ::
+  Tell Emacs where the root of your website sources is, so that tangled output
+  filenames can be given relative to it rather than the org files.  So for
+  instance, the ~BLOCK_SRC~ tangle parameter for ~Makefile~ looks like ~:tangle
+  Makefile~, instead of ~:tangle ../../Makefile~.
+- ~CLEODIR~ ::
+  Tell *~cleopatra~* where its sources live. If you place it inside the ~site/~
+  directory (as it is intended), and you enable the use of ~org~ files to author
+  your contents, then *~cleopatra~* documents will be part of your website. If
+  you don’t want that, just move the directory outside the ~site/~ directory,
+  and update the ~CLEODIR~ variable accordingly.
+
+For this website, these constants are defined as follows.
+
+#+BEGIN_SRC makefile :tangle Makefile :noweb no-export
+ROOT := $(shell pwd)
+CLEODIR := site/cleopatra
+#+END_SRC
+
+We then introduce two variables to list the output of the generation processes,
+with two purposes in mind: keeping the ~.gitignore~ up-to-date automatically,
+and providing rules to remove them.
+
+- ~ARTIFACTS~ ::
+  Short-term artifacts which can be removed frequently without too much
+  hassle. They will be removed by ~make clean~.
+- ~CONFIGURE~ ::
+  Long-term artifacts whose generation can be time consuming. They will only be
+  removed by ~make cleanall~.
+
+#+BEGIN_SRC makefile :tangle Makefile
+ARTIFACTS := build.log
+CONFIGURE :=
+#+END_SRC
+
+Generation processes shall declare new build outputs using the ~+=~ assignement
+operators. Using another operator will likely provent an underisable result.
+
+* Easy Tangling of Org Documents
+
+*~cleopatra~* is a literate program implemented with Org mode, an Emacs major
+editing mode. We provide the necessary bits to easily tangle Org documents.
+
+The configuration of Babel is done using an emacs lisp script called
+~tangle-org.el~ whose status is similar to ~Makefile~. It is part of the
+bootstrap process, and therefore lives “outside” of *~cleopatra~* (it is not
+deleted with ~make clean~ for instance).  However, it is overwritten. If you try
+to modify it and find that *~cleopatra~* does not work properly, you should
+restore it using ~git~.
+
+#+BEGIN_SRC emacs-lisp :tangle scripts/tangle-org.el
+(require 'org)
+(cd (getenv "ROOT"))
+(setq org-confirm-babel-evaluate nil)
+(setq org-src-preserve-indentation t)
+(add-to-list 'org-babel-default-header-args
+             '(:mkdirp . "yes"))
+(org-babel-do-load-languages
+ 'org-babel-load-languages
+ '((shell . t)))
+(org-babel-tangle)
+#+END_SRC
+
+We define variables that ensure that the ~ROOT~ environment variable is set and
+~tangle-org.el~ is loaded when using Emacs.
+
+#+BEGIN_SRC makefile :tangle Makefile
+EMACSBIN := emacs
+EMACS := ROOT="${ROOT}" ${EMACSBIN}
+TANGLE := --batch \
+          --load="${ROOT}/scripts/tangle-org.el" \
+          2>> build.log
+#+END_SRC
+
+Finally, we introduce a
+[[https://www.gnu.org/software/make/manual/html_node/Canned-Recipes.html#Canned-Recipes][canned
+recipe]] to seamlessly tangle a given file.
+
+#+BEGIN_SRC makefile :tangle Makefile
+define emacs-tangle =
+echo "  tangle  $<"
+${EMACS} $< ${TANGLE}
+endef
+#+END_SRC
+
+* Bootstrapping
+
+The core purpose of ~Makefile~ remains to bootstrap the chain of generation
+processes. This chain is divided into three stages: ~prebuild~, ~build~, and
+~postbuild~.
+
+This translates as follows in ~Makefile~.
+
+#+BEGIN_SRC makefile :tangle Makefile
+default : postbuild ignore
+
+init :
+	@rm -f build.log
+
+prebuild : init
+
+build : prebuild
+
+postbuild : build
+
+.PHONY : init prebuild build postbuild ignore
+#+END_SRC
+
+A *generation process* in *~cleopatra~* is a Makefile which provides rules for
+these three stages, along with the utilities used by these rules. More
+precisely, a generation process ~proc~ is defined in ~proc.mk~. The rules of
+~proc.mk~ for each stage are expected to be prefixed by ~proc-~, /e.g./,
+~proc-prebuild~ for the ~prebuild~ stage.
+
+Eventually, the following dependencies are expected between within the chain of
+generation processes.
+
+#+BEGIN_SRC makefile
+prebuild : proc-prebuild
+build : proc-build
+postbuild : proc-postbuild
+
+proc-build : proc-prebuild
+proc-postbuild : proc build
+#+END_SRC
+
+Because *~cleopatra~* is a literate program, generation processes are defined in
+Org documents –which may contains additional utilities like scripts or
+templates—, and therefore need to be tangled prior to be effectively
+useful. *~cleopatra~ relies on a particular behavior of ~make~ regarding the
+~include~ directive. If there exists a rule to generate a Makefile used as an
+operand of ~include~, ~make~ will use this rule to update (if necessary) said
+Makefile before actually including it.
+
+Therefore, rules of the following form achieve our ambition of extensibility.
+
+#+BEGIN_SRC makefile :noweb yes
+<<extends(PROC="${PROC}", IN="${IN}", AUX="${AUX}")>>
+#+END_SRC
+
+where
+
+- ~${IN}~ is the Org document which contains the generation process code
+- ~${PROC}~ is the name of the generation process
+- ~${AUX}~ lists the utilities of the generation process tangled from ~${IN}~
+  with ~${PROC}.mk~
+
+We use ~&:~ is used in place of ~:~ to separate the target from its dependencies
+in the “tangle rule.” This tells ~make~ that the recipe of this rule generates
+all these files.
+
+Writing these rules manually —has yours truly had to do in the early days of his
+website— has proven to be error-prone.
+
+One desirable feature for *~cleopatra~* would be to generate them automatically,
+by looking for relevant ~:tangle~ directives inside the input Org document. The
+challenge lies in the “relevant” part: the risk exists that we have false
+posivite. However and as a first steps towards a fully automated solution, we
+can leverage the evaluation features of Babel here.
+
+Here is a bash script which, given the proper variables, would generate the
+expected Makefile rule.
+
+#+NAME: extends
+#+BEGIN_SRC bash :var PROC="" :var AUX="" :var IN="" :results output
+cat <<EOF
+include ${PROC}.mk
+
+prebuild : ${PROC}-prebuild
+build : ${PROC}-build
+postbuild : ${PROC}-postbuild
+
+${PROC}-prebuild : ${PROC}.mk ${AUX}
+${PROC}-build : ${PROC}-prebuild
+${PROC}-postbuild : ${PROC}-build
+
+${PROC}.mk ${AUX} &:\\
+   \${CLEODIR}/${IN}
+	@\$(emacs-tangle)
+
+CONFIGURE += ${PROC}.mk ${AUX}
+
+.PHONY : ${PROC}-prebuild \\
+         ${PROC}-build \\
+         ${PROC}-postbuild
+EOF
+#+END_SRC
+
+The previous source block is given a name (=extends=), and an explicit lists of
+variables (~IN~, ~PROC~, and ~AUX~). Thanks to the
+[[https://orgmode.org/worg/org-tutorials/org-latex-export.html][noweb syntax of
+Babel]], we can insert the result of the evaluation of =extends= inside another
+source block when the latter is tangled.
+
+We derive the rule to tangle ~bootstrap.mk~ using =extends=, which gives us the
+following Makefile snippet.
+
+#+BEGIN_SRC makefile :tangle Makefile :noweb yes
+<<extends(IN="Bootstrap.org", PROC="bootstrap", AUX="scripts/update-gitignore.sh")>>
+#+END_SRC
+
+Beware that, as a consequence, modifying code block of =extends= is as
+“dangerous” as modifying ~Makefile~ itself. Keep that in mind if you start
+hacking *~cleopatra~*!
+
+Additional customizations of *~cleopatra~* will be parth ~bootstrap.mk~, rather
+than ~Makefile~.
+
+* Generation Processes
+
+Using the =extends= noweb reference, *~cleopatra~* is easily extensible. In
+this section, we first detail the structure of a typical generation process.
+Then, we construct ~bootstrap.mk~ by enumerating the generation processes that
+are currently used to generate the website you are reading.
+
+Each generation process shall
+
+1. Define ~proc-prebuild~, ~proc-build~, and ~proc-postbuild~
+2. Declare dependencies between stages of generation processes
+3. Declare build outputs (see ~ARTIFACTS~ and ~CONFIGURE~)
+
+* Wrapping-up
+
+#+BEGIN_SRC bash :tangle scripts/update-gitignore.sh :shebang "#+/bin/bash"
+BEGIN_MARKER="# begin generated files"
+END_MARKER="# begin generated files"
+
+# remove the previous list of generated files to ignore
+sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
+# remove trailing empty lines
+sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore
+
+# output the list of files to ignore
+echo "" >> .gitignore
+echo ${BEGIN_MARKER} >> .gitignore
+for f in $@; do
+    echo "${f}" >> .gitignore
+done
+echo ${END_MARKER} >> .gitignore
+#+END_SRC
+
+#+BEGIN_SRC makefile :tangle bootstrap.mk
+ignore :
+	@echo "  update  gitignore"
+	@scripts/update-gitignore.sh \
+	   ${ARTIFACTS} \
+	   ${CONFIGURE}
+
+clean :
+	@rm -rf ${ARTIFACTS}
+
+cleanall : clean
+	@rm -rf ${CONFIGURE}
+#+END_SRC
+
+# Local Variables:
+# org-src-preserve-indentation: t
+# End: