path: root/site/cleopatra/Bootstrap.org

#+BEGIN_EXPORT html
<h1>Bootstrapping an Extensible Toolchain</h1>
#+END_EXPORT

A literate program is a particular type of software program where code is not
directly written in source files, but rather in text document as code
snippets. In some sense, literate programming allows for writing in the same
place both the software program and its technical documentation.

That being said, *~cleopatra~* is a toolchain to build a website before being a
literate program, and one of its objective is to be /part of this very website
it is used to generate/. To acheive this, *~cleopatra~* has been written as a
collection of org files which can be either “tangled” using [[https://orgmode.org/worg/org-contrib/babel/][Babel]] or “exported”
as a HTML document. Tangling here refers to extracted marked code blocks into
files.

The page you are currently reading is *~cleopatra~* entry point. Its primilarly
purpose is to introduce two Makefiles: ~Makefile~ and ~bootstrap.mk~.

#+TOC: headlines 2

* The Root of Generation

~Makefile~ serves two purposes: it initiates a few global variables, and it
provides a rule to generate ~bootstrap.mk~.  At this point, some readers may
wonder /why/ we need ~Makefile~ in this context, and the motivation behind this
choice is really reminescent of a boot sequence. The rationale is that we need a
“starting point” for *~cleopatra~*. The toolchain cannot live solely inside
org-files, otherwise there would not have any code to execute the first time we
tried to generate the website. We need an initial Makefile, one that has little
chance to change, so that we can almost consider it read-only. Contrary to the
other Makefiles that we will generate, this one will not be deleted by ~make
clean~.

This is similar to your computer: it requires a firmware to boot, whose purpose
—in a nutshell— is to find and load an operating system.

Modifying the content of ~Makefile~ in this document /will/ modify
~Makefile~. This means one can easily put *~cleopatra~* into an inconsistent
state, which would prevent further generation. This is why the generated
~Makefile~ should be versioned, so that you can restore it using ~git~ if you
made a mistake when you modified it.

For readers interested in using *~cleopatra~* for their own websites, this
documents tries to highlight the potential modifications they would have to
make.

** Global Constants and Variables

First, ~Makefile~ defines several global “constants” (although as far as I know
~make~ does not support true constant values, it is expected further generation
process will not modify them).

In a nutshell,

- ~ROOT~ ::
  Tell Emacs where the root of your website sources is, so that tangled output
  filenames can be given relative to it rather than the org files.  So for
  instance, the ~BLOCK_SRC~ headers for ~Makefile~ looks like

  #+BEGIN_SRC org
  #+BEGIN_SRC makefile :tangle Makefile :noweb tangle
  #+END_SRC

  instead of, /e.g./,

  #+BEGIN_SRC org
  #+BEGIN_SRC makefile :tangle ../../Makefile :noweb tangle
  #+END_SRC

- ~CLEODIR~ ::
  Tell *~cleopatra~* where its sources live. If you place it inside the ~site/~
  directory (as it is intended), and you enable the use of ~org~ files to author
  your contents, then *~cleopatra~* documents will be part of your website. If
  you don’t want that, just move the directory outside the ~site/~ directory,
  and update the ~CLEODIR~ variable accordingly.

For this website, these constants are defined as follows.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
ROOT := $(shell pwd)
CLEODIR := site/cleopatra
#+END_SRC

We then introduce a variable that “generation” components will populate with
their output files (using ~+=~).

- ~GENFILES~ ::
  List *~cleopatra~* Makefiles and scripts tangled throughout the generation
  process.
- ~GENAUX~ ::
  List auxiliary files and directories used by the generation processes.
- ~GENSASS~ ::
  List auxiliary ~sass~ files which can later be imported by the main ~sass~
  files (see [[./Theme.org][“Theming and Templating”]]).
- ~CONTENTS~ ::
  List generated files which are part of the target website, and acts as inputs
  for ~soupault~.

#+BEGIN_SRC makefile :tangle Makefile :exports none
GENFILES :=
GENAUX :=
CONTENTS :=
GENSASS :=
#+END_SRC

** Easy Tangling of Org Documents

We provide the necessary bits to easily tangle Org documents.

The configuration of Babel is done using an emacs lisp script called
~tangle-org.el~ whose status is similar to ~Makefile~. It is part of the
bootstrap process, and therefore lives “outside” of *~cleopatra~* (it is not
deleted with ~make clean~ for instance).  However, it is overwritten. If you try
to modify it and find that *~cleopatra~* does not work properly, you should
restore it using ~git~.

#+BEGIN_SRC emacs-lisp :tangle scripts/tangle-org.el
(require 'org)
(cd (getenv "ROOT"))
(setq org-confirm-babel-evaluate nil)
(setq org-src-preserve-indentation t)
(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)))
(org-babel-tangle)
#+END_SRC

We define variables that ensure that the ~ROOT~ environment variable is set and
~tangle-org.el~ is loaded when using Emacs. You can modify ~EMACS~ to use a
custom Emacs that you build yourself if you so desire.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
EMACSBIN := emacs
EMACS := ROOT="${ROOT}" ${EMACSBIN}
TANGLE := --batch --load="${ROOT}/scripts/tangle-org.el" \
          2>> build.log
#+END_SRC

** Bootstrapping

The core purpose of ~Makefile~ remains *(1)* to bootstrap the generation process
by generating and loading ~bootstrap.mk~, and *(2)* to enforce the ~build~ rules
hopefully defined by the latter is called.

For *(2)*, we introduce a ~default~ rule with ~build~ as a
dependency.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
default: init-log build

init-log:
	@echo "===========[CLEOPATRA BUILD LOG]===========" \
	    > build.log

.PHONY: init-log default build
#+END_SRC

For *(1)*, we rely on a particular behavior of ~make~ regarding the ~include~
directive. If an operand of ~include~ does not yet exists, ~make~ will search
for a rule to generate it.

Basically, we are looking for recipes of the following form:

#+BEGIN_SRC makefile :noweb yes
<<extends(MK="${MK}", MF="${MF}", IN="${IN}", GF="${GF}", GS="${GS}")>>
#+END_SRC

where

- ~${IN}~ is the input Org document
- ~${MK}~ lists the tangled Makefiles (typically one, but it could be more)
- ~${GF}~ lists the tangled scripts
- ~${GS}~ lists the tangled SASS scripts

~&:~ is used in place of ~:~ to separate the target from its dependencies in
this rule to tell to ~make~ that the runned commands will generate all these
files.

Writing these rules manually —has yours truly had to do in the early days of his
website— has proven to be error-prone.

One desirable feature for *~cleopatra~* would be to generate them automatically,
by looking for relevant ~:tangle~ directives inside the input Org document. The
challenge lies in the “relevant” part: the risk exists that we have false
posivite. However and as a first steps towards a fully automated solution, we
can leverage the evaluation features of Babel here.

Here is a bash script which, given the proper variables, would generate the
expected Makefile rule.

#+NAME: extends
#+BEGIN_SRC bash :var MK="" :var IN="" :var GF="" :var GS="" :results output
cat <<EOF
GENFILES += ${MK} ${GF}
GENSASS += ${GS}

include ${MK}

${MK} ${GF} ${GS} \\
  &: \${CLEODIR}/${IN}
	@echo "  tangle  \$<"
	@\${EMACS} $< \${TANGLE}
EOF
#+END_SRC

The previous source block is given a name (=extends=), and an explicit lists of
variables (~IN~, ~MK~, ~GF~, and ~GS~). Thanks to the [[https://orgmode.org/worg/org-tutorials/org-latex-export.html][noweb syntax of Babel]], we
can insert the result of the evaluation of =extends= inside another source block
when the latter is tangled.

The twist is, we derive the rule to tangle ~bootstrap.mk~ using
=extends=. The syntax is the following:

#+BEGIN_SRC verbatim
<<extends(IN="Bootstrap.org", MK="bootstrap.mk", GF="scripts/update-gitignore.sh")>>
#+END_SRC

For purpose of illustrations, here is the snippet generated by Babel from the
previous source block.

#+BEGIN_SRC makefile :tangle Makefile :noweb yes
<<extends(IN="Bootstrap.org", MK="bootstrap.mk", GF="scripts/update-gitignore.sh")>>
#+END_SRC

Beware that, as a consequence, modifying code block of =extends= is as
“dangerous” as modifying ~Makefile~ itself. Keep that in mind if you start
hacking *~cleopatra~*!

From now on, the bootstrap process is completed: further generation processes
will fully be defined using literate programming, with no special treatment for
its output. For instance, you may not want to use ~soupault~? You can! Just
modify ~bootstrap.mk~ accordingly.

* Generation Processes

Thanks to =extends=, *~cleopatra~* is easily extensible. In this section, we
enumerate the generation processes that are currently used to generate the
website you are reading.

** Authoring Contents

The fact that *~cleopatra~* is a literate program which gradually generates
itself was not intended: it is a consequence of my desire to be able to easily
use whatever format I so desire for writing my contents, and Org documents in
particular.

In the present website, contents can be written in the following format:

- HTML Files ::
  This requires no particular set-up, since HTML is the /lingua franca/ of
  ~soupault~.
- Regular Coq files ::
  Coq is a system which allows to write machine-checked proofs, and it comes
  with a source “prettifier” called ~coqdoc~.
  [[./Contents/Coq.org][Learn more about the generation process for Coq files​]]
- Org documents ::
  Emacs comes with a powerful editing mode called [[https://orgmode.org/][Org mode]], and Org documents
  are really pleasant to work with.
  [[./Contents/Org.org][Learn more about the generation process for Org documents]]

If you want *~cleopatra~* to support more input formats, you have to

1. Create an Org document which, once tangled, provides a dedicated makefile
2. Edit this file (~Bootstrap.org~) here, and use =extends= to make sure it
   is actually tangled when necessary

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(MK="coq.mk", IN="Contents/Coq.org", GS="site/style/coq.sass")>>
<<extends(MK="org.mk", IN="Contents/Org.org", GF="scripts/packages.el scripts/export-org.el", GS="site/style/org.sass")>>
#+END_SRC

** Postprocessing HTML using ~soupault~

The drawback of using different input formats and generators (~coqdoc~, Org,
etc.) is the heterogeneity of the outputted HTML. This is why *~cleopatra~*
started using ~soupault~. You can read more about [[./Soupault.org][how the ~soupault~
configuration of the present website in the dedicated document]].

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(MK="soupault.mk", IN="Soupault.org", GF="soupault.conf package.json templates/history.html plugins/external-urls.lua plugins/urls-rewriting.lua scripts/katex.js scripts/history.sh", GS="site/style/plugins.sass")>>
#+END_SRC

** Theming and Templating

The last missing piece is the appearance of the website. By default, ~soupault~
assumes there exists a template available (~templates/main.html~). You can read
more about [[./Theme.org][the structure of this template and how its companion CSS file is
generated in the appropriate document]].

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(MK="theme.mk", IN="Theme.org", GF="templates/main.html", GS="site/style/main.sass")>>
#+END_SRC

** Wrapping-up

#+BEGIN_SRC makefile :tangle bootstrap.mk
build : ${CONTENTS} ${GENFILES}
	@echo "     run  soupault"
	@soupault
	@echo "  update  .gitignore"
	@scripts/update-gitignore.sh \
	     ${CONTENTS} \
	     ${GENFILES} \
	     ${GENAUX} \
	     ${GENSASS} \
	     build.log
#+END_SRC

#+BEGIN_SRC bash :tangle scripts/update-gitignore.sh :tangle-mode (identity #o755)
#!/bin/bash

BEGIN_MARKER="# begin generated files"
END_MARKER="# begin generated files"

# remove the previous list of generated files to ignore
sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
# remove trailing empty lines
sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore

# output the list of files to ignore
echo "" >> .gitignore
echo ${BEGIN_MARKER} >> .gitignore
for f in $@; do
    echo "${f}" >> .gitignore
done
echo ${END_MARKER} >> .gitignore
#+END_SRC

#+BEGIN_SRC makefile :tangle bootstrap.mk
serve :
	@echo "   start  a python server"
	@cd build; python -m http.server 2>/dev/null

clean :
	@echo "  remove  generated website"
	@rm -rf ${CONTENTS} build/

cleanall : clean
	@echo "  remove  everything else"
	@rm -rf ${GENSASS} ${GENFILES} ${GENAUX}

force : clean build

.PHONY : serve cleanall clean force build
#+END_SRC

# Local Variables:
# org-src-preserve-indentation: t
# End: