path: root/site/posts/meta/Bootstrap.org

#+BEGIN_EXPORT html
<h1>Bootstrapping an Extensible Toolchain</h1>
#+END_EXPORT

A literate program is a particular type of software program where code is not
directly written in source files, but rather in text document as code
snippets. In some sense, literate programming allows for writing in the same
place both the software program and its technical documentation.

That being said, *~cleopatra~* is a toolchain to build a website before being a
literate program, and one of its objective is to be /part of this very website
it is used to generate/. To acheive this, *~cleopatra~* has been written as a
collection of org files which can be either “tangled” using [[https://orgmode.org/worg/org-contrib/babel/][Babel]] or “exported”
as a HTML document. Tangling here refers to extracted marked code blocks into
files.

The page you are currently reading is *~cleopatra~* entry point. Its primilarly
purpose is to introduce two Makefiles: ~Makefile~ and ~bootstrap.mk~.

#+TOC: headlines 2

* The Root of Generation

~Makefile~ serves two purposes: it initiates a few global variables, and it
provides a rule to generate ~bootstrap.mk~.  At this point, some readers may
wonder /why/ we need ~Makefile~ in this context, and the motivation behind this
choice is really reminescent of a boot sequence. The rationale is that we need a
“starting point” for *~cleopatra~*. The toolchain cannot live solely inside
org-files, otherwise there would not have any code to execute the first time we
tried to generate the website. We need an initial Makefile, one that has little
chance to change, so that we can almost consider it read-only. Contrary to the
other Makefiles that we will generate, this one will not be deleted by ~make
clean~.

This is similar to your computer: it requires a firmware to boot, whose purpose
—in a nutshell— is to find and load an operating system.

Modifying the content of ~Makefile~ in this document /will/ modify
~Makefile~. This means one can easily put *~cleopatra~* into an inconsistent
state, which would prevent further generation. This is why the generated
~Makefile~ should be versioned, so that you can restore it using ~git~ if you
made a mistake when you modified it.

We now detail the rules introduce by ~Makefile~, and why they effectively
bootstrap a generation process. For readers interested in using *~cleopatra~*
for their own websites, we highlight the potential modifications they would have
to make.

** Global Constants and Variables

First, ~Makefile~ defines several global “constants” (although as far as I know
~make~ does not support true constant values, it is expected further generation
“components” will not modify them).

In a nutshell,

- ~ROOT~ ::
  Tell Emacs where the root of your website sources is, so that tangled output
  filenames can be given relative to it rather than the org files.  So for
  instance, the ~BLOCK_SRC~ headers for ~Makefile~ looks like

  #+BEGIN_SRC org
  #+BEGIN_SRC makefile :tangle Makefile :noweb tangle
  #+END_SRC

  instead of, /e.g./,

  #+BEGIN_SRC org
  #+BEGIN_SRC makefile :tangle ../../../Makefile :noweb tangle
  #+END_SRC

- ~CLEODIR~ ::
  Tell *~cleopatra~* where its sources live. If you place it inside the ~site/~
  directory (as it is intended), and you enable the use of ~org~ files to author
  your contents, then *~cleopatra~* documents will be part of your website. If
  you don’t want that, just move the directory outside the ~site/~ directory,
  and update the ~CLEODIR~ variable accordingly.

- ~EMACS~ ::
  Tell *~cleopatra~* the command to use to call Emacs. You can modify it to use
  a custom Emacs you build yourself if you so desire. Note that the command *has
  to be prefixed by ~ROOT=${ROOT}~, otherwise the source defined in
  *~cleopatra~* documents will not be tangled in the right places.

For this website, these constants are defined as follows.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
ROOT := $(shell pwd)
CLEODIR := site/posts/meta
#+END_SRC

We then introduce a variable that “generation” components will populate with
their output files (using ~+=~).

- ~GENFILES~ ::
  List *~cleopatra~* Makefiles and scripts tangled throughout the generation
  process (with the notable exception of ~Makefile~ itself).
- ~GENSASS~ ::
  List auxiliary ~sass~ files which can be imported by the main ~sass~ files
  (see [[/posts/meta/Theme/][“Theming and Templating”]]).
- ~CONTENTS~ ::
  List generated files which are part of the target website, and acts as inputs
  for ~soupault~.

#+BEGIN_SRC makefile :tangle Makefile :exports none
GENFILES :=
CONTENTS :=
GENSASS :=
#+END_SRC

#+BEGIN_REMARK
One desired feature for *~cleopatra~* would be to let it populate ~GENFILES~ and
~GENSASS~ automatically, by looking for relevant ~:tangle~ directives. The
challenge lies in the “relevant” part: the risk exists that we have false
posivite. Whether or not it is an issue remains an open question.
#+END_REMARK

** Easy Tangling of Org Documents

We provide the necessary bits to easily tangle Org documents.

The configuration of Babel is done using an emacs lisp script called
~tangle-org.el~ whose status is similar to ~Makefile~. It is part of the
bootstrap process, and therefore lives “outside” of *~cleopatra~* (it is not
deleted with ~make clean~ for instance).  However, it is overwritten. If you try
to modify it and find that *~cleopatra~* does not work properly, you should
restore it using ~git~.

#+BEGIN_SRC emacs-lisp :tangle scripts/tangle-org.el
(require 'org)
(cd (getenv "ROOT"))
(setq org-confirm-babel-evaluate nil)
(setq org-src-preserve-indentation t)
(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)))
(org-babel-tangle)
#+END_SRC

We define variables that ensure that the ~ROOT~ environment variable is set and
~tangle-org.el~ is loaded when using Emacs.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
EMACS := ROOT="${ROOT}" emacs
TANGLE := --batch --load="${ROOT}/scripts/tangle-org.el" 2>> build.log
#+END_SRC

** Bootstrapping

The core purpose of ~Makefile~ remains *(1)* to bootstrap the generation process
by generating and loading ~bootstrap.mk~, and *(2)* to enforce the ~build~ rules
hopefully defined by the latter is called.

For *(2)*, we introduce a ~default~ rule with ~build~ as a
dependency.

#+BEGIN_SRC makefile :tangle Makefile :noweb tangle
default: init-log build

init-log:
	@echo "==============[CLEOPATRA BUILD LOG]==============" \
	    > build.log

.PHONY: init-log default build
#+END_SRC

For *(1)*, we rely on a particular behavior of ~make~ regarding the ~include~
directive. If an operand of ~include~ does not yet exists, ~make~ will search
for a rule to generate it.

#+BEGIN_SRC makefile :noweb yes
<<extends(MK="${MK}", MF="${MF}", IN="${IN}", GF="${GF}", GS="${GS}")>>
#+END_SRC

~&:~ is used in place of ~:~ to separate the target from its dependencies in
this rule to tell to ~make~ that the runned commands will generate all these
files.

#+BEGIN_TODO
Introduce ~noweb~ and ~extends~.
#+END_TODO

#+NAME: extends
#+BEGIN_SRC bash :var MK="" :var IN="" :var GF="" :var GS="" :results output
cat <<EOF
GENFILES += ${MK} ${GF}
GENSASS += ${GS}

include ${MK}

${MK} ${GF} ${GS} \\
  &: \${CLEODIR}/${IN}
	@echo "  tangle  \$<"
	@\${EMACS} $< \${TANGLE}
EOF
#+END_SRC

The twist is, we derive the rule to tangle ~bootstrap.mk~ using
~<<extends>>~.

#+BEGIN_SRC verbatim
<<extends(IN="Bootstrap.org", MK="bootstrap.mk", GF="scripts/update-gitignore.sh")>>
#+END_SRC

This means that modifying code block of ~<<extends>>~ is as “dangerous” as
modifying ~Makefile~ itself. Keep that in mind if you start hacking
*~cleopatra~*!

For purpose of illustrations, here is the snippet generated by Babel from the
previous source block.

#+BEGIN_SRC makefile :tangle Makefile :noweb yes
<<extends(IN="Bootstrap.org", MK="bootstrap.mk", GF="scripts/update-gitignore.sh")>>
#+END_SRC

From now on, the bootstrap process is completed: further generation processes
will fully be defined using literate programming, with no special treatment for
its output. For instance, you may not want to use ~soupault~? You can! Just
modify ~bootstrap.mk~ accordingly.

* Generation Processes

Thanks to ~<<extends>>~, *~cleopatra~* is easily extensible. In this section, we
enumerate the generation processes that are currently used to generate the
website you are reading.

** Authoring Contents

The fact that *~cleopatra~* is a literate program which gradually generates
itself was not intended: it is a consequence of my desire to be able to easily
use whatever format I so desire for writing my contents, and Org documents in
particular.

In the present website, contents can be written in the following format:

- Regular Coq files ::
  Coq is a system which allows to write machine-checked proofs, and it comes
  with a source “prettifier” called ~coqdoc~.
  [[/posts/meta/Contents/Coq/][Learn more about the generation process for Coq files​]]
- Org documents ::
  Emacs comes with a powerful editing mode called [[https://orgmode.org/][Org mode]], and Org documents
  are really pleasant to work with.
  [[/posts/meta/Contents/Org/][Learn more about the generation process for Org documents]]

If you want *~cleopatra~* to support more input formats, you have to

1. Create a org file which, once tangled, provide a dedicated makefile
2. Edit this file (~Bootstrap.org~) here, and use ~<<extends>>~ to make sure it
   is actually tangled when necessary

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(MK="coq.mk", IN="Contents/Coq.org", GS="site/style/coq.sass")>>
<<extends(MK="org.mk", IN="Contents/Org.org", GF="scripts/export-org.el emacs.d", GS="site/style/org.sass")>>
#+END_SRC

** Theming and Templating

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(MK="theme.mk", IN="Theme.org", GS="site/style/main.sass")>>
#+END_SRC

** Postprocessing HTML using ~soupault~

#+BEGIN_SRC makefile :tangle bootstrap.mk :noweb tangle :exports none
<<extends(IN="Soupault.org", GF="soupault.conf")>>
#+END_SRC

#+BEGIN_SRC makefile :tangle bootstrap.mk
CONTENTS += soupault.conf
#+END_SRC

** Wrapping-up

#+BEGIN_SRC makefile :tangle bootstrap.mk
build : ${CONTENTS}
	@echo "     run  soupault"
	@soupault
	@echo "  update  .gitignore"
	@scripts/update-gitignore.sh ${CONTENTS} ${GENFILES} ${GENSASS}
#+END_SRC

#+BEGIN_SRC bash :tangle scripts/update-gitignore.sh :tangle-mode (identity #o755)
#!/bin/bash

BEGIN_MARKER="# begin generated files"
END_MARKER="# begin generated files"

# remove the previous list of generated files to ignore
sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
# remove trailing empty lines
sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore

# output the list of files to ignore
echo "" >> .gitignore
echo ${BEGIN_MARKER} >> .gitignore
for f in $@; do
    echo "${f}" >> .gitignore
done
echo ${END_MARKER} >> .gitignore
#+END_SRC

#+BEGIN_SRC makefile :tangle bootstrap.mk
serve :
	@echo "   start  a python server"
	@cd build; python -m http.server 2>/dev/null

clean :
	@echo "  remove  generated files"
	@rm -rf ${CONTENTS} ${GENFILES} build/

force : clean build

.PHONY : serve clean force build
#+END_SRC

# Local Variables:
# org-src-preserve-indentation: t
# End: