I have finally merged this pull request:

https://github.com/tomassedovic/roguelike-tutorial/pull/53

Started on 9th May, 2019 it took almost five months to finish. It again puts the tutorial in a place I am happy with for now, and it will likely stay there for quite some time. Given the priorities and circumstances in my life, I don’t expect to keep working on it for time to come.

The tutorial itself is here:

http://tomassedovic.github.io/roguelike-tutorial/

And the repo is:

https://github.com/tomassedovic/roguelike-tutorial

But it is in a shape where I can pick it up again.

There were two big motivations for this change: ease of contribution and reducing the mismatch between the code snippets in the tutorial and the actual samples at the end of each chapter.

The tutorial has 13 chapters, each building on top of the previous one and each ending with a file that lets you run the game up to that point.

That means in extremis, changing a single line of code could result in doing the same (or just similar) fix to 13 Rust files not to mention one to more documents (the tutorial pages themselves).

Invariably, people going through the tutorial would notice an earlier instance (which may not even have been the first one) and fix that in the one place (which may be the Rust file or the doc, rarely both).

Having to fix it everywhere was tedious, not obvious and meant that I ended up asking people who kindly help make this project a little better to do lot more work.

In addition, the code in the tutorial didn’t always correspond to the resulting files a 100%. Sometimes it did a hand-wavy “let the compiler guide you” (this seemed like a great idea at the time, sorry!) and other times I just forgot to put a bit of code in.

Moreover, even the files themselves were inconsistent. The order of definitions (functions, enums, structs) moved around, there were typos in comments and so on.

All of this made the tutorial not only hard to update but also hard to follow. The readers had to try to fill in the blanks or fix the mistakes.

I tried to follow the tutorial from scratch myself, but that takes a long time (so you don’t want to do it on every change) and you still miss things.

Something had to change.

Literate Programming

I’d read about Literate programming years ago. Over a decade ago for sure. And the concept itself is much much older than that.

Here’s what it’s about: you write all your text and all your code in a single source (one or more files). That file follows the structure of a document (book, manual, tutorial) and contains snippets of code.

You then have tools that take this source and generate two sets of outputs: the final document ready for human consumption (e.g. a PDF or HTML) and machine-readable source code (the program itself).

It uses metadata in that source to put the various code snippets together in the right order.

The big conceptual difference between this and some of the “generate docs from the source code” or even “generate annotated source code” tools is that these code snippets can be mixed and matched as necessary.

You can for example start with a few lines of a function body, then introduce a new concept, implement that and then finish the rest of the original function.

In other words, you follow the structure of the textual document and generate the code rather than insert text into the code, human order be damned.

Here is an example of an annotated code:

http://gitlet.maryrosecook.com/docs/gitlet.html

And here is the first page of the tutorial:

https://raw.githubusercontent.com/tomassedovic/roguelike-tutorial/master/doc/part-1-graphics.adoc

As you can see, while they both start with an introduction section, the annotated code has to follow the actual source, providing explanations. This would not work for the tutorial.

There, we follow the textual representation of the tutorial annotating the source code snippets (via the tag::name[] / end::name[] entries) so it can be put together later.

This lets us do things such as define the main structure of a function with parts of the body omitted and filled in later.

“Literate” Asciidoctor

The tutorial is already written in Asciidoctor which has all the tools necessary to implement this:

https://asciidoctor.org/docs/user-manual/#include-partial

The short of it is that you can name pieces of the document (including source code snippets) and then include those in other documents.

Here’s a contrived example in a Python-eque pseudocode:

# file: tutorial.adoc

First, we define a function that will process both types of entities
in our game, resetting the turn if they're both done:

[source]
----
tag::process_entities_header[]
def process_entities(game, player, npcs):
end::process_entities_header[]
tag::process_entities_new_turn[]
    if player.action_points == 0 and monster.action_points == 0:
        game.turn += 1
        new_turn(player, npcs)
end::process_entities_new_turn[]

tag::process_entities_if[]
    if player.action_points > 0:
end::process_entities_if[]
        # TODO: process player
tag::process_entities_else[]
    else:
end::process_entities_else[]
        # TODO: process monsters
----

Next, we'll fill in the player actions:

[source]
----
tag::process_player[]
key = process_keys()
if key == 'move':
    player.walk_forward()
    player.action_points -= 1
else if key == 'attack':
    player.attack(closest_monster(monsters))
    player.action_points -= 1
else:
    pass
end::process_player[]
----

And then, do the same for the monsters:

[source]
----
tag::process_monsters[]
for monster in monsters:
    action = monster.ai(game.map)
    monster.act()
end::process_monsters[]
----

As you can see, we will have to implement the `ai` and `act` methods next.

This little source will produce a document looking something like this:

First, we define a function that will process both types of entities in our game, resetting the turn if they’re both done:

def process_entities(game, player, npcs):
    if player.action_points == 0 and monster.action_points == 0:
        game.turn += 1
        new_turn(player, npcs)

    if player.action_points > 0:
        # TODO: process player
    else:
        # TODO: process monsters

Next, we’ll fill in the player actions:

key = process_keys()
if key == 'move':
    player.walk_forward()
    player.action_points -= 1
else if key == 'attack':
    player.attack(closest_monster(monsters))
    player.action_points -= 1
else:
    pass

And then, do the same for the monsters:

for monster in monsters:
    action = monster.ai(game.map)
    monster.act()

As you can see, we will have to implement the ai and act methods next.

As well as the following executable code:

if player.action_points == 0 and monster.action_points == 0:
    game.turn += 1
    new_turn(player, npcs)

if player.action_points > 0:
    key = process_keys()
    if key == 'move':
        player.walk_forward()
        player.action_points -= 1
    else if key == 'attack':
        player.attack(closest_monster(monsters))
        player.action_points -= 1
    else:
        pass
else:
    for monster in monsters:
        action = monster.ai(game.map)
        monster.act()

The code will need to be helped by something that understands the tags and knows how to piece them together. Asciidoctor is able to do that as well:

# file: game.adoc

:doctype: inline
:outfilesuffix: .py

++++
include::tutorial.adoc[tag=process_entities_header]
include::tutorial.adoc[tag=process_entities_new_turn]
include::tutorial.adoc[tag=process_entities_if]
include::tutorial.adoc[tag=process_player,indent=4]
include::tutorial.adoc[tag=process_entities_else]
include::tutorial.adoc[tag=process_monsters,indent=4]
++++

There’s a ton more options I hadn’t even internalised yet (filtering, negative filtering, multiple includes, etc.), but this gets you really really far.

Far enough that all the Rust code for the roguelike tutorial is generated from the adoc files containing the tutorial and all the snippets.

I hesitate to call this full literate programming, because I had not dug deep into Knuth’s paper, the source of the TeX book and the tooling he has written for this purpose.

The syntax he uses is definitely different from the Asciidoctor one. And I’m also unsure whether he needs to build the list of include statements like I do or whether that’s somehow captured in the annotation itself.

Still, on the “annotated source code” vs. “full-on literate programming” scale this is probably much closer to the latter.

Closing Thoughts

It’s been a long process getting here. A lot of that was due to the sheer tedium of marking all these sections in the tutorial and building the include lists for each file.

Plus, the tutorial is really quite long.

Next, there’s been a bunch of refactoring I decided to do in the middle of this. Mostly to make the code generation easier, but these also caused the document to be clearer and easier to follow.

This would definitely been easier if I’d started with this “literate” approach form the start. The original tutorial had a lot of code refactorings (let’s add this parameter to every function and follow the compiler’s complaints). These are much more tedious to do than say adding brand new functions.

And finally, Rust – especially rust-fmt processed idiomatic Rust – is not the greatest language for this. The curly braces are a tedium. So is the fact then when you add one argument too many, you will have to restructure the entire function header (because rust-fmt will break it into multiple lines).

Still, none of this was so bad and I’m really happy with the end-result. I’ve done a couple of smaller changes after this has merged and they were much easier to do than before.

I have to mention that this is not a panacea. For example, if you write snippets that do not make it to any of the final code (because they’re just illustrating a concept for example), nothing will catch that these will get out of date if you rename a variable or something similar.

And the workflow could be better. In the example above, we had to define the function body, prelude, if statement, else statement and both if/else blocks.

It would be much better if we could inverse this: defining the whole function block with holes for the if/else blocks that the code generation would put in.

Something like:

[source]
----
tag::process_entities[]
def process_entities(game, player, npcs):
    if player.action_points == 0 and monster.action_points == 0:
        game.turn += 1
        new_turn(player, npcs)

    if player.action_points > 0:
replace::process_player[]
        # TODO: process player
end::process_player[]
    else:
replace::process_monsters[]
        # TODO: process monsters
end::process_monsters[]
end::process_entities[]
----

The final document would ignore the replace/end directives and keep the todos in place, but the code generator would replace them with the process_player and process_monsters tags.

The default behaviour is really super useful (I wrote the whole tutorial with it!) but I found myself wishing for something like this often.

Also, it’s completely possible that some of the many tagging and filtering options Asciidoctor has can already do this.

And if not, I could write a plugin that does it.

Asciidoctor is awesome.

by Tomas Sedovic on 2nd October, 2019