Data Engineering

I'm talking specifically about Mermaid. Something like DOT is slightly better. The ultimate goal is to extract the formal structure. For Mermaid or DOT, this extraction requires a text parser and a walk over an AST; it's about half of a compiler!

This might not sound like a problem compared to something like PBs or JSON, which also require something that looks like a parser and a tree-walker. The difference is in the tooling; the DOT tools can't directly yank a DAG from a file or iterate over its edges, but jq can do that for DAGs encoded in JSON.

For a complete worked example, consider this tool which combines JSON and DOT. It produces diagrams that look like this image by building a DAG, packing the DAG into JSON, compiling the DAG to DOT, compiling the DOT into a PNG, and finally packing the JSON into a custom PNG chunk. This workflow itself is a DAG! The JSON is in the PNG:

$ nix build
$ result/bin/zaha json complexity/decision/hierarchy/polynomial.png 
{"labels": ["P", "coNP", "NP", "\u0394\u2082P", "\u03a3\u2082P", "\u03a0\u2082P", "\u0394\u2083P", "\u03a0\u2083P", "\u03a3\u2083P", "PH"], "structure": 28074104194051, "title": "Polynomial Hierarchy"}

And then I can use this tool to help write a book. In these build instructions, I call zaha several times to prepare some JSON, then use jq and Python to build some tables and emit some Markdown.