In Parts 1, 2 and 2a I made reference to such things as “environment variables” and a file called texmf.cnf. I this section of the tutorial I will try to bring these concepts together and get down to some practical details. But first we need to set the scene with a very important topic: Kpathsea.

Kpathsea, what’s that?

The name Kpathsea originates from Karl’s path searching, named after the author of the library, Karl Berry. As discussed on the Kpathsea web pages, it was written to provide a number of programs with a unified method for path-searching: finding files and resources that those programs needed during the course of execution. TeX and related programs are designed to run on very many different platforms (Windows, Linux, Mac…) and one of the keys to doing this successfully is ensure that the underlying source code from which they are built is as platform-independent as is possible to achieve.

So, in essence, Kpathsea is a path-searching library which is built into the TeX executable file (“TeX engine”) to provide unified methods for path-searching (finding files) that will work on many different computer platforms. When looking for font metric files, .tex files, .sty files, graphics and many other file resources, the “TeX engine” (e.g., LuaTeX) is using the Kpathsea library to find what it needs to typeset your document.

LuaTeX offers an alternative: a primer
For LuaTeX this is just one part of the story. With LuaTeX you can implement your own Lua code to replace the Kpathsea file searching methods via LuaTeX’s “callback” methods. That’s for another day, here we’ll stick to using Kpathsea. If you are interested to read more about the powerful LuaTeX callbacks, consult The LuaTeX Reference Manual.

Kpathsea, texmf.cnf and environment variables

So, at long last you may say :-), we’ve finally reached the point where the various components start to come together.

• texmf.cnf is the name given to the runtime configuration files used by the Kpathsea path-searching library to locate resources (find files) on your computer and tell the “TeX engine”, including LuaTeX, where the “TeX engine” can locate the file(s) it needs in order to typeset your document.
• a TeX installation may use multiple texmf.cnf files to configure the installation. The use of multiple texmf.cnf files is discussed in the main Kpathsea documention.
• In addition to using texmf.cnf files, Kpathsea also uses environment variables to find resources. In essence, Kpathsea uses a mixture of environment variables and values stored in texmf.cnf files as a way to locate resources.

Kpathsea also sets environment variables

In addition to reading environment variables from your computer, Kpathsea also
sets some environment variables too: SELFAUTOLOC, SELFAUTODIR and SELFAUTOPARENT to the location, parent and grandparent directory of the executable (LuaTeX in our case).

A simple example

If you are reading this and already have a working LuaTeX installation, the following LuaTeX example will typeset the values of SELFAUTOLOC, SELFAUTODIR and SELFAUTOPARENT from your LuaTeX configuration.

\documentclass[11pt,twoside]{article}
\begin{document}
\pagestyle{empty}
\let\temp\\%
\let\\\relax
\directlua{
         tex.print("SELFAUTOLOC = "..os.getenv("SELFAUTOLOC").."\\par")
         tex.print("SELFAUTODIR = "..os.getenv("SELFAUTODIR").."\\par") 
         tex.print("SELFAUTOPARENT = "..os.getenv("SELFAUTOPARENT").."\\par")  
}
\end{document}

I have a working LuaTeX installation but copied the luatex.exe file to a directory

c:\luatexblog\dir1\dir2\luatex.exe

Putting the above LuaTeX code into a file called test.tex produced this pdf file, showing that

SELFAUTOLOC = c:/luatexblog/dir1/dir2
SELFAUTODIR = c:/luatexblog/dir1
SELFAUTOPARENT = c:/luatexblog

I omitted an important point in the previous post which I should include before discussing installation of LuaTeX. I would like to briefly mention something called the TeX Directory Structure (TDS). Quoting from the TDS web page:

TeX is a powerful, flexible typesetting system used by many people around the world. It is extremely portable and runs on virtually all operating systems. One unfortunate side effect of TeX’s flexibility, however, is that there has been no single “right” way to install it. This has resulted in many sites having different installed arrangements.

In short, the TDS is a standard for organising a collection of TeX (i.e., and LuaTeX) files. Of course, with LuaTeX there is nothing stopping you from adopting a different directory structure but if your experiments with LuaTeX evolve into an installation which will need managing, it is certainly worth knowing about the TDS and following its recommendations.

For the purposes of these experiments and tutorials I may deviate from the TDS recommendations, but at least you know that the TDS exists, should you wish to adopt it.

Update: an interesting paper
By Michael J Downes: Managing Multiple TDS Trees.

Update: PDF version of TDS documents
Downloadable from http://tug.org/tds/

In the first part of this post we very briefly looked at what happens when you grab the raw luatex.exe binary and run it: in short, not a lot!

Of environment variables and search paths

In today’s era of graphical user interfaces: Windows, Mac OS X, Gnome, KDE and so forth, the experience of software installation, especially on Windows and Mac, is typically a point-and-click affair. Not a command line in sight. For those who are taking their first steps into the TeX world, it can, depending on your setup, be a very new experience. During the course of these tutorials, which focus on a do-it-yourself LuaTeX install, the DOS prompt and command line will become your closest allies. When writing this sort of tutorial it is always difficult to balance the desire to be thorough without boring more experienced readers with detail they find too trivial. I’ll do my best to get it right but I’ll assume very little apart from feeling comfortable with a command-line world.

Telling LuaTeX about its environment

One of the first things you need to realise is that LuaTeX needs to know something about the “computer environment” in which it is running; you need to provide it with some fundamental information so that it is able to locate the resources it needs to do its job of typesetting. This help or guidance for LuaTeX is provided through a mechanism called environment variables. In essence, environment variables are used to store values which a program can access when it is running. Windows predefines a number of standard “system environment variables”, such as PATH, but you are free to add new ones according to the needs or requirements of the different programs you use. For LuaTeX, there are a number of very important environment variables you need to set before it can find anything, such as fonts, .tex files, graphics, encoding vectors etc, on your computer.

Web2C and Kpathsea: a primer
If you want to skip past my explanations and plunge straight into the detail then head on over to http://www.tug.org/texinfohtml/kpathsea.html. A great resource which provides interesting historical background by Karl Berry.

LuaTeX environment variables

Firstly, I should say that I’m using the term “LuaTeX environment variables” which may imply that any environment variables I mention are specific to LuaTeX: generally, they are not, so please forgive this slight simplification in the interest of clarity. Occasionally there may be an environment variable which is specfic to LuaTeX and where that is the case I’ll try to point it out. In the next installment I’ll zoom in on texmf.cnf, your passport to the joys of a minimal LuaTeX installation.

The LuaTeX executable (luatex.exe) can be installed as part of mainstream TeX distributions such as TeX Live or, for Windows users, MiKTeX. However, with just a little bit of work you can create your own minimal LuaTeX setup under Windows, which is the route I chose to follow. TeX Live and MiKTeX are truly amazing pieces of work and provide extremely comprehensive TeX installations, but they are rather large. In addition, through the process of “rolling your own setup” you learn a lot of very useful things about the way that TeX looks for files and resources on your computer. I do have to admit that, initially, it was quite frustrating to “get the picture” but it soon made sense. I hope to share some of the lessons I learned, save you some time but also to provide the basic groundwork through which you can further explore the amazing LuaTeX engine.

To obtain the raw luatex.exe you can either compile the source code or download the latest beta via the LuaTeX web site. My personal preference is to compile LuaTeX from the latest source code but that requires you to install some additional software, namely MinGW and MSYS. I’m not going to cover MinGW and MSYS here because that deserves a separate post.

Getting the LuaTeX source code: a primer
Again, I’m restricting my discussions to Windows because that’s what I know. The LuaTeX source code is made publicy available from the GForge server at Supelec and can be obtained using an SVN client such as TortoiseSVN. The beauty of this process is that you can keep your local copy of the LuaTeX code fully synchronised with the master repository which is maintained by the LuaTeX developers. Every time the master codebase is modified you simply use TortoiseSVN to download the updates. Marvellous stuff!

The mysterious and magical texmf.cnf file

So, you downloaded luatex.exe, start a DOS prompt and type luatex to be presented with…

Um, OK. I’ll press enter to see what happens…

OK, I have a LaTeX file c:\test.tex

\documentclass[11pt,twoside]{article}
\begin{document}
\pagestyle{empty}
Hello Lua\TeX
\end{document}

I’ll run that, typing test.tex and I see… nothing, luatex.exe exits back to the DOS prompt. Clearly, something is not working!

What went wrong?: a primer
OK, we’re jumping forward and it is way, way too early to explain in detail here but the error is caused by the fact that we’ve not told luatex.exe anything about the world in which it is running. In ultra-simplistic terms, luatex.exe is completely unaware of its environment and needs to have some additional information, which is a combination of the mysterious and magical texmf.cnf file, together with something called ".fmt" files.


kpathsea: Running mktexfmt luatex.fmt
luatex.exe: fatal: kpathsea: CreateProcess() failed for `mktexfmt luatex.fmt' (Error 2)

Over the course of a number of tutorials I will do my best to explain the truly wacky world of texmf.cnf and the magic of .fmt files, which are compiled versions of macro packages such as plain TeX, LaTeX and so forth. Stay tuned…