Syntax highlighting engines: clean tex output

The vim module uses the vim editor to syntax highlight code snippets in ConTeXt. I thought that it should be straight forward to support other syntax highlighting engines: source-highlight, pygments, HsColor, etc. Unfortunately, that is not the case. None of these syntax highlighting engines were written with reuse in mind.

For example, consider a simple tex file:

\definestartstop[important]
[color=red,
style=\italic]
\starttext
This is an \important{important} text
\stoptext


Lets compare the tex file generated by various syntax highlighters:

source-highlight -f latex gives

% Generator: GNU source-highlight, by Lorenzo Bettini, http://www.gnu.org/software/src-highlite
\noindent
\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}definestartstop}}\textcolor{Purple}{[important]} \\

\mbox{}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ [color=red, \\

\mbox{}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ style=\textbf{\textcolor{Blue}{\textbackslash{}italic}}] \\

\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}starttext}} \\

\mbox{}This\ is\ an\ \textbf{\textcolor{Blue}{\textbackslash{}important}}\textcolor{ForestGreen}{\{important\}}\ text \\

\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}stoptext}} \\

\mbox{}


pygmentize -f latex gives

\begin{Verbatim}[commandchars=\\
\{\}]
\PY{k}{\PYZbs{}definestartstop}\PY{n+na}{[important]}
[color=red,
style=\PY{k}{\PYZbs{}italic}]
\PY{k}{\PYZbs{}starttext}
This is an \PY{k}{\PYZbs{}important}\PY{n+nb}{\PYZob{}}important\PY{n+nb}{\PYZcb{}} text
\PY{k}{\PYZbs{}stoptext}
\end{Verbatim}


HsColor-latex -partial gives

\textcolor{red}{$\backslash$}{\rm{}definestartstop}\textcolor{red}{[}{\rm{}important}\textcolor{red}{]}\\
\hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \textcolor{red}{[}{\rm{}color}\textcolor{red}{=}{\rm{}red}\textcolor{cyan}{,}\\
\hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace {\rm{}style}\textcolor{cyan}{=$\backslash$}{\rm{}italic}\textcolor{red}{]}\\
\textcolor{red}{$\backslash$}{\rm{}starttext}\\
{\rm{}This}\hsspace {\rm{}is}\hsspace {\rm{}an}\hsspace \textcolor{red}{$\backslash$}{\rm{}important}\textcolor{cyan}{\{}{\rm{}important}\textcolor{cyan}{\}}\hsspace {\rm{}text}\\
\textcolor{red}{$\backslash$}{\rm{}stoptext}\\


HsColor and source-highlight use explicit LaTeX commands for spacing and formatting. Ouch! Pygments uses logical markup, but with cryptic command names. But, from the point of view of using pygments output in ConTeXt, the \begin{Verbatim} and \end{Verbatim} are show stopper. (OK, not really. It can be bypassed with some effort).

Based on my experience, I decided to clean up the output generated by 2context.vim:

\SYN[Identifier]{\\definestartstop}[important]
[color=red,
style=\SYN[Identifier]{\\italic}]
\SYN[Statement]{\\starttext}
This is an \SYN[Identifier]{\\important}\{important\} text
\SYN[Statement]{\\stoptext}


I assume only four TeX commands to be defined: \\, \{, and \} for backslash, open brace, and close brace; and \SYN[...]{...} for syntax highlighting. Thus, if anyone wants to reuse 2context in plain TeX or LaTeX, or a yet to be written future macro package, they would not need to modify the output at all. I wish the other syntax highlighting programs did the same.

Programming in Maxima

Recently, in one of my papers I needed to do somewhat involved polynomial algebra. Basically, adding and substracting polynomials in p and 1-p with degree n. The calculations were relatively straight forward, but I always make silly mistakes with such calculations. So, I decided to check my analysis using a computer algebra system. Matlab and Mathematica, perhaps the most popular computer algebra softwares (although Matlab is focussed more on numerical calculations), were out of the picture because they are commercial and I did not have site licenses for them. So, I googled a bit and settled on Maxima.

To get started, I used wxMaxima, which provides a nice GUI for Maxima. wxMaxima makes learning maxima easy. It works as I expect an CAS IDE to work. After playing around the GUI for a while, I wanted to write a script to check my calculations. It turns out that writing a script is not that easy.

You can create a wxMaxima document, which is somewhat like a Mathematica notebook. The difference is that wxMaxima document is a text file, in fact a valid maxima file. Therefore, all the formatting information is stored in the comments. Here is an example:

/* [wxMaxima: comment start ]
In Proposition 1, we define a transformation $A_i$. Since we are only interested
in the symmetric arrival case, we only need to work with
[wxMaxima: comment end   ] */

/* [wxMaxima: input   start ] */
Ap(n) := 1 - (1-p)^(n+1) ;
/* [wxMaxima: input   end   ] */


The display output in wxMaxima looks really nice. But, for some unexplainable reason, I can only think logically when I am editing text using vim. All the moving up and down in the wxMaxima window makes me lose my train of thought. Writing an elaborate markup so that wxMaxima pretty prints the comments was too cumbersome, so I needed to figure out how to write a script in maxima. (On second thoughts, maybe using the wxMaxima syntax was not too bad. But then, all progress is made by the unsatisfied man!)

Now, Maxima is meant to be an interactive program. So, it echos everything that it reads and it is not easy to distinguish the desired output from the rest. After a bit of digging into the manual, I found that ending a line with $ instead of ; does not show the output. However, if you read the file using batch, the input lines are still displayed, which is a bit annoying. After more digging around in the manual, I realized that I should be loading the file using batchload—which does not display input or output lines; only the output of print and display are shown. Unfortunately, maxima does not have a command line switch to batchload a file, but that can be overcome using --batch-string flag. In short, if you are looking to use maxima as a normal programming language: 1. End your lines with $ rather than ;
2. Run the program using maxima --batch-string="batchload(\"filename\")\$"

and be aware that maxima is meant for interactive use. For some functions, it uses asksign which requires explicit user intervention. Of course, there are workarounds.

Edit: Roman pointed out another neat trick; run your script as an initialization file using --init-mac. The only trouble is that init-mac is meant for initialization files, so after


maxima -q --init-mac=filename


you end up in interactive mode. To overcome that, either add an explicit quit(); at the end of your script, or add the options --batch-string="quit();".