Removing multiple blank lines when typesetting code listings

The listings package in LaTeX has an option to collapse multiple empty lines into a single empty line when typesetting code lists. Today, there was a question on TeX.se how to do something similar when using the minted package. Since the vim module uses the same principle as the minted package, I wondered how one could collapse multiple empty lines into a single line?

One of the fetures of the vim module is that you can source an arbitrary vimrc file before processing the code through the vim editor to generate syntax highlighted code. This feature makes it possible to delegate the task to collapsing multiple blanks lines into a single blank line to vim, the editor. Since the vim module first writes the source code in a file with extension .tmp, the following vimrc snippet will collapse all multiple blank lines into a single blank line whenever a .tmp file is loaded:

au BufEnter *.tmp %s/\(^\s*\n\)\{2,\}/\r/ge | w

Use this inside the vim module as follows (example also available on github):

\usemodule[vim]

\startvimrc[name=collapse]
au BufEnter *.tmp %s/\(^\s*\n\)\{2,\}/\r/ge | w
\stopvimrc


\definevimtyping[CPPtyping][syntax=cpp, vimrc=collapse]

\starttext
\startCPPtyping
  i++;


  i++;






  i--;
\stopCPPtyping
\stoptext

Agreed, this is not as simple as the extralines=1 option in the listings package. But, it is not too complicated when you consider the fact that I had not thought about this feature at all when I wrote the vim module.

Reading remote files

Won’t it be nice if TeX could pretty-print files hosted on github, e.g.,

\typeRUBYfile{https://raw.github.com/adityam/filter/master/Rakefile}

or include a remotely hosted markdown file in your document

\processmarkdownfile{https://raw.github.com/adityam/filter/master/README.md}

I wanted to add this feature to the filter and vim modules.
Although I knew that ConTeXt could read remote files directly, I thought that it would be hard to plug into this mechanism.

Boy, was I wrong. Look at the commit history of the change needed to add this feature.

All I needed to do, was add \locfilename to get the local file name for a file. If the requested file is a remote file (i.e., starts with http:// or ftp://), ConTeXt downloads the file and stores it in the cache directory, and return the name of the cached file. Pretty neat, eh?

With this change, \process<filter>file macro of the filter module can read remote files. Since, the vim module is built on top of the filter module, the \type<vim>file can also read remote files.

The above feature is currently only available in the dev branch. I’ll make a new release once I add hooks to force re-download of remote files. Meanwhile, if you have a ConTeXt macro that reads files, just add a \locfilename at appropriate place, and your macro will be able to read remote files

Update for the filter module: faster caching

Over the last year, the code base of the filter module has matured considerably. Now, the module has all the features that I wanted when I started with it about a year and a half back. The last remaining limitation (in my eyes, at least) was that caching of results required a call to external programs (mtxrun) to calculate md5 hashes; as such, caching was slow. That is no longer the case. Now (since early December), md5 sums are calculated at the lua end, so there is no time penalty for caching. As a result, in MkIV, recompiling is much faster for documents having lots of external filter environments with caching enabled(i.e., environments defined with continue=yes option).

[Continue Reading]

Typewriter scroll mode in vim

I came across a Mac specific application Writeroom for distraction free writing and couldn’t help wonder that good ‘ol vim provides the same features.

One interesting feature was "typewriter scroll mode", in which, the line that you are writing is always in the middle of the screen. At times, while editing long files, I find myself too often at the bottom of the screen at find myself typing zz to redraw the buffer so that the current line is at the middle of the screen. It would be nice to have that as default. A quick search of the vim help revealed:


'scrolloff' 'so'    number  (default 0)
            global
            {not in Vi}
    Minimal number of screen lines to keep above and below the cursor.
    This will make some context visible around where you are working.  If
    you set it to a very large value (999) the cursor line will always be
    in the middle of the window (except at the start or end of the file or
    when long lines wrap).
    For scrolling horizontally see 'sidescrolloff'.
    NOTE: This option is set to 0 when 'compatible' is set.

So, the solution is:

set scrolloff=99

Syntax highlighting engines: clean tex output

The vim module uses the vim editor to syntax highlight code snippets in ConTeXt. I thought that it should be straight forward to support other syntax highlighting engines: source-highlight, pygments, HsColor, etc. Unfortunately, that is not the case. None of these syntax highlighting engines were written with reuse in mind.

For example, consider a simple tex file:

\definestartstop[important]
                [color=red,
                 style=\italic]
\starttext
This is an \important{important} text
\stoptext

Lets compare the tex file generated by various syntax highlighters:

source-highlight -f latex gives

% Generator: GNU source-highlight, by Lorenzo Bettini, http://www.gnu.org/software/src-highlite
\noindent
\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}definestartstop}}\textcolor{Purple}{[important]} \\

\mbox{}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ [color=red, \\

\mbox{}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ style=\textbf{\textcolor{Blue}{\textbackslash{}italic}}] \\

\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}starttext}} \\

\mbox{}This\ is\ an\ \textbf{\textcolor{Blue}{\textbackslash{}important}}\textcolor{ForestGreen}{\{important\}}\ text \\

\mbox{}\textbf{\textcolor{Blue}{\textbackslash{}stoptext}} \\

\mbox{}

pygmentize -f latex gives

\begin{Verbatim}[commandchars=\\
\{\}]
\PY{k}{\PYZbs{}definestartstop}\PY{n+na}{[important]}
                [color=red,
                 style=\PY{k}{\PYZbs{}italic}]
\PY{k}{\PYZbs{}starttext}
This is an \PY{k}{\PYZbs{}important}\PY{n+nb}{\PYZob{}}important\PY{n+nb}{\PYZcb{}} text
\PY{k}{\PYZbs{}stoptext}
\end{Verbatim}

HsColor-latex -partial gives

\textcolor{red}{$\backslash$}{\rm{}definestartstop}\textcolor{red}{[}{\rm{}important}\textcolor{red}{]}\\
\hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \textcolor{red}{[}{\rm{}color}\textcolor{red}{=}{\rm{}red}\textcolor{cyan}{,}\\
\hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace \hsspace {\rm{}style}\textcolor{cyan}{=$\backslash$}{\rm{}italic}\textcolor{red}{]}\\
\textcolor{red}{$\backslash$}{\rm{}starttext}\\
{\rm{}This}\hsspace {\rm{}is}\hsspace {\rm{}an}\hsspace \textcolor{red}{$\backslash$}{\rm{}important}\textcolor{cyan}{\{}{\rm{}important}\textcolor{cyan}{\}}\hsspace {\rm{}text}\\
\textcolor{red}{$\backslash$}{\rm{}stoptext}\\

HsColor and source-highlight use explicit LaTeX commands for spacing and formatting. Ouch! Pygments uses logical markup, but with cryptic command names. But, from the point of view of using pygments output in ConTeXt, the \begin{Verbatim} and \end{Verbatim} are show stopper. (OK, not really. It can be bypassed with some effort).

Based on my experience, I decided to clean up the output generated by 2context.vim:

\SYN[Identifier]{\\definestartstop}[important]
                [color=red,
                 style=\SYN[Identifier]{\\italic}]
\SYN[Statement]{\\starttext}
This is an \SYN[Identifier]{\\important}\{important\} text
\SYN[Statement]{\\stoptext}

I assume only four TeX commands to be defined: \\, \{, and \} for backslash, open brace, and close brace; and \SYN[...]{...} for syntax highlighting. Thus, if anyone wants to reuse 2context in plain TeX or LaTeX, or a yet to be written future macro package, they would not need to modify the output at all. I wish the other syntax highlighting programs did the same.

Beating an old horse back to life — the t-vim module

When including a source code listing in a document, I want (colored) syntax highlighting for the code. For HTML output, a huge number of tools are available for the last. However, most of the time, I want to generate PDF output using ConTeXt. ConTeXt does support syntax highlighting for about a dozen languages, but it does not support the languages that  I use regularly — MATLAB and R. Supporting syntax highlighting for a new language is not trivial—I had a headache the last time I tried! This is where the t-vim module comes in.

The idea of the module, like a lot of ideas in ConTeXt mailing list, was germinated  by Mojca Miklavec. She said (crica Sep 2005):

I am thinking of piping the code to vim, letting vim process it, and return something like

\highlight[Conditional]{if} \highlight[Delimiter]{(} \highlight[Identifier]{!}

One could modify the 2html.vim file. Vim can already transform the highlighted code to HTML, so ConTeXt should not be so difficult. Vim already has over 400 syntax file definitions, probably equivalent to some hundred thousand lines of syntax definition in ConTexT. Well, I don’t know (yet) how to do it, but if someone on the last has more experience with vim, please feel free to contribute.

A few months later (circa Dec 2005), Nikolai Webull provided such a modification of 2html.vim and called it 2context.vim. That file was the foundation of t-vim module.

About two years later, Mojca and I pickup up on this idea and released t-vim. In the last year years, neither I nor Mojca have done any maintenance of the module, and it has started showing signs of bit rot. Every four or five months, we get a bug report that says that the module does not work. I have finally decided to revive this module.

As a first step, I have cleaned up the 2context script. The generated ConTeXt output is now much cleaner than before. For example, the following C file

#include<stdio.h>

void main()
{
    printf("Hello world & its people \n") ;
}

gets converted to

\NL{}\SYN[PreProc]{#include}\SYN[Constant]{<stdio.h>}
\NL{}
\NL{}\SYN[Type]{void}\SYN[]{ main()}
\NL{}\SYN[]{\letteropenbrace{}}
\NL{}\SYN[]{    printf(}\SYN[Constant]{"Hello world & its people }\SYN[Special]{\letterbackslash{}n}\SYN[Constant]{"}\SYN[]{) ;}
\NL{}\SYN[]{\letterclosebrace{}}

As you can see, each line starts with \NL (new line). The generated output is logical and only \, {, } are escaped. To use this output in ConTeXt, all I need to do is provide a ConTeXt macro

\SYN[syntax]{code}

that generates the correct highlighting. Providing an environment that writes its contents to an external file, runs vim on it, and reads the output may be delegated to the t-filter module (that I blogged about earlier). A proof of concept reimplementation of the t-vim module is here, but I want to write a module that will easily allow us to use any external program (like pgyments) for syntax highlighting. Once that is done, t-vim can retire respectfully.