expand_pointing: fit a marked-up psalm or canticle to a tone

This program, expand_pointing, takes a plain text file with some markup, called `pointing', and fits the text to a psalm tone. The pointing is intended to be close to what is conventionally used in printed texts of pointed psalms. The text is usually that of a psalm or canticle.

The output is a Lilypond input file, and is intended to be processed by Lilypond for a score.


This program has only been tested on a Linux system, although it is quite likely that since it is written in Python it would work on any system with Python installed. But you'd need to copy the files by hand to sensible locations -- the install target of the Makefile is only for *nix systems.

To use the program you need Python 2.4 or later (only tested with 2.4 and 2.5; not python 3.x), and python-ply, the parser/lexer package.

Although Lilypond is not needed to run the package, the output is of little use without Lilypond. The Lilypond files generated are intended for Lilypond 2.8.7.


This should be as usual. Unpack the .tar.gz file, change to the directory thus created (expand_pointing-VERSION), and run make install as root. There is no configure script, since the program is just straightforward Python.


Before running the program, you need to create a text file of input. The format of these is described below.

For simple use, run

      expand_pointing text_file > ly_file
(the output goes to stdout by default).

Other options are:

The textual input files

These normally contain a few header lines, followed by the actual pointed text. The header lines all start with #.

The pointed text has one line for each verse, optionally preceded by a verse number terminated by a `.'. The text is used, with the markup described below.

A very small example

Here is a very short, one-verse, example of what might be in an input file:

      #tone I2
      #psalm 107
      #incipit "Confitemini Domino"
      27. They reel to and fro, and stagger like a drunk-_en man: and are _at their wits' end.

The underscores in the verse marks the start of the mediation or ending.

The heading items used in the text form of the psalms

In addition to the text of the psalm itself, the following may be used:

The markup used in the text form of the psalms

= (equals) indicates a hard hyphen, one that should always be printed no matter how the text is printed. Space after, no space before.

- (hyphen/minus) indicates a soft hyphen, one that is used to split syllables on different notes, or because of a line break. No space before or after.

~ (tilde) also indicates a soft hyphen; it is a useful alternative to - to mark hyphens added compared to the original source. For example, a source may not completely hyphenate a word that contains the start of the ending; such a word should be hyphenated manually using ~ to augment the original. No space before or after.

^ (circumflex) indicates (syllables of) two successive words which use up one note of the normal tone, but in this case the note is sung twice. This is a tie in the original. No space before or after.

-^ (hyphen circumflex) Like ^, but the two syllables are parts of the same word. Souces may not distinguish between this and ^, using a tie for both, but this program requires the distinction. No space before or after.

() (open and close parentheses) marks a non-syllable, to indicate that a note normally used is not used in this verse. Sometimes sources use an em-dash for this; this program does not support an em-dash, because it is too easy to confuse it with the different types of hyphen. Space as if () were the letters of a syllable; in the middle of a hyphenated word, ()- may be used. Note that in some cases this must be used more times than the source; for example, if two notes of the ending are omitted, this should be marked as _() () in the text.

.. (two full stops) is used as a pseudo-syllable, indicating that the previous syllable should be sung to two notes rather than the usual one, the note naturally its own and the note following. This is normally indicated by a diaresis on the (previous) syllable. Space as if .. were the letters of a syllable; in the middle of a hyphenated word, ..- may be used. If the tone leaves the reciting note in the middle of a syllable -- that is, the syllable has two notes, one the reciting note and the other the first note of the ending -- use _.. for this.

CAPITALS are used for the word(s) set to the intonation. But this does not affect the fitting of words to tone.

* (asterisk) marks the flex, a pause within a long half-verse. Space before but not after.

` (backquote) for tonus peregrinus only, in the 2nd half-verse, marks the use of the reintonation reciting note, for one syllable, after we've switched to the main reciting note. This may not work yet.

_ (underscore), before a syllable, marks the first syllable of the mediation or ending.

| (vertical line), before a syllable, marks the last syllable of the re-intonation (used only for the 2nd half-verse in tonus peregrinus).

The tonale

The program includes the tonale specified using the -x option: after this, give the name of a text file specifying the tonale. Again, the Sarum tonale is supplied as an example. This is described below.

The form of the tonale definition file

The tonale has a name, and then a series of tones, each with one or more endings.

A particular tone is contained within tone -- endtone. Items are each preceded by a keyword. The items making up a tone are:

  1. tone: The tone label itself, a string. This is used in concatenation with the ending label to identify the tone-ending combination used in the psalm .txt file.
  2. name: Its description. Not currently used.
  3. clef: The clef, c or f followed by the line on which it appears, 1--4 bottom to top.
  4. final: the final of the mode, as a note letter.
  5. notes: The notes of the tone itself, without the ending. The octave of the first note is chosen so that it is no more than a fourth from the absolute pitch specified by the middle space. Single notes may be written just as a letter. Notes are written as letters a--h, German fashion; so b for b-flat and h for b-natural. A / may follow a note intended to take a stressed syllable, and ? may follow a note optionally inserted where the words require extra notes; these are only output in the tonale produced by -T.

    Ligatures are written as a series of notes enclosed in brackets or parentheses. The program generates the correct ligature automatically; fortunately the psalm tones do not have any ambiguous cases. For example, if you write g [f e] then if these are used for two syllables then the [f e] will be converted to a clivis, but if a tie combines all three notes, then they will be converted to a climacus.

    The notes are divided by : and | into sections. The intonation is before the first |, then the reciting note, then after another | the mediation, then a : for the end-of-half-verse. After the : and before another | comes the reintonation (or second intonation), only for tonus peregrinus.

  6. ending: This introduces one ending, as described separately.

For the ending the items are:

  1. ending: The ending label itself, a string. This is used in concatenation with the tone label to identify the tone-ending combination used in the psalm .txt file.
  2. name: Its description. Not currently used.
  3. notes: As for the tone, but just a sequence of notes.
  4. gloria: The name of a file containing the gloria, written as a `psalm' of just two lines.

A # introduces a comment which is ignored.

Outstanding bugs

No attempt has been made to internationalize the program.

No divisio (perhaps a half-bar) is output at a semi-colon.

There is no support for the flex.

backquote may not yet be working properly.

The template ps has a hard-coded include mop.ily

The source location (line number and position in line) is not always correct for hyphenation errors and warnings.

It would be good if the backend were more separated, so that the Lilypond-related and Gregorio-related backends became plugins.

The Latin hyphenation dictionary is just a stub. Probably Latin (unlike English) can be hyphenated by rule, but this is not implemented.

Tonale emacs mode

A very simple emacs mode, tonale-mode, is provided to give syntax highlighting of the tonale text file. This is not yet installed -- you could install it manually in your site-lisp directory.

The remainder of the page is of little interest to anybody who just wants to use the program.

The program itself

This was my first essay in using Python; indeed, I used Python partly to teach myself the language. So don't be surprised, if you are proficient with Python, when you see clumsy constructions.

Programming: lexer/parser vs direct string manipulation

I originally wrote the program without a lexer and parser -- it seemed to me to be on the borderline between requiring simple handling, and full lexing/parsing.

The numbers of lines of code increased by about 20% overall when I changed to using the parser. The main program decreased a little in size, about 12%, but not enough to compensate for the 400 or so lines of the lexer/parser. The total number of nonblank lines without a parser was 1142, with 1382.

Having a parser made it easier to tweak details of input markup, and easier to extend if I ever think of doing so.

However, it became harder to have certain sorts of input e.g. anything which needs lexer states, like strings terminated by end-of-line (partly because python-ply's support for lexer states was incomplete). It was harder to do error reporting; syntax errors in particular are poorly handled, and it was harder to keep track of the source location. There is a messy relationship between the AST and the rest of main data structures (inheritance wasn't quite right, and doesn't match the obvious division into python modules; I used containment, but that introduced rather artificial differences between what was held in the AST structures, and what wasn't).


Since this was my first essay in Python, it prompted a few remarks on Python in general, as used for such a program.

David Stone
Last modified: Sun Apr 12 08:55:35 BST 2009