The mysteries of the wide-coverage translation system

GF Summer School 2015

Thomas Hallgren

The mysteries of the wide-coverage translation system

Take a look: Wide Coverage Translation Demo

The mysteries of the wide-coverage translation system

What are the mysteries? How does it work?

The App grammar: RGL + Phrasebook + Chunking
Things that aren't handled by the grammar:
- Segmentation
- Tokenization
- Capitalization

Segmentation

Done in JavaScript
function split_punct(s) { return s.split(/([.!?]+[ \t\n]+|\n\n+|[ \t\n]*[-•*+#]+[ \t\n]+)/) }
Multiple segments can be sent to the server and translated in parallel
But it seems ad-hoc: it should be part of the grammar...

Tokenization

Done in the server
- .../App14.pgf?command=c-translate&lexer=text&unlexer=text...
Separates punctuation from words
- Parentheses and quotes?

Tokenization

Was necessary before
Nowadays, grammars could be rewritten with the BIND, SOFT_BIND and SOFT_SPACE tokens instead.

Capitalization

Done in the server, as part of the tokenization by the lexer=text
Need to change the first word of a sentence to lower case
- Causes problems if the first word is a name, or in English I, I'm, etc...
- Keep upper case if the word is all caps
- Keep upper case if it is a valid word in the grammar (lookupMorpho)
  - Misses multi-word expressions?

Capitalization

Was necessary before
Nowadays, grammars could be rewritten to use the CAPIT token instead.
- For robustness, maybe there should be a SOFT_CAPIT token too?

The GF Cloud

GF Summer School 2015

Thomas Hallgren

Where is the GF cloud

On our server
- cloud.grammaticalframework.org
On your on computer
- Download and install GF (cabal install gf -fserver)
- run gf -server
- Open it your browser

What is in the GF cloud

Minibar
Syntax editor
Simple Translation Tool
Translation Quiz
GF online editor simple multilingual grammars
Wide Coverage Translation Demo (requires App14.pgf)
Some documentation

Some other web applications

Available from www.grammaticalframework.org/demos.
Tourist Phrasebook, Mathbar (from MOLTO)
Phrasomatic
Syllogisms (JavaScript + Haskell + ...)
Multilingual Wiki
Numeral translator, Letter editor (older, Java applets)

Using GF grammars in web applications

Compiling grammars to PGF (Portable Grammar Format) files
Using the PGF web API

The Portable Grammar Format

GF compiles grammar source files into PGF files (Portable Grammat Format).
PGF files can be used with the PGF run-time library.
Example:
- gf -make FoodsEng.gf FoodsSwe.gf
- Creates Foods.pgf

PGF run-time libraries

PGF (Haskell implementation)
PGF2 (C implementation)
- Haskell binding
- Java binding
- Python binding

PGF web service API

For client-side web applications in JavaScript
For (server-side) applications in other programming languages
Operations:
- list available grammars,
- parse, linearize, translate,
- completion, random generation, ...
Access to both PGF and PGF2
- c-parse, c-linearize, c-translate, ...

PGF JavaScript library

PGF service access

pgf_online.js, support.js
- Documentation: PGF web API examples
gftranslate.js

User interface components

minibar_input.js: user input with magnets and completion
minibar_translations.js: display translations
minibar.js: complete Minibar web app
syntax-editor/editor.js: syntax tree editor
minibar_support.js, support.js: auxiliary functions
Documentation: Minibar API

Running the PGF service on your own computer

GF server mode

Standalone HTTP Server included in GF distributions (>=3.2)
gf -server This is GF version 3.7. Document root = .../lib/gf-3.7/share/www Starting HTTP server, open http://localhost:41296/ in your web browser.
Copy your .pgf files to grammars under the document root shown when starting gf -server
This gives you the same web apps and services as provided on cloud.grammaticalframework.org.

Web applications without a web server

Translating grammars to JavaScript, example:

gf -make --output-format=js FoodEng.gf FoodIta.gf
Used in
- Muste
- The Multilingual Restaurant Wiki
  - Also uses a server-side database
Limited functionality
Currently not maintained?

The End

Questions?

More info

GF book (Chapter 7: Embedded grammars and code generation)
www.grammaticalframework.org (main GF web site)
cloud.grammaticalframework.org (demo apps and web API documentation)
PGF run-time library in Haskell
PGF run-time library in C (old page)
Python binding