How to Create Pragmatic, Lightweight Languages

Why Build a Language?

In this book we will show you how to build languages and the tools to support them.

For starters, it’s a challenge and that is motivation enough for some people. Maybe that’s why you are here. And if that is the reason, I promise it’s going to be a good challenge and you are going to have fun meeting it. But there are other good reasons.

With the help of this book you will learn how to:

• develop tools for a language, like a static analysis engine for Java or a transpiler from Python to JavaScript;
• build a domain-specific language (DSL), a language really good at specific tasks, like HTML or SQL;
• create a general-purpose language (GPL), a language suitable for any task, like C# or Ruby;
• create an esoteric-programming language, a  language designed to make a point, like Piet or Shakespeare.

Creating a tool or a DSL can simplify your life as a programmer. And when your employer hears the magic words “increase productivity” it can even help in your job.

Creating an esoteric programming language can be a great way to put a new idea out there. To show that things can be done differently.

It is harder to succeed creating a GPL, but it can be done: Kotlin, Go, Ruby, Julia, Rust, etc. Just think about the many new languages created in the last decade. Many of them have taken hold in various communities and made developers happier.

A Bit of Theory and Lots of Code

When I was learning how to build languages I was disappointed: I could just find basic tutorials here and there, but what was missing was a complete course. One organized to lead me from the very beginning to the end. Whenever I found something more advanced, it was too abstract: a lot of compiler theory and not enough code.

I wanted a book with a clear plan and a pragmatic approach, but there wasn’t one.

That is when I decided to learn how to build languages on my own. It took me years of professional experience in building languages to get there. Now I can share this knowledge, so that you can learn how to create your own language in a few weeks.

It’s still going to be a lot of work, but you will have help.

How This Book Will Help You

This book shows how to reuse existing libraries and frameworks to create the necessary ecosystem to support our languages.

The cost of building the machinery to properly support a language used to be very high.

Today, with a limited effort, we can design a language and:

• obtain a fast parser from a concise grammar definition;
• obtain validation and model transformations with little code;
• generate JVM bytecode and get interoperability with other JVM languages or generate the language with the help of LLVM;
• obtain rich editors with syntax highlighting and autocompletion.

We will show you how to build all of this and more by defining small components in a smart way. The result will be something solid, complete and understandable.

The programming language used is Kotlin, but the techniques are universal and can be used with any language.

Part 0: Creating a Language

1. Motivation: why do you want to build language tools?
2. The general plan
3. The example languages we are going to build

Part I: the foundation

4. Writing a lexer
5. Writing a parser
6. Mapping: from the parse-tree to the Abstract Syntax Tree
7. Symbol resolution
8. Typesystem
9. Validation

Part II: compiling

10. Build an interpreter
11. Generate JVM bytecode
12. Generate an executable using the LLVM

Part III: editing support

13. Syntax highlighting
14. Autocomplete