From some time I have been working on a book about creating languages.
Because I think that languages are extremely useful tools and by creating languages and supporting tools we have the power to change completely how people and organization works. I have seen the effect languages can have on productivity in real companies and I think that more organizations should get those benefits.
Creating a language has been regarded as a very difficult challenge. I think it does not need to be the case: with the right approach I think we can build languages with a reasonable effort.
There are two very different ways to build languages: one way is to use Language Workbenches like Jetbrains MPS or Xtext. In some cases this is the right choice: there are limitations attached to that choice but also advantages in terms of rapidity of development. The second way is to do without Language Workbenches. For this second way there is not a clear path, it is a path which requires a lot of learning, and trial and error. However it has the benefit of putting us in control: being able to shape an environment for our language exactly as we need it. The possibility of designing a simple architecture which we can completely understand, control, change. This idea is inspired by the unix philosophy of having small, simple components and combining them to obtain a complex result. I describe this path in this book.
How it looks like
This is the current table of contents:
1. Motivation: why do you want to build language machinery?
2. The general plan
Part I: the basics
3. Writing a lexer
4. Writing a parser
5. Mapping: from the parse-tree to the Abstract Syntax Tree
6. Symbol resolution
Part II: compiling
9. Writing an interprter
10. Generate JVM bytecode
11. Generate LLVM bytecode
Part III: editing support
12. Syntax highlighting
13. Auto completion
These are the chapters I will write for sure, with the first five chapters being already available. Depending on the support I will receive for this book I plan to add more chapters or leaving the table of contents as it is. I could also provide the examples in more languages (Python or C#, for example).
How you can help me
You can get the book from LeanPub. At this stage it would help me a lot if you could give me feedback on the table of contents, on what you would like to be added, on the chapters already written, on the examples provided. All of this could help me improving something better.
The book is now finished, polished and it is periodically updated. It also has a shining new and professional cover, so, it does not lack anything. You can find more about it on How to Create Pragmatic, Lightweight Languages.
For me writing this book is useful to crystallize my experience and think about how to efficiently build languages. I think that building the right languages would have a very strong impact on software development. I try to help on this with this book. Now let’s see how you readers reply to this.
Originally written in December 2016, updated on February 2017