What if sketching on a whiteboard was a form of programming?

There are different possible notations for languages: textual and graphical are the two larger families.

We can define graphical notations for formal languages. For example we can define a graphical notation to define state machines.

A Domain Specific Language for state machines can be defined using JetBrains MPS. However the user would have to edit the state machine inside MPS.

I wonder if we could instead specify a state machine just by sketching something on a whiteboard. Consider this example:


We have a start state (the white circle on top) which leads us to the state A without needing any event. Then from state A we can go to state B when an event of type b occurs. Or we can go to state C if an event of type a occurs. From C we need a state of type c to move to D.

Note that I did not represent the direction of transitions, we could assume that they always lead from the higher state to the lower.

I think that at this point we should just make a picture to this sketch and obtain a program which would permit to run this state machine.

To do that we should recognize the different shape, the text present and assign everything to a corresponding concept in our DSL.

The first step would be to recognize rectangles. I guess there are already libraries to do that. I tried OpenCV but I really do not like the way C++ dependencies are handled. I started playing with BoofCV instead, which is written in Java. I first used functions to recognize segments and I got something like this:

Screenshot from 2016-03-28 19-12-53

Then I wrote some functions to merge segments which have endpoints very close and similar slope, to reduce the number of segments. That did not work particularly well.

Screenshot from 2016-03-28 19-13-03

I realized I should probably use some pre-processing: first a Gaussian blur and then calculate the derivatives over the X and Y axis. I got something like this. The colors depends on the values present in the derivates X and Y images. I should use those values to recognize horizontal and vertical lines with common endpoints and so get rectangles.

Screenshot from 2016-03-29 18-49-46

This is not as easy as I hoped but I would love to spend more time on it in the future.


Still, people will need some ability to formalize their thoughts but we can definitely remove some of the incidental barriers due to languages, notations and tools.

Download the guide with 68 resources on Creating Programming Languages


Receive the guide to your inbox to read it on all your devices when you have time

Powered by ConvertKit
4 replies
  1. Jordi Cabot
    Jordi Cabot says:

    Not sure about with current technologies but a few years ago I tried to do the same for UML Class diagrams and failed miserably.

    I do believe something like this would be very useful so keep us posted if you manage to get it working!

  2. Federico Tomassetti
    Federico Tomassetti says:

    First of all I am very happy to have you among my readers: I follow your blog and your work regularly.

    Perhaps time could be taken into account if we were recording a drawing as it is created.

    I have a lot of study to do in the subject. For now I will play a little longer with this idea and let you know if I come up with something.

Trackbacks & Pingbacks

  1. […] far-fetched goal I have is using sketching on a whiteboard as a way to define programs. I mean formal programs that you can execute. Of course through your sketches you would define […]

  2. […] far-fetched goal I have is using sketching on a whiteboard as a way to define programs. I mean formal programs that you can execute. Of course through your sketches you would define […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply