Thursday, February 16, 2012

Easier Parallel Programming on Google App Engine

Google App Engine Java supports running parallel operations using its Task Queue API but it doesn't provide any reasonably simple way to create dependencies between tasks. For example running operations in fork-join manner (see explanation bellow if you aren't familiar with the concepts) is also very challenging. Our new framework AppSatori Pipes is trying to fill this gap.

Let's use the trivial yet illustrative example of work to be done: You are a farmer and you want to find the needle in the haystacks.

This is what you usually do if you are alone. You just start with the haystack one and continue to number two and if you don't succeed you just need to search in the third haystack to find the needle.
You application usually also looks like this. You just execute one task after another unless the work is done. 

Of course if there are more farmers each one can take care of particular number of different haystacks but since there is no easy way to let tasks depend on others so they just run independently. Everyone cares about his haystacks. If there is only one needle in multiple haystacks and one farmer will find it, there is no way how to tell it to the others. So they will be still trying to search the needle in empty haystacks.
This also describes what you can achieve on Google App Engine using the standard Task Queue API. You can run independent tasks but there is no easy way how to tell which tasks should be cancelled because they are no longer needed.

Aside of previous two ways of executing tasks AppSatori Pipes offers you two new options.

The first one is classic Fork-Join model when tasks knows that they should wait for their companions and send the results at once to the next task. In terms of our haystack-needle example this means that each farmer searches his own haystack but finally they will meet to check whether they found all the needles.

If there is only one needle to be found, you can just let the others stop as soon as you find it. We call this behaviour "Sprint" in AppSatori Pipes.
You are eager for some code examples? Here they are:

As you can see, there is no plubing, just a few pipes. You can find out more informations on the AppSatori Pipes GitHub site. If you are just looking for Google App Engine solution for you business feel free to contact us at

1 comment:

  1. Nevertheless It can non been that event by we as anyone may acquire entirely without charge performing tcgo rules aside applying their network pokemon tcg redeem codes