EXPERIMENTAL / DRAFT
The sample app can be found on Github here
Hi again folks!
Now, you may certainly have realized I’m Play2.1 Json API advocate. But you may also have understood that I’m not interested in Json as an end in itself. What catches my attention is that it’s a versatile arborescent data structure that can be used in web server&client, in DB such as ReactiveMongo and also when communicating between servers with WebServices.
So I keep exploring what can be done with Json (specially in the context of PlayFramework reactive architecture) and building the tools that are required to concretize my ideas.
My last article introduced JsPath Pattern Matching and I told you that I needed this tool to use it with JsZipper. It’s time to use it…
Here is why I want to do:
- Build dynamically a Json structure by aggregating data obtained by calling several external WS such as twitterAPI or github API or whatever API.
- Build this structure from a Json template stored in MongoDB in which I will find the URL and params of WebServices to call.
- Use Play2.1/WS & ReactiveMongo reactive API meaning resulting Json should be built in an asynchronous and non-blocking way.
- Use concept of JsZipper introduced in my previous article to be able to modify efficiently Play2.1/Json immutable structures.
Please note that this idea and its implementation is just an exercise of style to study the idea and introduce technical concepts but naturally it might seem a bit fake. Moreover, keep in mind, JsZipper API is still draft…
The idea of Json template
Imagine I want to gather twitter user timeline and github user profile in a single Json object.
I also would like to:
- configure the URL of WS and query parameters to fetch data
- customize the resulting Json structure
Let’s use a Json template such as:
1 2 3 4 5 6 7 8 9 10 11 12
user_id found in
__\streams\twitter, I can call twitter API to fetch the stream of tweets and the same for__\streams\github`. Finally I replace the content of each node as following:
1 2 3 4 5 6 7 8 9 10
Moreover, I’d like to store multiple templates like previous sample with multiple
user_id to be able to retrieve multiple streams at the same time.
Creating Json template in Play/ReactiveMongo (v0.9)
Recently, Stephane Godbillon has released ReactiveMongo v0.9 with corresponding Play plugin. This version really improves and eases the way you can manipulate Json directly with Play & Mongo from Scala.
Let’s store a few instance of previous templates using this API:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Hard isn’t it?
Note that I use
localhost URL because with real Twitter/Github API I would need OAuth2 tokens and this would be a pain for this sample :)
Reactive Json crafting
Now, let’s do the real job i.e the following steps:
- retrieve the template(s) from Mongo using ReactiveMongo
- call the WebServices to fetch the data using Play Async WS
- update the Json template(s) using Monadic JsZipper
The interesting technical points here are that:
- ReactiveMongo is async so we get
- Play/WS is Async so we get also
- We need to call multiple WS so we have a
We could use Play/Json transformers presented in a previous article but knowing that you have to manage Futures and multiple WS calls, it would create quite complicated code.
Here is where Monadic JsZipper becomes interesting:
JsZipperallows modifying immutable JsValue which is already cool
JsValuein the future and it’s even better!
Actually the real power of JsZipper (besides being able to modify/delete/create a node in immutable Json tree) is to transform a Json tree into a Stream of nodes that it can traverse in depth, in width or whatever you need.
Less code with WS sequential calls
Here is the code because you’ll see how easy it is:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
JsArraybecause we want to manipulate pure
.updateAllM( (JsPath, JsValue) => Future[JsValue] )is a wrapper API hiding the construction of a
JsZipperM[Future]: once built, the `JsZipperM[Future] traverses the Json tree and for each node, it calls the provided function flatMapping on Futures before going to next node. This makes the calls to WS sequential and not parallel.
case (_ \ "twitter", value): yes here is the JsPath pattern matching and imagine the crazy stuff you can do mixing Json traversal and pattern matching ;)
Asyncmeans the embedded code will return
Future[Result]but remember that it DOESN’T mean the
Actionis synchronous/blocking because in Play, everything is Asynchronous/non-blocking by default.
Then you could tell me that this is cool but the WS are not called in parallel but sequentially. Yes it’s true but imagine that it’s less than 10 lines of code and could even be reduced. Yet, here is the parallelized version…
Parallel WS calls
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
jsonTemplates.findAll( filter: (JsPath, JsValue) => Boolean )traverses the Json tree and returns a
Stream[(JsPath, JsValue)]containing the filtered nodes. This is not done with
Futurebecause we want to get all nodes now to be able to launch all WS calls in parallel.
Future.traverse(nodes)(T => Future[T])traverses the filtered values and calls all WS in parallel.
case (path@(_ \ "twitter"), value)is just JsPath pattern matching once again keeping track of full path to be able to return it with the value
path -> respfor next point.
jsonTemplates.set( (JsPath, JsValue)* )finally updates all values at given path. Note how easy it is to update multiple values at multiple paths.
A bit less elegant than the sequential case but not so much.
This sample is a bit stupid but you can see the potential of mixing those different tools together.
Alone, JsZipper and JsPath pattern matching provides very powerful ways of manipulating Json that Reads/Writes can’t do easily.
When you add reactive API on top of that, JsZipper becomes really interesting and elegant.
The sample app can be found on Github here