EXPERIMENTAL / DRAFT

The module code and sample app can be found on Github here

Here we go:

0’ : Create App

> play2 new auto-persons
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/

play! 2.1.1 (using Java 1.7.0_21 and Scala 2.10.0), http://www.playframework.org

The new application will be created in /Users/pvo/zenexity/workspaces/workspace_mandubian/auto-persons

What is the application name? [auto-persons]
>

Which template do you want to use for this new application?

  1             - Create a simple Scala application
  2             - Create a simple Java application

> 1
OK, application auto-persons is created.

Have fun!

10’ : edit project/Build.scala, add `play-autosource:reactivemongo` dependency

val mandubianRepo = Seq(
  "Mandubian repository snapshots" at "https://github.com/mandubian/mandubian-mvn/raw/master/snapshots/",
  "Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
)

val appDependencies = Seq()

val main = play.Project(appName, appVersion, appDependencies).settings(
  resolvers ++= mandubianRepo,
  libraryDependencies ++= Seq(
    "play-autosource"   %% "reactivemongo"       % "0.1-SNAPSHOT",
    "org.specs2"        %% "specs2"              % "1.13"        % "test",
    "junit"              % "junit"               % "4.8"         % "test"
  )
)

30’ : Create new ReactiveMongo AutoSource Controller in app/Person.scala

package controllers

import play.api._
import play.api.mvc._

// BORING IMPORTS
// Json
import play.api.libs.json._
import play.api.libs.functional.syntax._
// Reactive JSONCollection
import play.modules.reactivemongo.json.collection.JSONCollection
// Autosource
import play.autosource.reactivemongo._
// AutoSource is Async so imports Scala Future implicits
import scala.concurrent.ExecutionContext.Implicits.global
import play.api.Play.current

// >>> THE IMPORTANT PART <<<
object Persons extends ReactiveMongoAutoSourceController[JsObject] {
  val coll = db.collection[JSONCollection]("persons")
}

50’ : Add AutoSource routes at beginning `conf/routes`

->      /person                     controllers.Persons

60’ : Create `conf/play.plugins` to initialize ReactiveMongo Plugin

400:play.modules.reactivemongo.ReactiveMongoPlugin

70’ : Append to `conf/application.conf` to initialize MongoDB connection

mongodb.uri ="mongodb://localhost:27017/persons"

80’ : Launch application

> play2 run

[info] Loading project definition from /.../auto-persons/project
[info] Set current project to auto-persons (in build file:/.../auto-persons/)

[info] Updating {file:/.../auto-persons/}auto-persons...
[info] Done updating.
--- (Running the application from SBT, auto-reloading is enabled) ---

[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Ctrl+D to stop and go back to the console...)
[info] Compiling 5 Scala sources and 1 Java source to /.../auto-persons/target/scala-2.10/classes...
[warn] there were 2 feature warnings; re-run with -feature for details
[warn] one warning found
[success] Compiled in 6s

100’ : Insert your first 2 persons with Curl

>curl -X POST -d '{ "name":"bob", "age":25 }' --header "Content-Type:application/json" http://localhost:9000/person --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 33

{"id":"51b868ef31d4002c0bac8ba4"} -> oh a BSONObjectId

>curl -X POST -d '{ "name":"john", "age":43 }' --header "Content-Type:application/json" http://localhost:9000/person --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 33

{"id":"51b868fa31d4002c0bac8ba5"}

110’ : Get all persons

>curl http://localhost:9000/person --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 118

[
  {"name":"bob","age":25.0,"id":"51b868ef31d4002c0bac8ba4"},
  {"name":"john","age":43.0,"id":"51b868fa31d4002c0bac8ba5"}
]

115’ : Delete one person

>curl -X DELETE http://localhost:9000/person/51b868ef31d4002c0bac8ba4 --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 33

{"id":"51b868ef31d4002c0bac8ba4"}

120’ : Verify person was deleted

>curl -X GET http://localhost:9000/person/51b868ef31d4002c0bac8ba4 --include

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Content-Length: 37

ID 51b868ef31d4002c0bac8ba4 not found

125’ : Update person

>curl -X PUT -d '{ "name":"john", "age":35 }' --header "Content-Type:application/json" http://localhost:9000/person/51b868fa31d4002c0bac8ba5 --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 33

{"id":"51b868fa31d4002c0bac8ba5"}

130’ : Batch insert 2 persons (johnny & tom) with more properties

>curl -X POST -d '[{ "name":"johnny", "age":15, "address":{"city":"Paris", "street":"rue quincampoix"} },{ "name":"tom", "age":3, "address":{"city":"Trifouilly", "street":"rue des accidents de poucettes"} }]' --header "Content-Type:application/json" http://localhost:9000/person/batch --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 8

{"nb":2}

135’ : Get all persons whose name begins by “john”

>curl -X POST -d '{"name":{"$regex":"^john"}}' --header "Content-Type:application/json" http://localhost:9000/person/find --include

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 175

[
  {"name":"john","age":35.0,"id":"51b868fa31d4002c0bac8ba5"},
  {"id":"51b86a1931d400bc01ac8ba8","name":"johnny","age":15.0,"address":{"city":"Paris","street":"rue quincampoix"}}
]

140’ : Delete all persons

>curl -X DELETE -d '{}' --header "Content-Type:application/json" http://localhost:9000/person/batch --include

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 7

deleted

145’ : Take 5’ rest

150’ : Done

So what was demonstrated here?

With Play-Autosource, in a few lines, you obtain :

A backed abstract datasource (here implemented for ReactiveMongo but it could be implemented for other DBs)
All CRUD operations are exposed as pure REST services
The datasource is typesafe (here JsObject but we’ll show later that we can use any type)

It can be useful to kickstart any application in which you’re going to work iteratively on our data models in direct interaction with front-end.

It could also be useful to Frontend developers who need to bootstrap frontend code with Play Framework application backend. With Autosource, they don’t have to care about modelizing strictly a datasource on server-side and can dig into their client-side code quite quickly.

Adding constraints & validation

Now you tell me: “Hey that’s stupid, you store directly JsObject but my data are structured and must be validated before inserting them”

Yes you’re right so let’s add some type constraints on our data:

object Persons extends ReactiveMongoAutoSourceController[JsObject] {
  val coll = db.collection[JSONCollection]("persons")

  // we validate the received Json as JsObject because the autosource type is JsObject
  // and we add classic validations on types
  override val reader = __.read[JsObject] keepAnd (
    (__ \ "name").read[String] and
    (__ \ "age").read[Int](Reads.min(0) keepAnd Reads.max(117))
  ).tupled
}

Try it now:

curl -X POST -d '{ "nameXXX":"bob", "age":25 }' --header "Content-Type:application/json" http://localhost:9000/person --include

HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Content-Length: 62

{"obj.name":[{"msg":"validate.error.missing-path","args":[]}]}

You can add progressively constraints on your data in a few lines. With AutoSource, you don’t need to determine immediately the exact shape of your models and you can work with JsObject directly as long as you need. Sometimes, you’ll even discover that you don’t even need a structured model and JsObject will be enough. (but I also advise to design a bit things before implementing ;))

Keep in mind that our sample is based on an implementation for ReactiveMongo so using Json is natural. For other DB, other data structure might be more idiomatic…

Use typesafe models

Now you tell me: “Funny but but but JsObject is evil because it’s not strict enough. I’m a OO developer (maybe abused by ORM gurus when I was young) and my models are case-classes…”

Yes you’re right, sometimes, you need more business logic or you want to separate concerns very strictly and your model will be shaped as case-classes.

So let’s replace our nice little JsObject by a more serious case class.

// the model
case class Person(name: String, age: Int)
object Person{
  // the famous Json Macro which generates at compile-time a Reads[Person] in a one-liner
  implicit val fmt = Json.format[Person]
}

// The autosource... shorter than before
object Persons extends ReactiveMongoAutoSourceController[Person] {
  val coll = db.collection[JSONCollection]("persons")
}

Please note that I removed the validations I had introduced before because there are not useful anymore: using Json macros, I created an implicit Format[Person] which is used implicitly by AutoSource.

So, now you can see why I consider AutoSource as a typesafe datasource.

Let’s be front-sexy with AngularJS

You all know that AngularJS is the new kid on the block and that you must use it if you want to be sexy nowadays.

I’m already sexy so I must be able to use it without understanding anything to it and that’s exactly what I’ve done: in 30mn without knowing anything about Angular (but a few concepts), I wrote a dumb CRUD front page plugged on my wonderful AutoSource.

Client DS in app/assets/javascripts/persons.js

This is the most important part of this sample: we need to call our CRUD autosource endpoints from angularJS.

We are going to use Angular resources for it even if it’s not really the best feature of AngularJS. Anyway, in a few lines, it works pretty well in my raw case.

(thanks to Paul Dijou for reviewing this code because I repeat I don’t know angularJS at all and I wrote this in 20mn without trying to understand anything :D)

var app =
  // injects ngResource
  angular.module("app", ["ngResource"])
  // creates the Person factory backed by our autosource
  // Please remark the url person/:id which will use transparently our CRUD AutoSource endpoints
  .factory('Person', ["$resource", function($resource){
    return $resource('person/:id', { "id" : "@id" });
  }])
  // creates a controller
  .controller("PersonCtrl", ["$scope", "Person", function($scope, Person) {

    $scope.createForm = {};

    // retrieves all persons
    $scope.persons = Person.query();

    // creates a person using createForm and refreshes list
    $scope.create = function() {
      var person = new Person({name: $scope.createForm.name, age: $scope.createForm.age});
      person.$save(function(){
        $scope.createForm = {};
        $scope.persons = Person.query();
      })
    }

    // removes a person and refreshes list
    $scope.remove = function(person) {
      person.$remove(function() {
        $scope.persons = Person.query();
      })
    }

    // updates a person and refreshes list
    $scope.update = function(person) {
      person.$save(function() {
        $scope.persons = Person.query();
      })
    }
}]);

CRUD UI in index.scala.html

Now let’s create our CRUD UI page using angular directives. We need to be able to:

list persons
update/delete each person
create new persons

@(message: String)

@main("Welcome to Play 2.1") {

    <div ng-controller="PersonCtrl">
      <!-- create form -->
      <label for="name">name:</label><input ng-model="createForm.name"/>
      <label for="age">age:</label><input ng-model="createForm.age" type="number"/>
      <button ng-click="create()">Create new person</button>
      <hr/>
      <!-- List of persons with update/delete buttons -->
      <table>
      <thead><th>name</th><th>age</th><td>actions</td></thead>
      <tbody ng-repeat="person in persons">
        <tr>
          <td><input ng-model="person.name"/></td>
          <td><input type="number" ng-model="person.age"/></td>
          <td><button ng-click="update(person)">Update</button><button ng-click="remove(person)">Delete</button></td>
        </tr>
      </tbody>
      </div>
    </div>

}

Import Angular in main.scala.html

We need to import angularjs in our application and create angular application using ng-app

@(title: String)(content: Html)

<!DOCTYPE html>

<!-- please note the directive ng-app to initialize angular app-->
<html ng-app="app">
    <head>
        <title>@title</title>
        <link rel="stylesheet" media="screen" href="@routes.Assets.at("stylesheets/main.css")">
        <link rel="shortcut icon" type="image/png" href="@routes.Assets.at("images/favicon.png")">
        <script src="@routes.Assets.at("javascripts/jquery-1.9.0.min.js")" type="text/javascript"></script>
        <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.1.5/angular.min.js"></script>
        <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.1.5/angular-resource.min.js"></script>

        <script src="@routes.Assets.at("javascripts/person.js")" type="text/javascript"></script>
    </head>
    <body>
        @content
    </body>
</html>

What else??? Oh yes Security…

I know what you think: “Uhuh, the poor guy who exposes his DB directly on the network and who is able to delete everything without any security”

Once again, you’re right. (yes I know I love flattery)

Autosource is by default not secured in any way and actually I don’t really care about security because this is your job to secure your exposed APIs and there are so many ways to secure services that I prefer to let you choose the one you want.

Anyway, I’m a nice boy and I’m going to show you how you could secure the DELETE endpoint using the authentication action composition sample given in Play Framework documentation.

// FAKE USER class to simulate a user extracted from DB.
case class User(name: String)
object User {
  def find(name: String) = Some(User(name))
}

object Persons extends ReactiveMongoAutoSourceController[Person] {
  // The action composite directly copied for PlayFramework doc
  def Authenticated(action: User => EssentialAction): EssentialAction = {
    // Let's define a helper function to retrieve a User
    def getUser(request: RequestHeader): Option[User] = {
      request.session.get("user").flatMap(u => User.find(u))
    }

    // Now let's define the new Action
    EssentialAction { request =>
      getUser(request).map(u => action(u)(request)).getOrElse {
        Done(Unauthorized)
      }
    }
  }

  val coll = db.collection[JSONCollection]("persons")

  // >>> IMPORTANT PART <<<
  // We simply override the delete action
  // If authenticated, we call the original action
  override def delete(id: BSONObjectID) = Authenticated { _ =>
    super.delete(id)
  }

  def index = Action {
    Ok(views.html.index("ok"))
  }

  // the login action which log any user
  def login(name: String) = Action {
    Ok("logged in").withSession("user" -> name)
  }

  // the logout action which log out any user
  def logout = Action {
    Ok("logged out").withNewSession
  }
}

Nothing to complicated here. If you need to add headers in your responses and params to querystring, it’s easy to wrap autosource actions. Please refer to Play Framework doc for more info…

I won’t try it here, the article is already too long but it should work…

Play-Autosource is DB agnostic

Play-Autosource Core is independent of the DB and provides Reactive (Async/Nonblocking) APIs to fulfill PlayFramework requirements.

Naturally this 1st implementation uses ReactiveMongo which is one of the best sample of DB reactive driver. MongoDB fits very well in this concept too because document records are really compliant to JSON datasources.

But other implementations for other DB can be done and I count on you people to contribute them.

DB implementation contributions are welcome (Play-Autosource is just Apache2 licensed) and AutoSource API are subject to evolutions if they appear to be erroneous.

Conclusion

Play-Autosource provides a very fast & lightweight way to create a REST CRUD typesafe datasource in your Play/Scala application. You can begin with blob data such as JsObject and then elaborate the model of your data progressively by adding constraints or types to it.

There would be many more things to say about Play/Autosource:

you can also override writers to change output format
you have some alpha streaming API also
etc…

There are also lots of features to improve/add because it’s still a very draft module.

If you like it and have ideas, don’t hesitate to discuss, to contribute, to improve etc…

curl -X POST -d "{ "coding" : "Have fun"} http://localhost:9000/developer

PS: Thanks to James Roper for his article about advanced routing in Play Framework which I copied shamefully XD

EXPERIMENTAL / DRAFT

The sample app can be found on Github here

Hi again folks!

Now, you may certainly have realized I’m Play2.1 Json API advocate. But you may also have understood that I’m not interested in Json as an end in itself. What catches my attention is that it’s a versatile arborescent data structure that can be used in web server&client, in DB such as ReactiveMongo and also when communicating between servers with WebServices.

So I keep exploring what can be done with Json (specially in the context of PlayFramework reactive architecture) and building the tools that are required to concretize my ideas.

My last article introduced JsPath Pattern Matching and I told you that I needed this tool to use it with JsZipper. It’s time to use it…

Here is why I want to do:

Build dynamically a Json structure by aggregating data obtained by calling several external WS such as twitterAPI or github API or whatever API.
Build this structure from a Json template stored in MongoDB in which I will find the URL and params of WebServices to call.
Use Play2.1/WS & ReactiveMongo reactive API meaning resulting Json should be built in an asynchronous and non-blocking way.
Use concept of JsZipper introduced in my previous article to be able to modify efficiently Play2.1/Json immutable structures.

Please note that this idea and its implementation is just an exercise of style to study the idea and introduce technical concepts but naturally it might seem a bit fake. Moreover, keep in mind, JsZipper API is still draft…

The idea of Json template

Imagine I want to gather twitter user timeline and github user profile in a single Json object.

I also would like to:

configure the URL of WS and query parameters to fetch data
customize the resulting Json structure

Let’s use a Json template such as:

{
  "streams" : {
    "twitter" : {
      "url" : "http://localhost:9000/twitter/statuses/user_timeline",
      "user_id" : "twitter_nick"
    },
    "github" : {
      "url" : "http://localhost:9000/github/users",
      "user_id" : "github_nick"
    }
  }
}

Using the url and user_id found in __\streams\twitter, I can call twitter API to fetch the stream of tweets and the same for__\streams\github`. Finally I replace the content of each node as following:

{
  "streams" : {
    "twitter" : {
      // TWITTER USER TIMELINE HERE
    },
    "github" : {
      // GITHUB USER PROFILE HERE
    }
  }
}

Moreover, I’d like to store multiple templates like previous sample with multiple user_id to be able to retrieve multiple streams at the same time.

Creating Json template in Play/ReactiveMongo (v0.9)

Recently, Stephane Godbillon has released ReactiveMongo v0.9 with corresponding Play plugin. This version really improves and eases the way you can manipulate Json directly with Play & Mongo from Scala.

Let’s store a few instance of previous templates using this API:

// gets my mongo collection
def coll = db.collection[JSONCollection]("templates")

def provision = Action { Async {
  val values = Enumerator(
    Json.obj(
      "streams" -> Json.obj(
        "twitter" -> Json.obj(
          "url" -> "http://localhost:9000/twitter/statuses/user_timeline",
          "user_id" -> "twitter_nick1"
        ),
        "github" -> Json.obj(
          "url" -> "http://localhost:9000/github/users",
          "user_id" -> "github_nick1"
        )
      )
    ),
    ... more templates
  )

  coll.bulkInsert(values).map{ nb =>
    Ok(Json.obj("nb"->nb))
  }

} }

Hard isn’t it?

Note that I use localhost URL because with real Twitter/Github API I would need OAuth2 tokens and this would be a pain for this sample :)

Reactive Json crafting

Now, let’s do the real job i.e the following steps:

retrieve the template(s) from Mongo using ReactiveMongo JsonCollection
call the WebServices to fetch the data using Play Async WS
update the Json template(s) using Monadic JsZipper JsZipperM[Future]

The interesting technical points here are that:

ReactiveMongo is async so we get Future[JsValue]
Play/WS is Async so we get also Future[JsValue]
We need to call multiple WS so we have a Seq[Future[JsValue]]

We could use Play/Json transformers presented in a previous article but knowing that you have to manage Futures and multiple WS calls, it would create quite complicated code.

Here is where Monadic JsZipper becomes interesting:

JsZipper allows modifying immutable JsValue which is already cool
JsZipperM[Future] allows modifying JsValue in the future and it’s even better!

Actually the real power of JsZipper (besides being able to modify/delete/create a node in immutable Json tree) is to transform a Json tree into a Stream of nodes that it can traverse in depth, in width or whatever you need.

Less code with WS sequential calls

Here is the code because you’ll see how easy it is:

// a helper to call WS
def callWSFromTemplate(value: JsValue): Future[JsValue] =
  WS.url((value \ "url").as[String])
    .withQueryString( "user_id" -> (value \ "user_id").as[String] )
    .get().map{ resp => resp.json }

// calling WS sequentially
def dataSeq = Action{
  Async{
    for{
      templates <- coll.find(Json.obj()).cursor[JsObject].toList   // retrieves templates from Mongo
      updated   <- Json.toJson(templates).updateAllM{
        case (_ \ "twitter", value) => callWSFromTemplate(value)
        case (_ \ "github", value)  => callWSFromTemplate(value)
        case (_, value)             => Future.successful(value)
      }
    } yield Ok(updated)
  }
}

Please note:

Json.toJson(templates) transforms a List[JsObject] into JsArray because we want to manipulate pure JsValue with JsZipperM[Future].
.updateAllM( (JsPath, JsValue) => Future[JsValue] ) is a wrapper API hiding the construction of a JsZipperM[Future]: once built, the `JsZipperM[Future] traverses the Json tree and for each node, it calls the provided function flatMapping on Futures before going to next node. This makes the calls to WS sequential and not parallel.
case (_ \ "twitter", value) : yes here is the JsPath pattern matching and imagine the crazy stuff you can do mixing Json traversal and pattern matching ;)
Async means the embedded code will return Future[Result] but remember that it DOESN’T mean the Action is synchronous/blocking because in Play, everything is Asynchronous/non-blocking by default.

Then you could tell me that this is cool but the WS are not called in parallel but sequentially. Yes it’s true but imagine that it’s less than 10 lines of code and could even be reduced. Yet, here is the parallelized version…

Parallel WS calls

def dataPar = Action{
  Async{
    coll.find(Json.obj()).cursor[JsObject].toList.flatMap{ templates =>
      // converts List[JsObject] into JsArray
      val jsonTemplates = Json.toJson(templates)

      // gathers all nodes that need to be updated
      val nodes = jsonTemplates.findAll{
        case (_ \ "twitter", _) | (_ \ "github", _) => true
        case (_, value) => false
      }

      // launches WS calls in parallel and updates original JsArray
      Future.traverse(nodes){
        case (path@(_ \ "twitter"), value) => callWSFromTemplate(value).map( resp => path -> resp )
        case (path@(_ \ "github"), value)  => callWSFromTemplate(value).map( resp => path -> resp )
      }.map{ pathvalues => Ok(jsonTemplates.set(pathvalues:_*)) }
    }
  }
}

Note that:

jsonTemplates.findAll( filter: (JsPath, JsValue) => Boolean ) traverses the Json tree and returns a Stream[(JsPath, JsValue)] containing the filtered nodes. This is not done with Future because we want to get all nodes now to be able to launch all WS calls in parallel.
Future.traverse(nodes)(T => Future[T]) traverses the filtered values and calls all WS in parallel.
case (path@(_ \ "twitter"), value) is just JsPath pattern matching once again keeping track of full path to be able to return it with the value path -> resp for next point.
jsonTemplates.set( (JsPath, JsValue)* ) finally updates all values at given path. Note how easy it is to update multiple values at multiple paths.

A bit less elegant than the sequential case but not so much.

Conclusion

This sample is a bit stupid but you can see the potential of mixing those different tools together.

Alone, JsZipper and JsPath pattern matching provides very powerful ways of manipulating Json that Reads/Writes can’t do easily.

When you add reactive API on top of that, JsZipper becomes really interesting and elegant.

The sample app can be found on Github here

Have JsZipperM[fun]!

EXPERIMENTAL / DRAFT

While experimenting Play21/Json Zipper in my previous article, I needed to match patterns on JsPath and decided to explore a bit this topic.

This article just presents my experimentations on JsPath pattern matching so that people interested in the topic can tell me if they like it or not and what they would add or remove. So don’t hesitate to let comments about it.

If the result is satisfying, I’ll propose it to Play team ;)

Let’s go to samples as usual.

Very simple pattern matching

`match/scale`-style

scala> __ \ "toto" match {
  case __ \ key => Some(key)
  case _ => None
}
res0: Option[String] = Some(toto)

`val`-style

scala> val _ \ toto = __ \ "toto"
toto: String = toto

Note that I don’t write val __ \ toto = __ \ "toto" (2x Underscore) as you would expect.

Why? Let’s write it:

scala> val __ \ toto = __ \ "toto"
<console>:20: error: recursive value x$1 needs type
val __ \ toto = __ \ "toto"

Actually, 1st __ is considered as a variable to be affected by Scala compiler. Then the variable __ appears on left and right side which is not good.

So I use _ to ignore its value because I know it’s __. If I absolutely wanted to match with __, you would have written:

scala> val JsPath \ toto = __ \ "toto"
toto: String = toto

Pattern matching with indexed path

scala> val (_ \ toto)@@idx = (__ \ "toto")(2)
toto: String = toto
idx: Int = 2

scala> (__ \ "toto")(2) match {
  case (__ \ "toto")@@idx => Some(idx)
  case _      => None
}
res1: Option[Int] = Some(2)

Note the usage of @@ operator that you can dislike. I didn’t find anything better for now but if anyone has a better idea, please give it to me ;)

Pattern matching the last element of a JsPath

scala> val _ \ last = __ \ "alpha" \ "beta" \ "delta" \ "gamma"
last: String = gamma

Using _, I ignore everything before gamma node.

Matching only the first element and the last one

scala> val _ \ first \?\ last = __ \ "alpha" \ "beta" \ "gamma" \ "delta"
first: String = alpha
last: String = delta

scala> val (_ \ first)@@idx \?\ last = (__ \ "alpha")(2) \ "beta" \ "gamma" \ "delta"
first: String = alpha
idx: Int = 2
last: String = delta

Note the \?\ operator which is also a temporary choice: I didn’t want to choose \\ ause \?\ operator only works in the case where you match between the first and the last element of the path and not between anything and anything…

A few more complex cases

scala> val (_ \ alpha)@@idx \ beta \ gamma \ delta = (__ \ "alpha")(2) \ "beta" \ "gamma" \ "delta"
alpha: String = alpha
idx: Int = 2
beta: String = beta
gamma: String = gamma
delta: String = delta

scala> val (_ \ alpha)@@idx \ _ \ _ \ delta = (__ \ "alpha")(2) \ "beta" \ "gamma" \ "delta"
alpha: String = alpha
idx: Int = 2
delta: String = delta

scala> val _@@idx \?\ gamma \ delta = (__ \ "alpha")(2) \ "beta" \ "gamma" \ "delta"
idx: Int = 2
gamma: String = gamma
delta: String = delta

scala> (__ \ "alpha")(2) \ "beta" \ "gamma" \ "delta" match {
  case _@@2 \ "beta" \ "gamma" \ _ => true
  case _ => false
}
res4: Boolean = true

And finally using regex?

scala> val pattern = """al(\d)*pha""".r
pattern: scala.util.matching.Regex = al(\d)*pha

scala> (__ \ "foo")(2) \ "al1234pha" \ "bar" match {
  case (__ \ "foo")@@idx \ pattern(_) \ "bar" => true
  case _ => false
}
res6: Boolean = true

So, I think we can provide more features and now I’m going to use it with my JsZipper stuff in my next article ;)

If you like it, tell it!

Have fun!

EXPERIMENTAL / DRAFT

The code is available on Github project play-json-zipper

JsZipper is a new tool allowing much more complex & powerful manipulations of Json structures for Play2/Json Scala API (not a part of Play2 core for now)

JsZipper is inspired by the Zipper concept introduced by Gérard Huet in 1997.

The Zipper allows to update immutable traversable structures in an efficient way. Json is an immutable AST so it fits well. FYI, the Zipper behaves like a loupe that walks through each node of the AST (left/right/up/down) while keeping aware of the nodes on its left, its right and its upper. The interesting idea behind the loupe is that when it targets a node, it can modify and even delete the focused node. The analogy to the pants zipper is quite good too because when it goes down the tree, it behaves as if it was opening the tree to be able to drive the loupe through all nodes and when it goes up, it closes back the tree… I won’t tell more here, it would be too long.

JsZipper is a specific interpretation of Zipper concept for Play/Json API based on :

Scala Streams to go through / update / construct Json AST nodes in a lazy way
Monadic aspects to provide funnier ways of manipulating the Json AST (plz see below)

Please note, JsZipper is not an end in itself but a tool useful to provide new API to manipulate Json.

Let’s go to samples because it explains everything.

We’ll use following Json Object.

scala> val js = Json.obj(
  "key1" -> Json.obj(
    "key11" -> "TO_FIND",
    "key12" -> 123L,
    "key13" -> JsNull
  ),
  "key2" -> 123,
  "key3" -> true,
  "key4" -> Json.arr("TO_FIND", 345.6, "test", Json.obj("key411" -> Json.obj("key4111" -> "TO_FIND")))
)
js: play.api.libs.json.JsObject = {"key1":{"key11":"TO_FIND","key12":123,"key13":null},"key2":123,"key3":true,"key4":["TO_FIND",345.6,"test",{"key411":{"key4111":"TO_FIND"}}]}

Basic manipulations

Setting multiple paths/values

scala> js.set(
  (__ \ "key4")(2) -> JsNumber(765.23),
  (__ \ "key1" \ "key12") -> JsString("toto")
)
res1: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND","key12":"toto","key13":null},"key2":123,"key3":true,"key4":["TO_FIND",345.6,765.23,{"key411":{"key4111":"TO_FIND"}}]}

Deleting multiple paths/values

scala> js.delete(
  (__ \ "key4")(2),
  (__ \ "key1" \ "key12"),
  (__ \ "key1" \ "key13")
)
res2: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND"},"key2":123,"key3":true,"key4":["TO_FIND",345.6,{"key411":{"key4111":"TO_FIND"}}]}

Finding paths/values according to a filter

scala> js.findAll( _ == JsString("TO_FIND") ).toList
res5: List[(play.api.libs.json.JsPath, play.api.libs.json.JsValue)] = List(
  (/key1/key11,"TO_FIND"),
  (/key4(0),"TO_FIND"),
  (/key4(3)/key411/key4111,"TO_FIND")
)

Updating values according to a filter based on value

scala> js.updateAll( (_:JsValue) == JsString("TO_FIND") ){ js =>
  val JsString(str) = js
  JsString(str + "2")
}
res6: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND2","key12":123,"key13":null},"key2":123,"key3":true,"key4":["TO_FIND2",345.6,"test",{"key411":{"key4111":"TO_FIND2"}}]}

Updating values according to a filter based on path+value

scala> js.updateAll{ (path, js) =>
  JsPathExtension.hasKey(path) == Some("key4111")
}{ (path, js) =>
  val JsString(str) = js
  JsString(str + path.path.last)
}
res1: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND","key12":123,"key13":null},"key2":123,"key3":true,"key4":["TO_FIND",345.6,"test",{"key411":{"key4111":"TO_FIND/key4111"}}]}

Creating an object from scratch

scala> val build = JsExtensions.buildJsObject(
  __ \ "key1" \ "key11" -> JsString("toto"),
  __ \ "key1" \ "key12" -> JsNumber(123L),
  (__ \ "key2")(0)      -> JsBoolean(true),
  __ \ "key3"           -> Json.arr(1, 2, 3)
)
build: play.api.libs.json.JsValue = {"key1":{"key11":"toto","key12":123},"key3":[1,2,3],"key2":[true]}

Let’s be funnier with Monads now

Let’s use Future as our Monad because it’s… coooool to do things in the future ;)

Imagine you call several services returning Future[JsValue] and you want to build/update a JsObject from it. Until now, if you wanted to do that with Play2/Json, it was quite tricky and required some code.

Here is what you can do now.

Updating multiple FUTURE values at given paths

scala> val maybeJs = js.setM[Future](
  (__ \ "key4")(2)        -> future{ JsNumber(765.23) },
  (__ \ "key1" \ "key12") -> future{ JsString("toto") }
)
maybeJs: scala.concurrent.Future[play.api.libs.json.JsValue] = scala.concurrent.impl.Promise$DefaultPromise@6beb722d

scala> Await.result(maybeJs, Duration("2 seconds"))
res4: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND","key12":"toto","key13":null},"key2":123,"key3":true,"key4":["TO_FIND",345.6,765.23,{"key411":{"key4111":"TO_FIND"}}]}

Update multiple FUTURE values according to a filter

scala> val maybeJs = js.updateAllM[Future]( (_:JsValue) == JsString("TO_FIND") ){ js =>
  future {
    val JsString(str) = js
    JsString(str + "2")
  }
}
maybeJs: scala.concurrent.Future[play.api.libs.json.JsValue] = scala.concurrent.impl.Promise$DefaultPromise@35a4bb1a

scala> Await.result(maybeJs, Duration("2 seconds"))
res6: play.api.libs.json.JsValue = {"key1":{"key11":"TO_FIND2","key12":123,"key13":null},"key2":123,"key3":true,"key4":["TO_FIND2",345.6,"test",{"key411":{"key4111":"TO_FIND2"}}]}

Creating a FUTURE JsArray from scratch

scala> val maybeArr = JsExtensions.buildJsArrayM[Future](
  future { JsNumber(123.45) },
  future { JsString("toto") }
)
maybeArr: scala.concurrent.Future[play.api.libs.json.JsValue] = scala.concurrent.impl.Promise$DefaultPromise@220d48e4

scala> Await.result(maybeArr, Duration("2 seconds"))
res0: play.api.libs.json.JsValue = [123.45,"toto"]

It’s still draft so it can be improved but if you like it, don’t hesitate to comment and if people like it, it could become a part of Play Framework itself

Have fun!

The question

What’s the first word coming in your mind when I say:

“Most basic concept of functional programming?”

For info, this dendrograph was pre-computed using Play2.1 app sucking Tweets & filtering/grouping the results in a very manual-o-matic way…

Have Fun(ctional)

The code is on github project shapotomic

Datomisca is a Scala API for Datomic DB

If you want to know more about Datomisca/Datomic schema go to my recent article. What’s interesting with Datomisca schema is that they are statically typed allowing some compiler validations and type inference.

Shapeless HList are heterogenous polymorphic lists

HList are able to contain different types of data and able to keep tracks of these types.

This project is an experience trying to :

convert HList to/from Datomic Entities

check HList types against schema at compile-time

This uses :

Datomisca type-safe schema
Shapeless HList
Shapeless polymorphic functions

Please note that we don’t provide any Iso[From, To] since there is no isomorphism here. Actually, there are 2 monomorphisms (injective):

HList => AddEntity to provision an entity
DEntity => HList when retrieving entity

We would need to implement Mono[From, To] certainly for our case…

Code sample

Create schema based on `HList`

// Koala Schema
object Koala {
  object ns {
    val koala = Namespace("koala")
  }

  // schema attributes
  val name        = Attribute(ns.koala / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name")
  val age         = Attribute(ns.koala / "age", SchemaType.long, Cardinality.one).withDoc("Koala's age")
  val trees       = Attribute(ns.koala / "trees", SchemaType.string, Cardinality.many).withDoc("Koala's trees")

  // the schema in HList form
  val schema = name :: age :: trees :: HNil

  // the datomic facts corresponding to schema 
  // (need specifying upper type for shapeless conversion to list)
  val txData = schema.toList[Operation]
}

// Provision schema
Datomic.transact(Koala.txData) map { tx => ... }

Validate `HList` against Schema

// creates a Temporary ID & keeps it for resolving entity after insertion
val id = DId(Partition.USER)
// creates an HList entity 
val hListEntity =
  id :: "kaylee" :: 3L ::
  Set( "manna_gum", "tallowwood" ) ::
  HNil

// validates and converts at compile-time this HList against schema
hListEntity.toAddEntity(Koala.schema)

// If you remove a field from HList and try again, the compiler fails
val badHListEntity =
  id :: "kaylee" ::
  Set( "manna_gum", "tallowwood" ) ::
  HNil

scala> badHListEntity.toAddEntity(Koala.schema)
<console>:23: error: could not find implicit value for parameter pull:
  shapotomic.SchemaCheckerFromHList.Pullback2[shapeless.::[datomisca.TempId,shapeless.::[String,shapeless.::[scala.collection.immutable.Set[String],shapeless.HNil]]],
  shapeless.::[datomisca.RawAttribute[datomisca.DString,datomisca.CardinalityOne.type],
  shapeless.::[datomisca.RawAttribute[datomisca.DLong,datomisca.CardinalityOne.type],
  shapeless.::[datomisca.RawAttribute[datomisca.DString,datomisca.CardinalityMany.type],shapeless.HNil]]],datomisca.AddEntity]

The compiler error is a bit weird at first but if you take a few seconds to read it, you’ll see that there is nothing hard about it, it just says:

scala> I can't convert
(TempId ::) String             :: Set[String]      :: HNil =>
            Attr[DString, one] :: Attr[DLong, one] :: Attr[DString, many] :: HNil

Convert `DEntity` to static-typed `HList` based on schema

val e = Datomic.resolveEntity(tx, id)

// rebuilds HList entity from DEntity statically typed by schema
val postHListEntity = e.toHList(Koala.schema)

// Explicitly typing the value to show that the compiler builds the right typed HList from schema
val validateHListEntityType: Long :: String :: Long :: Set[String] :: HNil = postHListEntity

Conclusion

Using HList with compile-time schema validation is quite interesting because it provides a very basic and versatile data structure to manipulate Datomic entities in a type-safe style.

Moreover, as Datomic pushes atomic data manipulation (simple facts instead of full entities), it’s really cool to use HList instead of rigid static structure such as case-class.

For ex:

val simplerOp = (id :: "kaylee" :: 5L).toAddEntity(Koala.name :: Koala.age :: HNil)

Have TypedFun

One more step in our progressive unveiling of Datomisca, our opensource Scala API (sponsored by Pellucid & Zenexity) trying to enhance Datomic experience for Scala developers…

After evoking queries compiled by Scala macros in previous article and then reactive transaction & fact operation API, let’s explain how Datomisca manages Datomic schema attributes.

Datomic Schema Reminders

As explained in previous articles, Datomic stores lots of atomic facts called datoms which are constituted of entity-id, attribute, value and transaction-id.

An attribute is just a namespaced keyword :<namespace>.<nested-namespace>/<name> such as:person.address/street`:

person.address is just a hierarchical namespace person -> address
street is the name of the attribute

It’s cool to provision all thoses atomic pieces of information but what if we provision non existing attribute with bad format, type, …? Is there a way to control the format of data in Datomic?

In a less strict way than SQL, Datomic provides schema facility allowing to constrain the accepted attributes and their type values.

Schema attribute definition

Datomic schema just defines the accepted attributes and some constraints on those attributes. Each schema attribute can be defined by following fields:

value type

basic types : string, long, float, bigint, bigdec, boolean, instant, uuid, uri, bytes (yes NO int).
reference : in Datomic you can reference other entities (these are lazy relations not as strict as the ones in RDBMS)

cardinality

one : one-to-one relation if you want an analogy with RDBMS
many : one-to-many relation

Please note that in Datomic, all relations are bidirectional even for one-to-many.

optional constraints:

unicity
index creation
fulltext indexation
a few more exotic ones that you’ll find in Datomic doc about schema

Schema attributes are entities

The schema validation is applied at fact insertion and allows to prevent from inserting unknown attributes or bad value types. But how are schema attributes defined?

Actually, schema attributes are themselves entities.

Remember, in previous article, I had introduced entities as being just loose aggregation of datoms just identified by the same entity ID (the first attribute of a datom).

So a schema attribute is just an entity stored in a special partition :db.part/db and defined by a few specific fields corresponding to the ones in previous paragraph. Here are the fields used to define a Datomic schema attribute technically speaking:

mandatory fields

:db/ident : specifies unique name of the attribute
:db/valueType : specifies one the previous types - Please note that even those types are not hard-coded in Datomic and in the future, adding new types could be a new feature.
:db/cardinality : specifies the cardinality one or many of the attribute - a many attribute is just a set of values and type Set is important because Datomic only manages sets of unique values as it won’t return multiple times the same value when querying.

optional fields

:db/unique
:db/doc (useful to document your schema)
:db/index
:db/fulltext
:db/isComponent
:db/noHistory

Here is an example of schema attribute declaration written in Clojure:

{:db/id #db/id[:db.part/db]
 :db/ident :person/name
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/doc "A person's name"}

As you can see, creating schema attributes just means creating new entities in the right partition. So, to add new attributes to Datomic, you just have to add new facts.

Schema sample

Let’s create a schema defining a Koala living in an eucalyptus.

Yes I’m a super-Koala fan! Don’t ask me why, this is a long story not linked at all to Australia :D… But saving Koalas is important to me so I put this little banner for them…

Let’s define a koala by following attributes:

a name String
an age Long
a sex which can be male or `female
a few eucalyptus trees in which to feed defined by:
- a species being a reference to one of the possible species of eucalyptus trees
- a row Long (let’s imagine those trees are planted in rows/columns)
- a column Long

Here is the Datomic schema for this:

[
{:db/id #db/id[:db.part/db]
 :db/ident :koala/name
 :db/valueType :db.type/string
 :db/unique :db.unique/value
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's name"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/age
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's age"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/sex
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's sex"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/eucalyptus
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/many
 :db/doc "A koala's eucalyptus trees"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/species
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus specie"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/row
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus row"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/col
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus column"}

;; koala sexes as keywords
[:db/add #db/id[:db.part/user] :db/ident :sex/male]
[:db/add #db/id[:db.part/user] :db/ident :sex/female]

;; eucalyptus species
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/manna_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/tasmanian_blue_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/swamp_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/grey_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/river_red_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/tallowwood]

]

In this sample, you can see that we have defined 4 namespaces:

koala used to logically regroup koala entity fields
eucalyptus used to logically regroup eucalyptus entity fields
sex used to identify koala sex male or female as unique keywords
eucalyptus.species to identify eucalyptus species as unique keywords

Remark also:

:koala/name field is uniquely valued meaning no koala can have the same name
:koala/eucalyptus field is a one-to-many reference to eucalyptus entities

Datomisca way of declaring schema

First of all, initialize your Datomic DB

import scala.concurrent.ExecutionContext.Implicits.global

import datomisca._
import Datomic._

val uri = "datomic:mem://koala-db"

Datomic.createDatabase(uri)
implicit val conn = Datomic.connect(uri)

The NOT-preferred way

Now, you must know it but Datomisca intensively uses Scala 2.10 macros to provide compile-time parsing and validation of Datomic queries or operations written in Clojure.

Previous Schema attributes definition is just a set of classic operations so you can ask Datomisca to parse them at compile-time as following:

val ops = Datomic.ops("""[
{:db/id #db/id[:db.part/db]
 :db/ident :koala/name
 :db/valueType :db.type/string
 :db/unique :db.unique/value
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's name"}
...
]""")

Then you can provision the schema into Datomic using:

Datomic.transact(ops) map { tx =>
  ...
  // do something
  //
}

The preferred way

Ok the previous is cool as you can validate and provision a clojure schema using Datomisca. But Datomisca provides a programmatic way of writing schema in Scala. This brings :

scala idiomatic way of manipulating schema
Type-safety to Datomic schema attributes.

Let’s see the code directly:

// Sex Schema
object SexSchema {
  // First create your namespace
  object ns {
    val sex = Namespace("sex")
  }

  // enumerated values
  val FEMALE  = AddIdent(ns.sex / "female") // :sex/female
  val MALE    = AddIdent(ns.sex / "male")   // :sex/male

  // facts representing the schema to be provisioned
  val txData = Seq(FEMALE, MALE)
}

// Eucalyptus Schema
object EucalyptusSchema {
  object ns {
    val eucalyptus  = new Namespace("eucalyptus") { // new is just here to allow structural construction
      val species   = Namespace("species")
    }
  }

  // different species
  val MANNA_GUM           = AddIdent(ns.eucalyptus.species / "manna_gum")
  val TASMANIAN_BLUE_GUM  = AddIdent(ns.eucalyptus.species / "tasmanian_blue_gum")
  val SWAMP_GUM           = AddIdent(ns.eucalyptus.species / "swamp_gum")
  val GRY_GUM             = AddIdent(ns.eucalyptus.species / "grey_gum")
  val RIVER_RED_GUM       = AddIdent(ns.eucalyptus.species / "river_red_gum")
  val TALLOWWOOD          = AddIdent(ns.eucalyptus.species / "tallowwood")

  // schema attributes
  val species  = Attribute(ns.eucalyptus / "species", SchemaType.ref, Cardinality.one).withDoc("Eucalyptus's species")
  val row      = Attribute(ns.eucalyptus / "row", SchemaType.long, Cardinality.one).withDoc("Eucalyptus's row")
  val col      = Attribute(ns.eucalyptus / "col", SchemaType.long, Cardinality.one).withDoc("Eucalyptus's column")

  // facts representing the schema to be provisioned
  val txData = Seq(
    species, row, col,
    MANNA_GUM, TASMANIAN_BLUE_GUM, SWAMP_GUM,
    GRY_GUM, RIVER_RED_GUM, TALLOWWOOD
  )
}

// Koala Schema
object KoalaSchema {
  object ns {
    val koala = Namespace("koala")
  }

  // schema attributes
  val name         = Attribute(ns.koala / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name").withUnique(Unique.value)
  val age          = Attribute(ns.koala / "age", SchemaType.long, Cardinality.one).withDoc("Koala's age")
  val sex          = Attribute(ns.koala / "sex", SchemaType.ref, Cardinality.one).withDoc("Koala's sex")
  val eucalyptus   = Attribute(ns.koala / "eucalyptus", SchemaType.ref, Cardinality.many).withDoc("Koala's trees")

  // facts representing the schema to be provisioned
  val txData = Seq(name, age, sex, eucalyptus)
}


// Provision Schema by just accumulating all txData
Datomic.transact(
  SexSchema.txData ++
  EucalyptusSchema.txData ++
  KoalaSchema.txData
) map { tx =>
  ...
}

Nothing complicated, isn’t it?

Exactly the same as writing Clojure schema but in Scala…

Datomisca type-safe schema

Datomisca takes advantage of Scala type-safety to enhance Datomic schema attribute and make them static-typed. Have a look at Datomisca Attribute definition:

sealed trait Attribute[DD <: DatomicData, Card <: Cardinality]

So an Attribute is typed by 2 parameters:

a DatomicData type
a Cardinality type

So when you define a schema attribute using Datomisca API, the compiler also infers those types.

Take this example:

val name  = Attribute(ns / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name").withUnique(Unique.value)

SchemaType.string implies this is a Attribute[DString, _]
Cardinality.one implies this is a `Attribute[_, Cardinality.one]

So name is a Attribute[DString, Cardinality.one]

In the same way:

age is Attribute[DLong, Cardinality.one]
sex is Attribute[DRef, Cardinality.one]
eucalyptus is Attribute[DRef, Cardinality.many]

As you can imagine, using this type-safe schema attributes, Datomisca can ensure consistency between the Datomic schema and the types manipulated in Scala.

Taking advantage of type-safe schema

Checking types when creating facts

Based on the typed attribute, the compiler can help us a lot to validate that we give the right type for the right attribute.

Schema facilities are extensions of basic Datomisca so you must import following to use them:

import DatomicMapping._

Here is a code sample:

//////////////////////////////////////////////////////////////////////
// correct tree with right types
scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> 5L) +
  (EucalyptusSchema.col     -> 8L)
)
tree58: datomisca.AddEntity =
{
  :eucalyptus/species :species/swamp_gum
  :eucalyptus/row 5
  :eucalyptus/col 8
  :db/id #db/id[:db.part/user -1000000]
}

//////////////////////////////////////////////////////////////////////
// incorrect tree with a string instead of a long for row
scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> "toto") +
  (EucalyptusSchema.col     -> 8L)
)
<console>:18: error: could not find implicit value for parameter attrC:
  datomisca.Attribute2PartialAddEntityWriter[datomisca.DLong,datomisca.CardinalityOne.type,String]
         (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +

In second case, compiling fails because DLong => String doesn’t exist.

In first case, it works because DLong => Long is valid.

Checking types when getting fields from Datomic entities

First of all, let’s create our first little Koala named Rose which loves feeding from 2 eucalyptus trees.

scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> 5L) +
  (EucalyptusSchema.col     -> 8L)
)
tree74: datomisca.AddEntity =
{
  :eucalyptus/species :species/swamp_gum
  :eucalyptus/row 5
  :eucalyptus/col 8
  :db/id #db/id[:db.part/user -1000002]
}

scala> val tree74 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.RIVER_RED_GUM.ref) +
  (EucalyptusSchema.row     -> 7L) +
  (EucalyptusSchema.col     -> 4L)
)
tree74: datomisca.AddEntity =
{
  :eucalyptus/species :species/river_red_gum
  :eucalyptus/row 7
  :eucalyptus/col 4
  :db/id #db/id[:db.part/user -1000004]
}

scala> val rose = SchemaEntity.add(DId(Partition.USER))(Props() +
  (KoalaSchema.name        -> "rose" ) +
  (KoalaSchema.age         -> 3L ) +
  (KoalaSchema.sex         -> SexSchema.FEMALE.ref ) +
  (KoalaSchema.eucalyptus  -> Set(DRef(tree58.id), DRef(tree74.id)) )
)
rose: datomisca.AddEntity =
{
  :koala/eucalyptus [#db/id[:db.part/user -1000001], #db/id[:db.part/user -1000002]]
  :koala/name "rose"
  :db/id #db/id[:db.part/user -1000003]
  :koala/sex :sex/female
  :koala/age 3
}

Now let’s provision those koala & trees into Datomic and retrieve real entity corresponding to our little Rose kitty.

Datomic.transact(tree58, tree74, rose) map { tx =>
  val realRose = Datomic.resolveEntity(tx, rose.id)
  ...
}

Finally let’s take advantage of typed schema attribute to access safely to fiels of the entity:

scala> val maybeRose = Datomic.transact(tree58, tree74, rose) map { tx =>
  val realRose = Datomic.resolveEntity(tx, rose.id)

  val name = realRose(KoalaSchema.name)
  val age = realRose(KoalaSchema.age)
  val sex = realRose(KoalaSchema.sex)
  val eucalyptus = realRose(KoalaSchema.eucalyptus)

  (name, age, sex, eucalyptus)
}
maybeRose: scala.concurrent.Future[(String, Long, Long, Set[Long])] = scala.concurrent.impl.Promise$DefaultPromise@49f454d6

What’s important here is that you get a (String, Long, Long, Set[Long]) which means the compiler was able to infer the right types from the Schema Attribute…

Greattt!!!

Ok that’s all for today!

Next article about an extension Datomisca provides for convenience : mapping Datomic entities to Scala structures such as case-classes or tuples. We don’t believe this is really the philosophy of Datomic in which atomic operations are much more interesting. But sometimes it’s convenient when you want to have data abstraction layer…

Have KoalaFun!

Do you like Shapeless, this great API developed by Miles Sabin studying generic/polytypic programming in Scala?

Do you like Play-json, the Play Json 2.1 Json API developed for Play 2.1 framework and now usable as stand-alone module providing functional & typesafe Json validation and Scala conversion?

Here is Shapelaysson an API interleaving Play-Json with Shapeless to be able to manipulate Json from/to Shapeless HList

HList are heterogenous polymorphic lists able to contain different types of data and able to keep tracks of these types

Shapelaysson is a Github project with test/samples

Shapelaysson takes part in my reflexions around manipulating pure data structures from/to JSON.

A few pure Json from/to HList samples

import play.api.libs.json._
import shapeless._
import HList._
import Tuples._
import shapelaysson._

// validates + converts a JsArray into HList
scala> Json.arr("foo", 123L).validate[ String :: Long :: HNil ]
res1: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] =
JsSuccess(foo :: 123 :: HNil,)

// validates + converts a JsObject into HList
scala> Json.obj("foo" -> "toto", "bar" -> 123L).validate[ String :: Long :: HNil ]
res3: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] =
JsSuccess(toto :: 123 :: HNil,)

// validates + converts imbricated JsObject into HList
scala> Json.obj(
     |   "foo" -> "toto",
     |   "foofoo" -> Json.obj("barbar1" -> 123.45, "barbar2" -> "tutu"),
     |      "bar" -> 123L,
     |      "barbar" -> Json.arr(123, true, "blabla")
     |   ).validate[ String :: (Float :: String :: HNil) :: Long :: (Int :: Boolean :: String :: HNil) :: HNil ]
res4: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[shapeless.::[Float,shapeless.::[String,shapeless.HNil]],shapeless.::[Long,shapeless.::[shapeless.::[Int,shapeless.::[Boolean,shapeless.::[String,shapeless.HNil]]],shapeless.HNil]]]]] =
JsSuccess(toto :: 123.45 :: tutu :: HNil :: 123 :: 123 :: true :: blabla :: HNil :: HNil,)

// validates with ERROR JsArray into HList
scala> Json.arr("foo", 123L).validate[ Long :: Long :: HNil ] must beEqualTo( JsError("validate.error.expected.jsnumber") )
<console>:23: error: value must is not a member of play.api.libs.json.JsResult[shapeless.::[Long,shapeless.::[Long,shapeless.HNil]]]
                    Json.arr("foo", 123L).validate[ Long :: Long :: HNil ] must beEqualTo( JsError("validate.error.expected.jsnumber") )

// converts HList to JsValue
scala> Json.toJson(123.45F :: "tutu" :: HNil)
res6: play.api.libs.json.JsValue = [123.44999694824219,"tutu"]

A few Json Reads/Writes[HList] samples

import play.api.libs.functional.syntax._

// creates a Reads[ String :: Long :: (String :: Boolean :: HNil) :: HNil]
scala> val HListReads2 = (
     |    (__ \ "foo").read[String] and
     |    (__ \ "bar").read[Long] and
     |    (__ \ "toto").read(
     |      (
     |        (__ \ "alpha").read[String] and
     |        (__ \ "beta").read[Boolean]
     |      ).tupled.hlisted
     |    )
     | ).tupled.hlisted
HListReads2: play.api.libs.json.Reads[shapeless.::[String,shapeless.::[Long,shapeless.::[shapeless.::[String,shapeless.::[Boolean,shapeless.HNil]],shapeless.HNil]]]] = play.api.libs.json.Reads$$anon$8@7e4a09ee

// validates/converts JsObject to HList
scala> Json.obj(
     |   "foo" -> "toto",
     |   "bar" -> 123L,
     |   "toto" -> Json.obj(
     |      "alpha" -> "chboing",
     |      "beta" -> true
     |   )
     | ).validate(HListReads2)
res7: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.::[shapeless.::[String,shapeless.::[Boolean,shapeless.HNil]],shapeless.HNil]]]] =
JsSuccess(toto :: 123 :: chboing :: true :: HNil :: HNil,)

// Create a Writes[String :: Long :: HNil]
scala> implicit val HListWrites: Writes[ String :: Long :: HNil ] = (
     |         (__ \ "foo").write[String] and
     |         (__ \ "bar").write[Long]
     |       ).tupled.hlisted
HListWrites: play.api.libs.json.Writes[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] = play.api.libs.json.Writes$$anon$5@7c9d07e2

// writes a HList to JsValue
scala> Json.toJson("toto" :: 123L :: HNil)
res8: play.api.libs.json.JsValue = {"foo":"toto","bar":123}

Adding shapelaysson in your dependencies

In your Build.scala, add:

import sbt._
import Keys._

object ApplicationBuild extends Build {

  val mandubianRepo = Seq(
    "Mandubian repository snapshots" at "https://github.com/mandubian/mandubian-mvn/raw/master/snapshots/",
    "Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
  )

  val sonatypeRepo = Seq(
    "Sonatype OSS Releases" at "http://oss.sonatype.org/content/repositories/releases/",
    "Sonatype OSS Snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
  )

  lazy val playJsonAlone = Project(
    BuildSettings.buildName, file("."),
    settings = BuildSettings.buildSettings ++ Seq(
      resolvers ++= mandubianRepo ++ sonatypeRepo,
      libraryDependencies ++= Seq(
        "org.mandubian"  %% "shapelaysson"  % "0.1-SNAPSHOT",
        "org.specs2"     %% "specs2"        % "1.13" % "test",
        "junit"           % "junit"         % "4.8" % "test"
      )
    )
  )
}

More to come maybe in this draft project… Suggestions are welcome too

Have Fun :: HNil!

A short article to talk about an interesting issue concerning Scala 2.10.0 Future that might interest you.

Summary

When a Fatal exception is thrown in your Future callback, it’s not caught by the Future and is thrown to the provided ExecutionContext.

But the current default Scala global ExecutionContext doesn’t register an UncaughtExceptionHandler for these fatal exceptions and your Future just hangs forever without notifying anything to anybody.

This issue is well known and a solution to the problem has already been merged into branch 2.10.x. But this issue is present in Scala 2.10.0 so it’s interesting to keep this issue in mind IMHO. Let’s explain clearly about it.

Exceptions can be contained by Future

Let’s write some stupid code with Futures.

scala> import scala.concurrent._
scala> import scala.concurrent.duration._

// Take default Scala global ExecutionContext which is a ForkJoin Thread Pool
scala> val ec = scala.concurrent.ExecutionContext.global
ec: scala.concurrent.ExecutionContextExecutor = scala.concurrent.impl.ExecutionContextImpl@15f445b7

// Create an immediately redeemed Future with a simple RuntimeException
scala> val f = future( throw new RuntimeException("foo") )(ec)
f: scala.concurrent.Future[Nothing] = scala.concurrent.impl.Promise$DefaultPromise@27380357

// Access brutally the value to show that the Future contains my RuntimeException
scala> f.value
res22: Option[scala.util.Try[Nothing]] = Some(Failure(java.lang.RuntimeException: foo))

// Use blocking await to get Future result
scala> Await.result(f, 2 seconds)
warning: there were 1 feature warnings; re-run with -feature for details
java.lang.RuntimeException: foo
  at $anonfun$1.apply(<console>:14)
  at $anonfun$1.apply(<console>:14)
  ...

You can see that a Future can contain an Exception (or more generally Throwable).

Fatal Exceptions can’t be contained by Future

If you look in Scala 2.10.0 Future.scala, in the scaladoc, you can find:

* The following throwable objects are not contained in the future:
* - `Error` - errors are not contained within futures
* - `InterruptedException` - not contained within futures
* - all `scala.util.control.ControlThrowable` except `NonLocalReturnControl` - not contained within futures

and in the code, in several places, in map or flatMap for example, you can read:

try {
...
} catch {
  case NonFatal(t) => p failure t
}

This means that every Throwable that is Fatal can’t be contained in the Future.Failure.

What’s a Fatal Throwable?

To define what’s fatal, let’s see what’s declared as non-fatal in NonFatal ScalaDoc.

* Extractor of non-fatal Throwables.  
* Will not match fatal errors like VirtualMachineError  
* (for example, OutOfMemoryError, a subclass of VirtualMachineError),  
* ThreadDeath, LinkageError, InterruptedException, ControlThrowable, or NotImplementedError. 
*
* Note that [[scala.util.control.ControlThrowable]], an internal Throwable, is not matched by
* `NonFatal` (and would therefore be thrown).

Let’s consider Fatal exceptions are just critical errors that can’t be recovered in general.

So what’s the problem?

It seems right not to catch fatal errors in the `Future, isn’t it?

But, look at following code:

// Let's throw a simple Fatal exception
scala> val f = future( throw new NotImplementedError() )(ec)
f: scala.concurrent.Future[Nothing] = scala.concurrent.impl.Promise$DefaultPromise@59747b17

scala> f.value
res0: Option[scala.util.Try[Nothing]] = None

Ok, the Future doesn’t contain the Fatal Exception as expected.

But where is my Fatal Exception if it’s not caught??? No crash, notification or whatever?

There should be an `UncaughtExceptionHandler at least notifying it!

The problem is in the default Scala `ExecutionContext`.

As explained in this issue, the exception is lost due to the implementation of the default global ExecutionContext provided in Scala.

This is a simple ForkJoin pool of threads but it has no UncaughtExceptionHandler. Have a look at code in Scala 2.10.0 ExecutionContextImpl.scala

try {
      new ForkJoinPool(
        desiredParallelism,
        threadFactory,
        null, //FIXME we should have an UncaughtExceptionHandler, see what Akka does
        true) // Async all the way baby
    } catch {
      case NonFatal(t) =>
        ...
    }

Here it’s quite clear: there is no registered `UncaughtExceptionHandler.

What’s the consequence?

Your Future hangs forever

scala> Await.result(f, 30 seconds)
warning: there were 1 feature warnings; re-run with -feature for details
java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:96)

As you can see, you can wait as long as you want, the Future is never redeemed properly, it just hangs forever and you don’t even know that a Fatal Exception has been thrown.

As explained in the issue, please note, if you use a custom ExecutionContext based on SingleThreadExecutor, this issue doesn’t appear!

scala> val es = java.util.concurrent.Executors.newSingleThreadExecutor
es: java.util.concurrent.ExecutorService = java.util.concurrent.Executors$FinalizableDelegatedExecutorService@1e336f59

scala> val ec = ExecutionContext.fromExecutorService(es)
ec: scala.concurrent.ExecutionContextExecutorService = scala.concurrent.impl.ExecutionContextImpl$$anon$1@34f43dac

scala>  val f = Future[Unit](throw new NotImplementedError())(ec)
Exception in thread "pool-1-thread-1" f: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise@7d01f935
scala.NotImplementedError: an implementation is missing
  at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:15)
  at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:15)

Conclusion

In Scala 2.10.0, if you have a Fatal Exception in a Future callback, your Future just trashes the Fatal Exception and hangs forever without notifying anything.

Hopefully, due to this already merged PR, in a future delivery of Scala 2.10.x, this problem should be corrected.

To finish, in the same old good issue, Viktor Klang also raised the question of what should be considered as fatal or not:

there’s a bigger topic at hand here, the one whether NotImplementedError, InterruptedException and ControlThrowable are to be considered fatal or not.

Meanwhile, be aware and take care ;)

Have Promise[NonFatal]!

In a very recent Pull Request, `play-json has been made a stand-alone module in Play2.2-SNAPSHOT master as play-iteratees.

It means:

You can take Play2 Scala Json API as a stand-alone library and keep using Json philosophy promoted by Play Framework anywhere.
play-json module is stand-alone in terms of dependencies but is a part & parcel of Play2.2 so it will evolve and follow Play2.x releases (and following versions) always ensuring full compatibility with play ecosystem.
play-json module has 3 ultra lightweight dependencies:
- play-functional
- play-datacommons
- play-iteratees

These are pure Scala generic pieces of code from Play framework so no Netty or whatever dependencies in it.
You can then import play-json in your project without any fear of bringing unwanted deps.

play-json will be released with future Play2.2 certainly so meanwhile, I provide:

a build published in my Maven Github repository
a sample project called play-json-alone

Even if the version is 2.2-SNAPSHOT, be aware that this is the same code as the one released in Play 2.1.0. This API has reached a good stability level. Enhancements and bug corrections will be brought to it but it’s production-ready right now.

Adding play-json 2.2-SNAPSHOT in your dependencies

In your Build.scala, add:

import sbt._
import Keys._

object ApplicationBuild extends Build {

  val mandubianRepo = Seq(
    "Mandubian repository snapshots" at "https://github.com/mandubian/mandubian-mvn/raw/master/snapshots/",
    "Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
  )

  lazy val playJsonAlone = Project(
    BuildSettings.buildName, file("."),
    settings = BuildSettings.buildSettings ++ Seq(
      resolvers ++= mandubianRepo,
      libraryDependencies ++= Seq(
        "play"        %% "play-json" % "2.2-SNAPSHOT",
        "org.specs2"  %% "specs2" % "1.13" % "test",
        "junit"        % "junit" % "4.8" % "test"
      )
    )
  )
}

Using play-json 2.2-SNAPSHOT in your code:

Just import the following and get everything from Play2.1 Json API:

import play.api.libs.json._
import play.api.libs.functional._

case class EucalyptusTree(col:Int, row: Int)

object EucalyptusTree{
  implicit val fmt = Json.format[EucalyptusTree]
}

case class Koala(name: String, home: EucalyptusTree)

object Koala{
  implicit val fmt = Json.format[Koala]
}

val kaylee = Koala("kaylee", EucalyptusTree(10, 23))

println(Json.prettyPrint(Json.toJson(kaylee)))

Json.fromJson[Koala](
  Json.obj(
    "name" -> "kaylee",
    "home" -> Json.obj(
      "col" -> 10,
      "row" -> 23
    )
)

Using play-json, you can get some bits of Play Framework pure Web philosophy.
Naturally, to unleash its full power, don’t hesitate to dive into Play Framework and discover 100% full Web Reactive Stack ;)

Thanks a lot to Play Framework team for promoting play-json as stand-alone module!
Lots of interesting features incoming soon ;)

Have fun!

EXPERIMENTAL / DRAFT

0’ : Create App

10’ : edit project/Build.scala, add play-autosource:reactivemongo dependency

30’ : Create new ReactiveMongo AutoSource Controller in app/Person.scala

50’ : Add AutoSource routes at beginning conf/routes

60’ : Create conf/play.plugins to initialize ReactiveMongo Plugin

70’ : Append to conf/application.conf to initialize MongoDB connection

80’ : Launch application

100’ : Insert your first 2 persons with Curl

110’ : Get all persons

115’ : Delete one person

120’ : Verify person was deleted

125’ : Update person

130’ : Batch insert 2 persons (johnny & tom) with more properties

135’ : Get all persons whose name begins by “john”

140’ : Delete all persons

145’ : Take 5’ rest

150’ : Done

So what was demonstrated here?

Adding constraints & validation

Use typesafe models

Let’s be front-sexy with AngularJS

Client DS in app/assets/javascripts/persons.js

CRUD UI in index.scala.html

Import Angular in main.scala.html

What else??? Oh yes Security…

Play-Autosource is DB agnostic

Conclusion

EXPERIMENTAL / DRAFT

The idea of Json template

Creating Json template in Play/ReactiveMongo (v0.9)

Reactive Json crafting

Less code with WS sequential calls

Parallel WS calls

Conclusion

EXPERIMENTAL / DRAFT

Very simple pattern matching

match/scale-style

val-style

Pattern matching with indexed path

Pattern matching the last element of a JsPath

Matching only the first element and the last one

A few more complex cases

And finally using regex?

EXPERIMENTAL / DRAFT

Basic manipulations

Setting multiple paths/values

Deleting multiple paths/values

Finding paths/values according to a filter

Updating values according to a filter based on value

Updating values according to a filter based on path+value

Creating an object from scratch

Let’s be funnier with Monads now

Updating multiple FUTURE values at given paths

Update multiple FUTURE values according to a filter

Creating a FUTURE JsArray from scratch

The question

What’s the first word coming in your mind when I say:

“Most basic concept of functional programming?”

Datomisca is a Scala API for Datomic DB

Shapeless HList are heterogenous polymorphic lists

Code sample

Create schema based on HList

Validate HList against Schema

Convert DEntity to static-typed HList based on schema

Conclusion

Datomic Schema Reminders

Schema attribute definition

value type

cardinality

optional constraints:

Schema attributes are entities

mandatory fields

optional fields

Schema sample

Datomisca way of declaring schema

First of all, initialize your Datomic DB

The NOT-preferred way

The preferred way

Datomisca type-safe schema

10’ : edit project/Build.scala, add `play-autosource:reactivemongo` dependency

50’ : Add AutoSource routes at beginning `conf/routes`

60’ : Create `conf/play.plugins` to initialize ReactiveMongo Plugin

70’ : Append to `conf/application.conf` to initialize MongoDB connection

`match/scale`-style

`val`-style

Create schema based on `HList`

Validate `HList` against Schema

Convert `DEntity` to static-typed `HList` based on schema

The problem is in the default Scala `ExecutionContext`.