The code is on github project shapotomic

Datomisca is a Scala API for Datomic DB

If you want to know more about Datomisca/Datomic schema go to my recent article. What’s interesting with Datomisca schema is that they are statically typed allowing some compiler validations and type inference.

Shapeless HList are heterogenous polymorphic lists

HList are able to contain different types of data and able to keep tracks of these types.

This uses :

• Datomisca type-safe schema
• Shapeless HList
• Shapeless polymorphic functions

Please note that we don’t provide any Iso[From, To] since there is no isomorphism here. Actually, there are 2 monomorphisms (injective):

• HList => AddEntity to provision an entity
• DEntity => HList when retrieving entity

We would need to implement Mono[From, To] certainly for our case…

Code sample

Create schema based on HList

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// Koala Schema
object Koala {
  object ns {
    val koala = Namespace("koala")
  }

  // schema attributes
  val name        = Attribute(ns.koala / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name")
  val age         = Attribute(ns.koala / "age", SchemaType.long, Cardinality.one).withDoc("Koala's age")
  val trees       = Attribute(ns.koala / "trees", SchemaType.string, Cardinality.many).withDoc("Koala's trees")

  // the schema in HList form
  val schema = name :: age :: trees :: HNil

  // the datomic facts corresponding to schema 
  // (need specifying upper type for shapeless conversion to list)
  val txData = schema.toList[Operation]
}

// Provision schema
Datomic.transact(Koala.txData) map { tx => ... }

Validate HList against Schema

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// creates a Temporary ID & keeps it for resolving entity after insertion
val id = DId(Partition.USER)
// creates an HList entity 
val hListEntity =
  id :: "kaylee" :: 3L ::
  Set( "manna_gum", "tallowwood" ) ::
  HNil

// validates and converts at compile-time this HList against schema
hListEntity.toAddEntity(Koala.schema)

// If you remove a field from HList and try again, the compiler fails
val badHListEntity =
  id :: "kaylee" ::
  Set( "manna_gum", "tallowwood" ) ::
  HNil

scala> badHListEntity.toAddEntity(Koala.schema)
<console>:23: error: could not find implicit value for parameter pull:
  shapotomic.SchemaCheckerFromHList.Pullback2[shapeless.::[datomisca.TempId,shapeless.::[String,shapeless.::[scala.collection.immutable.Set[String],shapeless.HNil]]],
  shapeless.::[datomisca.RawAttribute[datomisca.DString,datomisca.CardinalityOne.type],
  shapeless.::[datomisca.RawAttribute[datomisca.DLong,datomisca.CardinalityOne.type],
  shapeless.::[datomisca.RawAttribute[datomisca.DString,datomisca.CardinalityMany.type],shapeless.HNil]]],datomisca.AddEntity]

The compiler error is a bit weird at first but if you take a few seconds to read it, you’ll see that there is nothing hard about it, it just says:

1
2
3

scala> I can't convert
(TempId ::) String             :: Set[String]      :: HNil =>
            Attr[DString, one] :: Attr[DLong, one] :: Attr[DString, many] :: HNil

Convert DEntity to static-typed HList based on schema

1
2
3
4
5
6
7

val e = Datomic.resolveEntity(tx, id)

// rebuilds HList entity from DEntity statically typed by schema
val postHListEntity = e.toHList(Koala.schema)

// Explicitly typing the value to show that the compiler builds the right typed HList from schema
val validateHListEntityType: Long :: String :: Long :: Set[String] :: HNil = postHListEntity

Conclusion

Using HList with compile-time schema validation is quite interesting because it provides a very basic and versatile data structure to manipulate Datomic entities in a type-safe style.

Moreover, as Datomic pushes atomic data manipulation (simple facts instead of full entities), it’s really cool to use HList instead of rigid static structure such as case-class.

For ex:

1

val simplerOp = (id :: "kaylee" :: 5L).toAddEntity(Koala.name :: Koala.age :: HNil)

Have TypedFun

One more step in our progressive unveiling of Datomisca, our opensource Scala API (sponsored by Pellucid & Zenexity) trying to enhance Datomic experience for Scala developers…

After evoking queries compiled by Scala macros in previous article and then reactive transaction & fact operation API, let’s explain how Datomisca manages Datomic schema attributes.

Datomic Schema Reminders

As explained in previous articles, Datomic stores lots of atomic facts called datoms which are constituted of entity-id, attribute, value and transaction-id.

An attribute is just a namespaced keyword :<namespace>.<nested-namespace>/<name> such as:person.address/street`:

• person.address is just a hierarchical namespace person -> address
• street is the name of the attribute

It’s cool to provision all thoses atomic pieces of information but what if we provision non existing attribute with bad format, type, …? Is there a way to control the format of data in Datomic?

In a less strict way than SQL, Datomic provides schema facility allowing to constrain the accepted attributes and their type values.

Schema attribute definition

Datomic schema just defines the accepted attributes and some constraints on those attributes. Each schema attribute can be defined by following fields:

value type

• basic types : string, long, float, bigint, bigdec, boolean, instant, uuid, uri, bytes (yes NO int).
• reference : in Datomic you can reference other entities (these are lazy relations not as strict as the ones in RDBMS)

cardinality

• one : one-to-one relation if you want an analogy with RDBMS
• many : one-to-many relation

Please note that in Datomic, all relations are bidirectional even for one-to-many.

optional constraints:

• unicity
• index creation
• fulltext indexation
• a few more exotic ones that you’ll find in Datomic doc about schema

Schema attributes are entities

The schema validation is applied at fact insertion and allows to prevent from inserting unknown attributes or bad value types. But how are schema attributes defined?

Actually, schema attributes are themselves entities.

Remember, in previous article, I had introduced entities as being just loose aggregation of datoms just identified by the same entity ID (the first attribute of a datom).

So a schema attribute is just an entity stored in a special partition :db.part/db and defined by a few specific fields corresponding to the ones in previous paragraph. Here are the fields used to define a Datomic schema attribute technically speaking:

mandatory fields

• :db/ident : specifies unique name of the attribute
• :db/valueType : specifies one the previous types - Please note that even those types are not hard-coded in Datomic and in the future, adding new types could be a new feature.
• :db/cardinality : specifies the cardinality one or many of the attribute - a many attribute is just a set of values and type Set is important because Datomic only manages sets of unique values as it won’t return multiple times the same value when querying.

optional fields

• :db/unique
• :db/doc (useful to document your schema)
• :db/index
• :db/fulltext
• :db/isComponent
• :db/noHistory

Here is an example of schema attribute declaration written in Clojure:

1
2
3
4
5

{:db/id #db/id[:db.part/db]
 :db/ident :person/name
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/doc "A person's name"}

As you can see, creating schema attributes just means creating new entities in the right partition. So, to add new attributes to Datomic, you just have to add new facts.

Schema sample

Let’s create a schema defining a Koala living in an eucalyptus.

Let’s define a koala by following attributes:

• a name String
• an age Long
• a sex which can be male or `female

• a few eucalyptus trees in which to feed defined by:

  • a species being a reference to one of the possible species of eucalyptus trees
  • a row Long (let’s imagine those trees are planted in rows/columns)
  • a column Long

Here is the Datomic schema for this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

[
{:db/id #db/id[:db.part/db]
 :db/ident :koala/name
 :db/valueType :db.type/string
 :db/unique :db.unique/value
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's name"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/age
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's age"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/sex
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's sex"}

{:db/id #db/id[:db.part/db]
 :db/ident :koala/eucalyptus
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/many
 :db/doc "A koala's eucalyptus trees"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/species
 :db/valueType :db.type/ref
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus specie"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/row
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus row"}

{:db/id #db/id[:db.part/db]
 :db/ident :eucalyptus/col
 :db/valueType :db.type/long
 :db/cardinality :db.cardinality/one
 :db/doc "A eucalyptus column"}

;; koala sexes as keywords
[:db/add #db/id[:db.part/user] :db/ident :sex/male]
[:db/add #db/id[:db.part/user] :db/ident :sex/female]

;; eucalyptus species
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/manna_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/tasmanian_blue_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/swamp_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/grey_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/river_red_gum]
[:db/add #db/id[:db.part/user] :db/ident :eucalyptus.species/tallowwood]

]

In this sample, you can see that we have defined 4 namespaces:

• koala used to logically regroup koala entity fields
• eucalyptus used to logically regroup eucalyptus entity fields
• sex used to identify koala sex male or female as unique keywords
• eucalyptus.species to identify eucalyptus species as unique keywords

Remark also:

• :koala/name field is uniquely valued meaning no koala can have the same name
• :koala/eucalyptus field is a one-to-many reference to eucalyptus entities

Datomisca way of declaring schema

First of all, initialize your Datomic DB

1
2
3
4
5
6
7
8
9

import scala.concurrent.ExecutionContext.Implicits.global

import datomisca._
import Datomic._

val uri = "datomic:mem://koala-db"

Datomic.createDatabase(uri)
implicit val conn = Datomic.connect(uri)

The NOT-preferred way

Now, you must know it but Datomisca intensively uses Scala 2.10 macros to provide compile-time parsing and validation of Datomic queries or operations written in Clojure.

Previous Schema attributes definition is just a set of classic operations so you can ask Datomisca to parse them at compile-time as following:

1
2
3
4
5
6
7
8
9

val ops = Datomic.ops("""[
{:db/id #db/id[:db.part/db]
 :db/ident :koala/name
 :db/valueType :db.type/string
 :db/unique :db.unique/value
 :db/cardinality :db.cardinality/one
 :db/doc "A koala's name"}
...
]""")

Then you can provision the schema into Datomic using:

1
2
3
4
5

Datomic.transact(ops) map { tx =>
  ...
  // do something
  //
}

The preferred way

Ok the previous is cool as you can validate and provision a clojure schema using Datomisca. But Datomisca provides a programmatic way of writing schema in Scala. This brings :

• scala idiomatic way of manipulating schema
• Type-safety to Datomic schema attributes.

Let’s see the code directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

// Sex Schema
object SexSchema {
  // First create your namespace
  object ns {
    val sex = Namespace("sex")
  }

  // enumerated values
  val FEMALE  = AddIdent(ns.sex / "female") // :sex/female
  val MALE    = AddIdent(ns.sex / "male")   // :sex/male

  // facts representing the schema to be provisioned
  val txData = Seq(FEMALE, MALE)
}

// Eucalyptus Schema
object EucalyptusSchema {
  object ns {
    val eucalyptus  = new Namespace("eucalyptus") { // new is just here to allow structural construction
      val species   = Namespace("species")
    }
  }

  // different species
  val MANNA_GUM           = AddIdent(ns.eucalyptus.species / "manna_gum")
  val TASMANIAN_BLUE_GUM  = AddIdent(ns.eucalyptus.species / "tasmanian_blue_gum")
  val SWAMP_GUM           = AddIdent(ns.eucalyptus.species / "swamp_gum")
  val GRY_GUM             = AddIdent(ns.eucalyptus.species / "grey_gum")
  val RIVER_RED_GUM       = AddIdent(ns.eucalyptus.species / "river_red_gum")
  val TALLOWWOOD          = AddIdent(ns.eucalyptus.species / "tallowwood")

  // schema attributes
  val species  = Attribute(ns.eucalyptus / "species", SchemaType.ref, Cardinality.one).withDoc("Eucalyptus's species")
  val row      = Attribute(ns.eucalyptus / "row", SchemaType.long, Cardinality.one).withDoc("Eucalyptus's row")
  val col      = Attribute(ns.eucalyptus / "col", SchemaType.long, Cardinality.one).withDoc("Eucalyptus's column")

  // facts representing the schema to be provisioned
  val txData = Seq(
    species, row, col,
    MANNA_GUM, TASMANIAN_BLUE_GUM, SWAMP_GUM,
    GRY_GUM, RIVER_RED_GUM, TALLOWWOOD
  )
}

// Koala Schema
object KoalaSchema {
  object ns {
    val koala = Namespace("koala")
  }

  // schema attributes
  val name         = Attribute(ns.koala / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name").withUnique(Unique.value)
  val age          = Attribute(ns.koala / "age", SchemaType.long, Cardinality.one).withDoc("Koala's age")
  val sex          = Attribute(ns.koala / "sex", SchemaType.ref, Cardinality.one).withDoc("Koala's sex")
  val eucalyptus   = Attribute(ns.koala / "eucalyptus", SchemaType.ref, Cardinality.many).withDoc("Koala's trees")

  // facts representing the schema to be provisioned
  val txData = Seq(name, age, sex, eucalyptus)
}


// Provision Schema by just accumulating all txData
Datomic.transact(
  SexSchema.txData ++
  EucalyptusSchema.txData ++
  KoalaSchema.txData
) map { tx =>
  ...
}

Nothing complicated, isn’t it?

Exactly the same as writing Clojure schema but in Scala…

Datomisca type-safe schema

Datomisca takes advantage of Scala type-safety to enhance Datomic schema attribute and make them static-typed. Have a look at Datomisca Attribute definition:

1

sealed trait Attribute[DD <: DatomicData, Card <: Cardinality]

So an Attribute is typed by 2 parameters:

• a DatomicData type
• a Cardinality type

So when you define a schema attribute using Datomisca API, the compiler also infers those types.

Take this example:

1

val name  = Attribute(ns / "name", SchemaType.string, Cardinality.one).withDoc("Koala's name").withUnique(Unique.value)

• SchemaType.string implies this is a Attribute[DString, _]
• Cardinality.one implies this is a `Attribute[_, Cardinality.one]

So name is a Attribute[DString, Cardinality.one]

In the same way:

• age is Attribute[DLong, Cardinality.one]
• sex is Attribute[DRef, Cardinality.one]
• eucalyptus is Attribute[DRef, Cardinality.many]

As you can imagine, using this type-safe schema attributes, Datomisca can ensure consistency between the Datomic schema and the types manipulated in Scala.

Taking advantage of type-safe schema

Checking types when creating facts

Based on the typed attribute, the compiler can help us a lot to validate that we give the right type for the right attribute.

Schema facilities are extensions of basic Datomisca so you must import following to use them:

1

import DatomicMapping._

Here is a code sample:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

//////////////////////////////////////////////////////////////////////
// correct tree with right types
scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> 5L) +
  (EucalyptusSchema.col     -> 8L)
)
tree58: datomisca.AddEntity =
{
  :eucalyptus/species :species/swamp_gum
  :eucalyptus/row 5
  :eucalyptus/col 8
  :db/id #db/id[:db.part/user -1000000]
}

//////////////////////////////////////////////////////////////////////
// incorrect tree with a string instead of a long for row
scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> "toto") +
  (EucalyptusSchema.col     -> 8L)
)
<console>:18: error: could not find implicit value for parameter attrC:
  datomisca.Attribute2PartialAddEntityWriter[datomisca.DLong,datomisca.CardinalityOne.type,String]
         (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +

In second case, compiling fails because DLong => String doesn’t exist.

In first case, it works because DLong => Long is valid.

Checking types when getting fields from Datomic entities

First of all, let’s create our first little Koala named Rose which loves feeding from 2 eucalyptus trees.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

scala> val tree58 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.SWAMP_GUM.ref) +
  (EucalyptusSchema.row     -> 5L) +
  (EucalyptusSchema.col     -> 8L)
)
tree74: datomisca.AddEntity =
{
  :eucalyptus/species :species/swamp_gum
  :eucalyptus/row 5
  :eucalyptus/col 8
  :db/id #db/id[:db.part/user -1000002]
}

scala> val tree74 = SchemaEntity.add(DId(Partition.USER))(Props() +
  (EucalyptusSchema.species -> EucalyptusSchema.RIVER_RED_GUM.ref) +
  (EucalyptusSchema.row     -> 7L) +
  (EucalyptusSchema.col     -> 4L)
)
tree74: datomisca.AddEntity =
{
  :eucalyptus/species :species/river_red_gum
  :eucalyptus/row 7
  :eucalyptus/col 4
  :db/id #db/id[:db.part/user -1000004]
}

scala> val rose = SchemaEntity.add(DId(Partition.USER))(Props() +
  (KoalaSchema.name        -> "rose" ) +
  (KoalaSchema.age         -> 3L ) +
  (KoalaSchema.sex         -> SexSchema.FEMALE.ref ) +
  (KoalaSchema.eucalyptus  -> Set(DRef(tree58.id), DRef(tree74.id)) )
)
rose: datomisca.AddEntity =
{
  :koala/eucalyptus [#db/id[:db.part/user -1000001], #db/id[:db.part/user -1000002]]
  :koala/name "rose"
  :db/id #db/id[:db.part/user -1000003]
  :koala/sex :sex/female
  :koala/age 3
}

Now let’s provision those koala & trees into Datomic and retrieve real entity corresponding to our little Rose kitty.

1
2
3
4

Datomic.transact(tree58, tree74, rose) map { tx =>
  val realRose = Datomic.resolveEntity(tx, rose.id)
  ...
}

Finally let’s take advantage of typed schema attribute to access safely to fiels of the entity:

1
2
3
4
5
6
7
8
9
10
11

scala> val maybeRose = Datomic.transact(tree58, tree74, rose) map { tx =>
  val realRose = Datomic.resolveEntity(tx, rose.id)

  val name = realRose(KoalaSchema.name)
  val age = realRose(KoalaSchema.age)
  val sex = realRose(KoalaSchema.sex)
  val eucalyptus = realRose(KoalaSchema.eucalyptus)

  (name, age, sex, eucalyptus)
}
maybeRose: scala.concurrent.Future[(String, Long, Long, Set[Long])] = scala.concurrent.impl.Promise$DefaultPromise@49f454d6

What’s important here is that you get a (String, Long, Long, Set[Long]) which means the compiler was able to infer the right types from the Schema Attribute…

Greattt!!!

Ok that’s all for today!

Next article about an extension Datomisca provides for convenience : mapping Datomic entities to Scala structures such as case-classes or tuples. We don’t believe this is really the philosophy of Datomic in which atomic operations are much more interesting. But sometimes it’s convenient when you want to have data abstraction layer…

Have KoalaFun!

Do you like Shapeless, this great API developed by Miles Sabin studying generic/polytypic programming in Scala?

Do you like Play-json, the Play Json 2.1 Json API developed for Play 2.1 framework and now usable as stand-alone module providing functional & typesafe Json validation and Scala conversion?

Here is Shapelaysson an API interleaving Play-Json with Shapeless to be able to manipulate Json from/to Shapeless HList

HList are heterogenous polymorphic lists able to contain different types of data and able to keep tracks of these types

Shapelaysson is a Github project with test/samples

Shapelaysson takes part in my reflexions around manipulating pure data structures from/to JSON.

A few pure Json from/to HList samples

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

import play.api.libs.json._
import shapeless._
import HList._
import Tuples._
import shapelaysson._

// validates + converts a JsArray into HList
scala> Json.arr("foo", 123L).validate[ String :: Long :: HNil ]
res1: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] =
JsSuccess(foo :: 123 :: HNil,)

// validates + converts a JsObject into HList
scala> Json.obj("foo" -> "toto", "bar" -> 123L).validate[ String :: Long :: HNil ]
res3: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] =
JsSuccess(toto :: 123 :: HNil,)

// validates + converts imbricated JsObject into HList
scala> Json.obj(
     |   "foo" -> "toto",
     |   "foofoo" -> Json.obj("barbar1" -> 123.45, "barbar2" -> "tutu"),
     |      "bar" -> 123L,
     |      "barbar" -> Json.arr(123, true, "blabla")
     |   ).validate[ String :: (Float :: String :: HNil) :: Long :: (Int :: Boolean :: String :: HNil) :: HNil ]
res4: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[shapeless.::[Float,shapeless.::[String,shapeless.HNil]],shapeless.::[Long,shapeless.::[shapeless.::[Int,shapeless.::[Boolean,shapeless.::[String,shapeless.HNil]]],shapeless.HNil]]]]] =
JsSuccess(toto :: 123.45 :: tutu :: HNil :: 123 :: 123 :: true :: blabla :: HNil :: HNil,)

// validates with ERROR JsArray into HList
scala> Json.arr("foo", 123L).validate[ Long :: Long :: HNil ] must beEqualTo( JsError("validate.error.expected.jsnumber") )
<console>:23: error: value must is not a member of play.api.libs.json.JsResult[shapeless.::[Long,shapeless.::[Long,shapeless.HNil]]]
                    Json.arr("foo", 123L).validate[ Long :: Long :: HNil ] must beEqualTo( JsError("validate.error.expected.jsnumber") )

// converts HList to JsValue
scala> Json.toJson(123.45F :: "tutu" :: HNil)
res6: play.api.libs.json.JsValue = [123.44999694824219,"tutu"]

A few Json Reads/Writes[HList] samples

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import play.api.libs.functional.syntax._

// creates a Reads[ String :: Long :: (String :: Boolean :: HNil) :: HNil]
scala> val HListReads2 = (
     |    (__ \ "foo").read[String] and
     |    (__ \ "bar").read[Long] and
     |    (__ \ "toto").read(
     |      (
     |        (__ \ "alpha").read[String] and
     |        (__ \ "beta").read[Boolean]
     |      ).tupled.hlisted
     |    )
     | ).tupled.hlisted
HListReads2: play.api.libs.json.Reads[shapeless.::[String,shapeless.::[Long,shapeless.::[shapeless.::[String,shapeless.::[Boolean,shapeless.HNil]],shapeless.HNil]]]] = play.api.libs.json.Reads$$anon$8@7e4a09ee

// validates/converts JsObject to HList
scala> Json.obj(
     |   "foo" -> "toto",
     |   "bar" -> 123L,
     |   "toto" -> Json.obj(
     |      "alpha" -> "chboing",
     |      "beta" -> true
     |   )
     | ).validate(HListReads2)
res7: play.api.libs.json.JsResult[shapeless.::[String,shapeless.::[Long,shapeless.::[shapeless.::[String,shapeless.::[Boolean,shapeless.HNil]],shapeless.HNil]]]] =
JsSuccess(toto :: 123 :: chboing :: true :: HNil :: HNil,)

// Create a Writes[String :: Long :: HNil]
scala> implicit val HListWrites: Writes[ String :: Long :: HNil ] = (
     |         (__ \ "foo").write[String] and
     |         (__ \ "bar").write[Long]
     |       ).tupled.hlisted
HListWrites: play.api.libs.json.Writes[shapeless.::[String,shapeless.::[Long,shapeless.HNil]]] = play.api.libs.json.Writes$$anon$5@7c9d07e2

// writes a HList to JsValue
scala> Json.toJson("toto" :: 123L :: HNil)
res8: play.api.libs.json.JsValue = {"foo":"toto","bar":123}

Adding shapelaysson in your dependencies

In your Build.scala, add:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import sbt._
import Keys._

object ApplicationBuild extends Build {

  val mandubianRepo = Seq(
    "Mandubian repository snapshots" at "https://github.com/mandubian/mandubian-mvn/raw/master/snapshots/",
    "Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
  )

  val sonatypeRepo = Seq(
    "Sonatype OSS Releases" at "http://oss.sonatype.org/content/repositories/releases/",
    "Sonatype OSS Snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
  )

  lazy val playJsonAlone = Project(
    BuildSettings.buildName, file("."),
    settings = BuildSettings.buildSettings ++ Seq(
      resolvers ++= mandubianRepo ++ sonatypeRepo,
      libraryDependencies ++= Seq(
        "org.mandubian"  %% "shapelaysson"  % "0.1-SNAPSHOT",
        "org.specs2"     %% "specs2"        % "1.13" % "test",
        "junit"           % "junit"         % "4.8" % "test"
      )
    )
  )
}

More to come maybe in this draft project… Suggestions are welcome too

Have Fun :: HNil!

A short article to talk about an interesting issue concerning Scala 2.10.0 Future that might interest you.

This issue is well known and a solution to the problem has already been merged into branch 2.10.x. But this issue is present in Scala 2.10.0 so it’s interesting to keep this issue in mind IMHO. Let’s explain clearly about it.

Exceptions can be contained by Future

Let’s write some stupid code with Futures.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

scala> import scala.concurrent._
scala> import scala.concurrent.duration._

// Take default Scala global ExecutionContext which is a ForkJoin Thread Pool
scala> val ec = scala.concurrent.ExecutionContext.global
ec: scala.concurrent.ExecutionContextExecutor = scala.concurrent.impl.ExecutionContextImpl@15f445b7

// Create an immediately redeemed Future with a simple RuntimeException
scala> val f = future( throw new RuntimeException("foo") )(ec)
f: scala.concurrent.Future[Nothing] = scala.concurrent.impl.Promise$DefaultPromise@27380357

// Access brutally the value to show that the Future contains my RuntimeException
scala> f.value
res22: Option[scala.util.Try[Nothing]] = Some(Failure(java.lang.RuntimeException: foo))

// Use blocking await to get Future result
scala> Await.result(f, 2 seconds)
warning: there were 1 feature warnings; re-run with -feature for details
java.lang.RuntimeException: foo
  at $anonfun$1.apply(<console>:14)
  at $anonfun$1.apply(<console>:14)
  ...

You can see that a Future can contain an Exception (or more generally Throwable).

Fatal Exceptions can’t be contained by Future

If you look in Scala 2.10.0 Future.scala, in the scaladoc, you can find:

1
2
3
4

* The following throwable objects are not contained in the future:
* - `Error` - errors are not contained within futures
* - `InterruptedException` - not contained within futures
* - all `scala.util.control.ControlThrowable` except `NonLocalReturnControl` - not contained within futures

and in the code, in several places, in map or flatMap for example, you can read:

1
2
3
4
5

try {
...
} catch {
  case NonFatal(t) => p failure t
}

This means that every Throwable that is Fatal can’t be contained in the Future.Failure.

What’s a Fatal Throwable?

To define what’s fatal, let’s see what’s declared as non-fatal in NonFatal ScalaDoc.

1
2
3
4
5
6
7

* Extractor of non-fatal Throwables.  
* Will not match fatal errors like VirtualMachineError  
* (for example, OutOfMemoryError, a subclass of VirtualMachineError),  
* ThreadDeath, LinkageError, InterruptedException, ControlThrowable, or NotImplementedError. 
*
* Note that [[scala.util.control.ControlThrowable]], an internal Throwable, is not matched by
* `NonFatal` (and would therefore be thrown).

Let’s consider Fatal exceptions are just critical errors that can’t be recovered in general.

So what’s the problem?

It seems right not to catch fatal errors in the `Future, isn’t it?

But, look at following code:

1
2
3
4
5
6

// Let's throw a simple Fatal exception
scala> val f = future( throw new NotImplementedError() )(ec)
f: scala.concurrent.Future[Nothing] = scala.concurrent.impl.Promise$DefaultPromise@59747b17

scala> f.value
res0: Option[scala.util.Try[Nothing]] = None

Ok, the Future doesn’t contain the Fatal Exception as expected.

But where is my Fatal Exception if it’s not caught??? No crash, notification or whatever?

There should be an `UncaughtExceptionHandler at least notifying it!

The problem is in the default Scala ExecutionContext.

As explained in this issue, the exception is lost due to the implementation of the default global ExecutionContext provided in Scala.

This is a simple ForkJoin pool of threads but it has no UncaughtExceptionHandler. Have a look at code in Scala 2.10.0 ExecutionContextImpl.scala

1
2
3
4
5
6
7
8
9
10

try {
      new ForkJoinPool(
        desiredParallelism,
        threadFactory,
        null, //FIXME we should have an UncaughtExceptionHandler, see what Akka does
        true) // Async all the way baby
    } catch {
      case NonFatal(t) =>
        ...
    }

Here it’s quite clear: there is no registered `UncaughtExceptionHandler.

What’s the consequence?

Your Future hangs forever

1
2
3
4

scala> Await.result(f, 30 seconds)
warning: there were 1 feature warnings; re-run with -feature for details
java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:96)

As you can see, you can wait as long as you want, the Future is never redeemed properly, it just hangs forever and you don’t even know that a Fatal Exception has been thrown.

As explained in the issue, please note, if you use a custom ExecutionContext based on SingleThreadExecutor, this issue doesn’t appear!

1
2
3
4
5
6
7
8
9
10
11

scala> val es = java.util.concurrent.Executors.newSingleThreadExecutor
es: java.util.concurrent.ExecutorService = java.util.concurrent.Executors$FinalizableDelegatedExecutorService@1e336f59

scala> val ec = ExecutionContext.fromExecutorService(es)
ec: scala.concurrent.ExecutionContextExecutorService = scala.concurrent.impl.ExecutionContextImpl$$anon$1@34f43dac

scala>  val f = Future[Unit](throw new NotImplementedError())(ec)
Exception in thread "pool-1-thread-1" f: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise@7d01f935
scala.NotImplementedError: an implementation is missing
  at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:15)
  at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:15)

Conclusion

In Scala 2.10.0, if you have a Fatal Exception in a Future callback, your Future just trashes the Fatal Exception and hangs forever without notifying anything.

Hopefully, due to this already merged PR, in a future delivery of Scala 2.10.x, this problem should be corrected.

To finish, in the same old good issue, Viktor Klang also raised the question of what should be considered as fatal or not:

there’s a bigger topic at hand here, the one whether NotImplementedError, InterruptedException and ControlThrowable are to be considered fatal or not.

Meanwhile, be aware and take care ;)

Have Promise[NonFatal]!

Now play-json stand-alone is officially a stand-alone module in Play2.2. So you don’t have to use the dependency given in this article anymore but use Typesafe one like com.typesafe.play:play-json:2.2

In a very recent Pull Request, `play-json has been made a stand-alone module in Play2.2-SNAPSHOT master as play-iteratees.

It means:

• You can take Play2 Scala Json API as a stand-alone library and keep using Json philosophy promoted by Play Framework anywhere.
• play-json module is stand-alone in terms of dependencies but is a part & parcel of Play2.2 so it will evolve and follow Play2.x releases (and following versions) always ensuring full compatibility with play ecosystem.
• play-json module has 3 ultra lightweight dependencies:
  • play-functional
  • play-datacommons
  • play-iteratees

These are pure Scala generic pieces of code from Play framework so no Netty or whatever dependencies in it.
You can then import play-json in your project without any fear of bringing unwanted deps.

play-json will be released with future Play2.2 certainly so meanwhile, I provide:

• a build published in my Maven Github repository
• a sample project called play-json-alone

Even if the version is 2.2-SNAPSHOT, be aware that this is the same code as the one released in Play 2.1.0. This API has reached a good stability level. Enhancements and bug corrections will be brought to it but it’s production-ready right now.

Adding play-json 2.2-SNAPSHOT in your dependencies

In your Build.scala, add:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

import sbt._
import Keys._

object ApplicationBuild extends Build {

  val mandubianRepo = Seq(
    "Mandubian repository snapshots" at "https://github.com/mandubian/mandubian-mvn/raw/master/snapshots/",
    "Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
  )

  lazy val playJsonAlone = Project(
    BuildSettings.buildName, file("."),
    settings = BuildSettings.buildSettings ++ Seq(
      resolvers ++= mandubianRepo,
      libraryDependencies ++= Seq(
        "play"        %% "play-json" % "2.2-SNAPSHOT",
        "org.specs2"  %% "specs2" % "1.13" % "test",
        "junit"        % "junit" % "4.8" % "test"
      )
    )
  )
}

Using play-json 2.2-SNAPSHOT in your code:

Just import the following and get everything from Play2.1 Json API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import play.api.libs.json._
import play.api.libs.functional._

case class EucalyptusTree(col:Int, row: Int)

object EucalyptusTree{
  implicit val fmt = Json.format[EucalyptusTree]
}

case class Koala(name: String, home: EucalyptusTree)

object Koala{
  implicit val fmt = Json.format[Koala]
}

val kaylee = Koala("kaylee", EucalyptusTree(10, 23))

println(Json.prettyPrint(Json.toJson(kaylee)))

Json.fromJson[Koala](
  Json.obj(
    "name" -> "kaylee",
    "home" -> Json.obj(
      "col" -> 10,
      "row" -> 23
    )
)

Using play-json, you can get some bits of Play Framework pure Web philosophy.
Naturally, to unleash its full power, don’t hesitate to dive into Play Framework and discover 100% full Web Reactive Stack ;)

Thanks a lot to Play Framework team for promoting play-json as stand-alone module!
Lots of interesting features incoming soon ;)

Have fun!

Let’s go on unveiling Datomisca a bit more.

Remember Datomisca is an opensource Scala API (sponsored by Pellucid and Zenexity) trying to enhance Datomic experience for Scala developers.

After evoking queries compiled by Scala macros in previous article, I’m going to describe how Datomisca allows to create Datomic fact operations in a programmatic way and sending them to Datomic transactor using asynchronous/non-blocking API based on Scala 2.10 Future/ExecutionContext.

Facts about Datomic

First, let’s remind a few facts about Datomic:

Datomic is a immutable fact-oriented distributed schema-constrained database

It means:

Datomic stores very small units of data called facts

Yes no tables, documents or even columns in Datomic. Everything stored in it is a very small fact.

Fact is the atomic unit of data

Facts are represented by the following tuple called Datom

1

datom = [entity attribute value tx]

• entity is an ID and several facts can share the same ID making them facts of the same entity. Here you can see that an entity is very loose concept in Datomic.
• attribute is just a namespaced keyword : :person/name which is generally constrained by a typed schema attribute. The namespace can be used to logically identify an entity like “person” by regrouping several attributes in the same namespace.
• value is the value of this attribute for this entity at this instant
• tx uniquely identifies the transaction in which this fact was inserted. Naturally a transaction is associated with a time.

Facts are immutable & temporal

It means that:

• You can’t change the past
  Facts are immutable ie you can’t mutate a fact as other databases generally do: Datomic always creates a new version of the fact with a new value.
• Datomic always grows
  If you add more facts, nothing is deleted so the DB grows. Naturally you can truncate a DB, export it and rebuild a new smaller one.
• You can foresee a possible future
  From your present, you can temporarily add facts to Datomic without committing them on central storage thus simulating a possible future.

Reads/writes are distributed across different components

• One Storage service storing physically the data (Dynamo DB/Infinispan/Postgres/Riak/…)
• Multiple Peers (generally local to your app instances) behaving like high-speed synchronized cache obfuscating all the local data storage and synchro mechanism and providing the Datalog queries.
• One (or several) transactor(s) centralizing the write mechanism allowing ACID transactions and notifying peers about those evolutions.

For more info about architecture, go to this page

Immutability means known DB state is always consistent

You might not be up-to-date with central data storage as Datomic is distributed, you can even lose connection with it but the data you know are always consistent because nothing can be mutated.

This immutability concept is one of the most important to understand in Datomic.

Schema contrains entity attributes

Datomic allows to define that a given attribute must :

• be of given type : String or Long or Instant etc…
• have cardinality (one or many)
• be unique or not
• be fullsearchable or not
• be documented
• …

It means that if you try to insert a fact with an attribute and a value of the wrong type, Datomic will refuse it.

Datomic entity can also reference other entities in Datomic providing relations in Datomic (even if Datomic is not RDBMS). One interesting thing to know is that all relations in Datomic are bidirectional.

I hope you immediately see the link between these typed schema attributes and potential Scala type-safe features…

Author’s note : Datomic is more about evolution than mutation
I’ll let you meditate this sentence linked to theory of evolution ;)

Datomic operations

When you want to create a new fact in Datomic, you send a write operation request to the Transactor.

Basic operations

There are 2 basic operations:

Add a Fact

1

[:db/add entity-id attribute value]

Adding a fact for the same entity will NOT update existing fact but create a new fact with same entity-id and a new tx.

Retract a Fact

1

[:db/retract entity-id attribute value]

Retracting a fact doesn’t erase any fact but just tells: “for this entity-id, from now, there is no more this attribute”

You might wonder why providing the value when you want to remove a fact? This is because an attribute can have a MANY cardinality in which case you want to remove just a value from the set of values.

Entity operations

In Datomic, you often manipulate groups of facts identifying an entity. An entity has no physical existence in Datomic but is just a group of facts having the same entity-id. Generally, the attributes constituting an entity are logically grouped under the same namespace (:person/name, :person/age…) but this is not mandatory at all.

Datomic provides 2 operations to manipulate entities directly

Add Entity

1
2
3

{:db/id #db/id[:db.part/user -1]
  :person/name "Bob"
  :person/spouse #db/id[:db.part/user -2]}

Actually this is equivalent to 2 Add-Fact operations:

1
2
3

(def id #db/id[:db.part/user -1])
[:db/add id :person/name "Bob"]
[:db/add id :person/age 30]

Retract Entity

1

[:db.fn/retractEntity entity-id]

Special case of identified values

In Datomic, there are special entities built using the special attribute :db/ident of type Keyword which are said to be identified by the given keyword.

There are created as following:

1
2

[:db/add #db/id[:db.part/user] :db/ident :person.characters/clever]
[:db/add #db/id[:db.part/user] :db/ident :person.characters/dumb]

If you use :person.characters/clever or :person.characters/dumb, it references directly one of those 2 entities without using their ID.

You can see those identified entities as enumerated values also.

Now that you know how it works in Datomic, let’s go to Datomisca!

Datomisca programmatic operations

Datomisca’s preferred way to build Fact/Entity operations is programmatic because it provides more flexibility to Scala developers. Here are the translation of previous operations in Scala:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import datomisca._
import Datomic._

// creates a Namespace
val person = Namespace("person")

// creates a add-fact operation 
// It creates the datom (id keyword value _) from
//   - a temporary id (or a final long ID)
//   - the couple `(keyword, value)`
val addFact = Fact.add(DId(Partition.USER))(person / "name" -> "Bob")

// creates a retract-fact operation
val retractFact = Fact.retract(DId(Partition.USER))(person / "age" -> 123L)

// creates identified values
val violent = AddIdent(person.character / "violent")
val dumb = AddIdent(person.character / "dumb")

// creates a add-entity operation
val addEntity = Entity.add(DId(Partition.USER))(
  person / "name" -> "Bob",
  person / "age" -> 30L,
  person / "characters" -> Set(violent.ref, dumb.ref)
)

// creates a retract-entity operation from real Long ID of the entity
val retractEntity = Entity.retract(3L)

val ops = Seq(addFact, retractFact, addEntity, retractEntity)

Note that:

• person / "name" creates the keyword :person/name from namespace person
• DId(Partition.USER) generates a temporary Datomic Id in Partition USER. Please note that you can create your own partition too.
• violent.ref is used to access the keyword reference of the identified entity.
• ops = Seq(…) represents a collection of operations to be sent to transactor.

Datomisca Macro operations

Remember the way Datomisca dealt with query by parsing/validating Datalog/Clojure queries at compile-time using Scala macros?

You can do the same in Datomisca with operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

val id = DId(Partition.USER)
val weak = AddIdent( person / "character", "weak"))
val dumb = AddIdent( person / "character", "dumb"))

val ops = Datomic.ops("""[
   [:db/add #db/id[:db.part/user] :db/ident :region/n]
   [:db/add \${DId(Partition.USER)} :db/ident :region/n]
   [:db/retract #db/id[:db.part/user] :db/ident :region/n]
   [:db/retractEntity 1234]
   {
      :db/id \${id}
      :person/name "toto"
      :person/age 30
      :person/characters [ \$weak \$dumb ]
   }
]""")

It compiles what’s between """…""" at compile-time and tells you if there are errors and then it builds Scala corresponding operations.

Ok it’s cool but if you look better, you’ll see there is some sugar in this Clojure code:

• \${DId(Partition.USER)}
• \$weak
• \$dumb

You can use Scala variables and inject them into Clojure operations at compile-time as you do for Scala string interpolation

For Datomic queries, the compiled way is really natural but we tend to prefer programmatic way to build operations because it feels to be much more “scala-like” after experiencing both methods.

Datomisca runtime parsing

There is a last way to create operations by parsing at runtime a String and throwing an exception if the syntax is not valid.

1

val ops = Datomic.parseOps(""" … """)

It’s very useful if you have existing Datomic Clojure files (containing schema or bootstrap data) that you want to load into Datomic.

Datomisca reactive transactions

Last but not the least, let’s send those operations to Datomic Transactor.

In its Java API, Datomic Connection provides a transact asynchronous API based on a ListenableFuture. This API can be enhanced in Scala because Scala provides much more evolved asynchronous/non-blocking facilities than Java based on Scala 2.10 Future/ExecutionContext.

Future allows to implement your asynchronous call using continuation style based on Scala classic map/flatMap methods. ExecutionContext is a great tool allowing to specify in which pool of threads your asynchronous call will be executed making it non-blocking with respect to your current execution context (or thread).

This new feature is really important when you work with reactive API such as Datomisca or Play too so don’t hesitate to study it further.

Let’s look at code directly to show how it works in Datomisca:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

import datomisca._
import Datomic._

// don't forget to bring an ExecutionContext in your scope… 
// Here is default Scala ExecutionContext which is a simple pool of threads with one thread per core by default
import scala.concurrent.ExecutionContext.Implicits.global

// creates an URI
val uri = "datomic:mem://mydatomicdn"
// creates implicit connection
implicit val conn = Datomic.connect(uri)

// a few operations
val ops = Seq(addFact, retractFact, addEntity, retractEntity)

val res: Future[R] = Datomic.transact(ops) map { tx : TxReport =>
   // do something
   …

   // return a value of type R (anything you want)
   val res: R = …

   res
}

// Another example by building ops directly in the transact call and using flatMap
Datomic.transact(
  Entity.add(id)(
    person / "name"      -> "toto",
    person / "age"       -> 30L,
    person / "character" -> Set(weak.ref, dumb.ref)
  ),
  Entity.add(DId(Partition.USER))(
    person / "name"      -> "tutu",
    person / "age"       -> 54L,
    person / "character" -> Set(violent.ref, clever.ref)
  ),
  Entity.add(DId(Partition.USER))(
    person / "name"      -> "tata",
    person / "age"       -> 23L,
    person / "character" -> Set(weak.ref, clever.ref)
  )
) flatMap { tx =>
  // do something
  …
  val res: Future[R] = …

  res
}

Please note the tx: TxReport which is a structure returned by Datomic transactor containing information about last transaction.

Datomisca resolving Real ID

In all samples, we create operations based on temporary ID built by Datomic in a given partition.

1

DId(Partition.USER)

But once you have inserted a fact or an entity into Datomic, you need to resolve the real final ID to use it further because the temporary ID is no more meaningful.

The final ID is resolved from the TxReport send back by Datomic transactor. This TxReport contains a map between temporary ID and final ID. Here is how you can use it in Datomisca:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

val id1 = DId(Partition.USER)
val id2 = DId(Partition.USER)

Datomic.transact(
  Entity.add(id1)(
    person / "name"      -> "toto",
    person / "age"       -> 30L,
    person / "character" -> Set(weak.ref, dumb.ref)
  ),
  Entity.add(id2)(
    person / "name"      -> "tutu",
    person / "age"       -> 54L,
    person / "character" -> Set(violent.ref, clever.ref)
  )
) map { tx =>
  val finalId1: Long = tx.resolve(id1)
  val finalId2: Long = tx.resolve(id2)
  // or
  val List(finalId1, finalId2) = List(id1, id2) map { tx.resolve(_) }
}

Have Promise[Fun]!

Last week, we have launched Datomisca, our opensource Scala API trying to enhance Datomic experience for Scala developers.

Datomic is great in Clojure because it is was made for it. Yet, we believe Scala can provide a very good platform for Datomic too because the functional concepts found in Clojure are also in Scala except that Scala is a compiled and statically typed language whereas Clojure is dynamic. Scala could also bring a few features on top of Clojure based on its features such as static typing, typeclasses, macros…

This article is the first of a serie of articles aiming at describing as shortly as possible specific features provided by Datomisca. Today, let’s present how Datomisca enhances Datomic queries.

Query in Datomic?

Let’s take the same old example of a Person having :

• a name String
• a age Long
• a birth Date

So, how do you write a query in Datomic searching a person by its name? Like that…

1
2
3
4

(def q [ :find ?e
  :in $ ?name
  :where [ ?e :person/name ?name ]
])

As you can see, this is Clojure using Datalog rules.

In a summary, this query:

• accepts 2 inputs parameters:
  • a datasource $
  • a name parameter ?name
• searches facts respecting datalog rule [ ?e :person/name ?name ]: a fact having attribute :person/name with value ?name
• returns the ID of the found facts ?e

Reminders about Datomic queries

Query is a static data structure

An important aspect of queries to understand in Datomic is that a query is purely a static data structure and not something functional. We could compare it to a prepared statement in SQL: build it once and reuse it as much as you need.

Query has input/ouput parameters

In previous example:

• :in enumerates input parameters
• :find enumerates output parameters

When executing this query, you must provide the right number of input parameters and you will retrieve the given number of output parameters.

Query in Datomisca?

So now, how do you write the same query in Datomisca?

1
2
3
4
5

val q  = Query("""
[ :find ?e
  :in $ ?name
  :where [ ?e :person/name ?name ] 
]""")

I see you’re a bit disappointed: a query as a string whereas in Clojure, it’s a real data structure…

This is actually the way the Java API sends query for now. Moreover, using strings like implies potential bad practices such as building queries by concatenating strings which are often the origin of risks of code injection in SQL for example…

But in Scala we can do a bit better using new Scala 2.10 features : Scala macros.

So, using Datomisca, when you write this code, in fact, the query string is parsed by a Scala macro:

• If there are any error, the compilation breaks showing where the error was detected.
• If the query seems valid (with respect to our parser), the String is actually replaced by a AST representing this query as a data structure.
• The input/output parameters are infered to determine their numbers.

Please note that the compiled query is a simple immutable AST which could be manipulated as a Clojure query and re-used as many times as you want.

Example OK with single output

1
2
3
4
5
6
7
8
9

scala> import datomisca._
import datomisca._

scala> val q  = Query("""
     [ :find ?e
       :in $ ?name
       :where [ ?e :person/name ?name ] 
     ]""")
q: datomisca.TypedQueryAuto2[datomisca.DatomicData,datomisca.DatomicData,datomisca.DatomicData] = [ :find ?e :in $ ?name :where [?e :person/name ?name] ]

Without going in deep details, here you can see that the compiled version of q isn’t a Query[String] but a TypedQueryAuto2[DatomicData, DatomicData, DatomicData] being an AST representing the query.

TypedQueryAuto2[DatomicData, DatomicData, DatomicData] means you have:

• 2 input parameters $ ?name of type DatomicData and DatomicData
• Last type parameter represents output parameter ?e of type DatomicData

Note : DatomicData is explained in next paragraph.

Example OK with several outputs

1
2
3
4
5
6
7
8
9
10

scala> import datomisca._
import datomisca._

scala> val q  = Query("""
     [ :find ?e ?age
       :in $ ?name
       :where [ ?e :person/name ?name ] 
              [ ?e :person/age ?age ]  
     ]""")
q: datomisca.TypedQueryAuto2[datomisca.DatomicData,datomisca.DatomicData,(datomisca.DatomicData, datomisca.DatomicData)] = [ :find ?e ?age :in $ ?name :where [?e :person/name ?name] [?e :person/age ?age] ]

TypedQueryAuto2[DatomicData,DatomicData,(DatomicData, DatomicData)] means you have:

• 2 input parameters $ ?name of type DatomicData
• last tupled type parameter represents the 2 output parameters ?e ?age of type DatomicData

Examples with syntax-error

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

scala> import datomisca._
import datomisca._

scala> val q  = Query("""
     [ :find ?e
       :in $ ?name
       :where [ ?e :person/name ?name 
     ]""")
<console>:14: error: `]' expected but end of source found
     ]""")
      ^

scala> val q  = Query("""
     [ :find ?e
       :in $ ?name
       :where [ ?e person/name ?name ]
     ]""")
<console>:13: error: `]' expected but `p' found
       :where [ ?e person/name ?name ]
                   ^

Here is see that the compiler will tell you where it detects syntax errors.

The query compiler is not yet complete so don’t hesitate to report us when you discover issues.

What’s DatomicData ?

Datomisca wraps completely Datomic API and types. So Datomisca doesn’t let any Datomic/Clojure types perspirating into its domain and wraps them all in the so-called DatomicData which is the abstract parent trait of all Datomic types seen from Datomisca. For each Datomic type, you have the corresponding specific DatomicData:

• DString for String
• DLong for Long
• DatomicFloat for Float
• DSet for Set
• DInstant for Instant
• …

Why not using Pure Scala types directly?

Firstly, because type correspondence is not exact between Datomic types and Scala. The best sample is Instant: is it a java.util.Date or a jodatime.DateTime?

Secondly, we wanted to keep the possibility of converting Datomic types into different Scala types depending on our needs so we have abstracted those types.

This abstraction also isolates us and we can decide exactly how we want to map Datomic types to Scala. The trade-off is naturally that, if new types appear in Datomic, we must wrap them.

Keep in mind that Datomisca queries accept and return DatomicData

All query data used as input and output paremeters shall be DatomicData. When getting results, you can convert those generic DatomicData into one of the previous specific types (DString, DLong, … ).

From DatomicData, you can also convert to Scala pure types based on implicit typeclasses:

1
2
3
4
5
6
7
8

DatomicData.as[T](implicit rd: DDReader[DatomicData, T])

scala> DString("toto").as[String]
res0: String = toto

scala> DString("toto").as[Long]
java.lang.ClassCastException: datomisca.DString cannot be cast to datomisca.DLong
...

Note 1 : that current Scala query compiler is a bit restricted to the specific domain of Datomic queries and doesn’t support all Clojure syntax which might create a few limitations when calling Clojure functions in queries. Anyway, a full Clojure syntax Scala compiler is in the TODO list so these limitations will disappear once it’s implemented…

Note 2 : Current macro just infers the number of input/output parameters but, using Schema typed attributes that we will present in a future article, we will provide some deeper features such as parameter type inference.

Execute the query

You can create queries independently of any connection to Datomic. But you need an implicit DatomicConnection in your scope to execute it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import datomisca._
import Datomic._

// Creates an implicit connection
val uri = "…"
implicit lazy val conn = Datomic.connect(uri)

// Creates the query
val queryFindByName = Query("""
[ :find ?e ?birth
  :in $ ?name
  :where [ ?e :person/name ?name ]
         [ ?e :person/birth ?birth ]        
]""")

// Executes the query     
val results: List[(DatomicData, DatomicData] = Datomic.q(queryFindByName, database, DString("John"))
// Results type is precised for the example but not required

Please note we made the database input parameter mandatory even if it’s implicit in when importing Datomic._ because in Clojure, it’s also required and we wanted to stick to it.

Compile-error if wrong number of inputs

If you don’t provide 2 input parameters, you will get a compile error because the query expects 2 input parameters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// Following would not compile because query expects 2 input parameters
val results: List[(DatomicData, DatomicData] = Datomic.q(queryFindByName, DString("John"))

[info] Compiling 1 Scala source to /Users/pvo/zenexity/workspaces/workspace_pellucid/datomisca/samples/getting-started/target/scala-2.10/classes...
[error] /Users/pvo/zenexity/workspaces/workspace_pellucid/datomisca/samples/getting-started/src/main/scala/GettingStarted.scala:87: overloaded method value q with alternatives:
[error]   [A, R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)](query: datomisca.TypedQueryAuto1[A,R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)], a: A)(implicit db: datomisca.DDatabase, implicit ddwa: datomisca.DD2Writer[A], implicit outConv: datomisca.DatomicDataToArgs[R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)])List[R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)] <and>
[error]   [R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)](query: datomisca.TypedQueryAuto0[R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)], db: datomisca.DDatabase)(implicit outConv: datomisca.DatomicDataToArgs[R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)])List[R(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)(in method q)] <and>
[error]   [OutArgs <: datomisca.Args, T](q: datomisca.TypedQueryInOut[datomisca.Args1,OutArgs], d1: datomisca.DatomicData)(implicit db: datomisca.DDatabase, implicit outConv: datomisca.DatomicDataToArgs[OutArgs], implicit ott: datomisca.ArgsToTuple[OutArgs,T])List[T] <and>
[error]   [InArgs <: datomisca.Args](query: datomisca.PureQuery, in: InArgs)(implicit db: datomisca.DDatabase)List[List[datomisca.DatomicData]]
[error]  cannot be applied to (datomisca.TypedQueryAuto2[datomisca.DatomicData,datomisca.DatomicData,(datomisca.DatomicData, datomisca.DatomicData)], datomisca.DString)
[error]         val results = Datomic.q(queryFindByName, DString("John"))
[error]                               ^
[error] one error found
[error] (compile:compile) Compilation failed

The compile error seems a bit long as the compiler tries a few different version of Datomic.q but just remind that when you see cannot be applied to (datomisca.TypedQueryAuto2[…, it means you provided the wrong number of input parameters.

Use query results

Query results are List[DatomicData…] depending on the output parameters inferred by the Scala macros.

In our case, we have 2 output parameters so we expect a List[(DatomicData, DatomicData)]. Using List.map (or headOption to get the first one only), you can then use pattern matching to specialize your (DatomicData, DatomicData) to (DLong, DInstant) as you expect.

1
2
3
4
5
6

results map {
  case (e: DLong, birth: DInstant) =>
    // converts into Scala types
    val eAsLong = e.as[Long]
    val birthAsDate = birth.as[java.util.Date]
}

Note 1: that when you want to convert your DatomicData, you can use our converters based on implicit typeclasses as following

Note 2: The Scala macro has not way just based on query to infer the real types of output parameters but ther is a TODO in the roadmap: using typed schema attributes presented in a future article, we will be able to do better certainly… Be patient ;)

More complex queries

As Datomisca parses the queries, you may wonder what is the level of completeness of the query parser for now?

Here are a few examples showing what can be executed already:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

////////////////////////////////////////////////////
// using variable number of inputs
val q = Query("""[
 :find ?e
 :in $ [?names ...] 
 :where [?e :person/name ?names]
]""")

Datomic.q(q, database, DSet(DString("toto"), DString("tata")))

////////////////////////////////////////////////////
// using tuple inputs
val q = Query("""[
  :find ?e ?name ?age
  :in $ [[?name ?age]]
  :where [?e :person/name ?name]
         [?e :person/age ?age]
]""")

Datomic.q(q,
  database,
  DSet(
    DSet(DString("toto"), DLong(30L)),
    DSet(DString("tutu"), DLong(54L))
  )
)

////////////////////////////////////////////////////
// using function such as fulltext search
val q = Query("""[
  :find ?e ?n
  :where [(fulltext $ :person/name "toto") [[ ?e ?n ]]]
]""")

////////////////////////////////////////////////////
// using rules
val totoRule = Query.rules("""
[ [ [toto ?e]
    [?e :person/name "toto"]
] ]
""")

val q = Query("""[
 :find ?e ?age
 :in $ %
 :where [?e :person/age ?age]
        (toto ?e)
]
""")

////////////////////////////////////////////////////
// using query specifying just the field in fact to be searched
val q = Query("""[:find ?e :where [?e :person/name]]""")

Note that currently Datomisca reserializes queries to string when executing because Java API requires it but once Datomic Java API accepts that we pass List[List[Object]] instead of strings for query, the interaction will be more direct…

Have datomiscafun!

Maquereau [Scomber Scombrus]

[/makʀo/] in french phonetics

Since I discovered Scala Macros with Scala 2.10, I’ve been really impressed by their power. But great power means great responsibility as you know. Nevertheless, I don’t care about responsability as I’m just experimenting. As if mad scientists couldn’t experiment freely!

Besides being a very tasty pelagic fish from scombroid family, Maquereau is my new sandbox project to experiment eccentric ideas with Scala Macros.

Here is my first experiment which aims at studying the concepts of pataphysics applied to Scala Macros.

Pataphysics applied to Macros

Programming is math

I’ve heard people saying that programming is not math.
This is really wrong, programming is math.

And let’s be serious, how would you seek attention in urbane cocktails without those cute little words such as functors, monoids, contravariance, monads?

She/He> What do you do?
You> I’m coding a list fold.
She/He> Ah ok, bye.
You> Hey wait…

She/He> What do you do?
You> I’m deconstructing my list with a catamorphism based on a F-algebra as underlying functor.
She/He> Whahhh this is so exciting! Actually you’re really sexy!!!
You> Yes I known insignificant creature!

Programming is also a bit of Physics

Code is static meanwhile your program is launched in a runtime environment which is dynamic and you must take these dynamic aspects into account in your code too (memory, synchronization, blocking calls, resource consumption…). For the purpose of the demo, let’s accept programming also implies some concepts of physics when dealing with dynamic aspects of a program.

Compiling is Programming Metaphysics

Between code and runtime, there is a weird realm, the compiler runtime which is in charge of converting static code to dynamic program:

• The compiler knows things you can’t imagine.
• The compiler is aware of the fundamental nature of math & physics of programming.
• The compiler is beyond these rules of math & physics, it’s metaphysics.

Macro is Pataphysics

Now we have Scala Macros which are able:

• to intercept the compiling process for a given piece of code
• to analyze the compiler AST code and do some computation on it
• to generate another AST and inject it back into the compile-chain

When you write a Macro in your own code, you write code which runs in the compiler runtime. Moreover a macro can go even further by asking for compilation from within the compiler: c.universe.reify{ some code }… Isn’t it great to imagine those recursive compilers?

So Scala macro knows the fundamental rules of the compiler. Given compiler is metaphysics, Scala macro lies beyond metaphysics and the science studying this topic is called pataphysics.

This science provides very interesting concepts and this article is about applying them to the domain of Scala Macros.

I’ll let you discover pataphysics by yourself on wikipedia

Let’s explore the realm of pataphysics applied to Scala macro development by implementing the great concept of patamorphism, well-known among pataphysicians.

Defining Patamorphism

In 1976, the great pataphysician, Ernst Von Schmurtz defined patamorphism as following:

A patamorphism is a patatoid in the category of endopatafunctors…

Explaining the theory would be too long with lots of weird formulas. Let’s just skip that and go directly to the conclusions.

First of all, we can consider the realm of pataphysics is the context of Scala Macros.

Now, let’s take point by point and see if Scala Macro can implement a patamorphism.

A patamorphism should be callable from outside the realm of pataphysics

A Scala macro is called from your code which is out of the realm of Scala macro.

A patamorphism can’t have effect outside the realm of pataphysics after execution

This means we must implement a Scala Macro that :

• has effect only at compile-time
• has NO effect at run-time

From outside the compiler, a patamorphism is an identity morphism that could be translated in Scala as:

1

def pataMorph[T](t: T): T

A patamorphism can change the nature of things while being computed

Even if it has no effect once applied, meanwhile it is computed, it can :

• have side-effects on anything
• be blocking
• be mutable

Concerning these points, nothing prevents a Scala Macro from respecting those points.

A patamorphism is a patatoid

You may know it but patatoid principles require that the morphism should be customisable by a custom abstract seed. In Scala samples, patatoid are generally described as following:

1
2
3
4
5
6

trait Patatoid {
  // Seed is specific to a Patatoid and is used to configure the sprout mechanism  
  type Seed
  // sprout is the classic name
  def sprout[T](t: T)(seed: Seed): T
}

So a patamorphism could be implemented as :

1

trait PataMorphism extends Patatoid

A custom patamorphism implemented as a Scala Macro would be written as :

1
2
3
4
5
6
7

object MyPataMorphism extends PataMorphism {
  type Seed = MyCustomSeed
  def sprout[T](t: T)(seed: Seed): T = macro sproutImpl

  // here is the corresponding macro implementation
  def sproutImpl[T: c1.WeakTypeTag](c1: Context)(t: c1.Expr[T])(seed: c1.Expr[Seed]): c1.Expr[T] = { … }
}

But in current Scala Macro API, this is not possible for a Scala Macro to override an abstract function so we can’t write it like that and we need to trick a bit. Here is how we can do simply :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

trait Patatoid{
  // Seed is specific to a Patatoid and is used to configure the sprout mechanism  
  type Seed
  // we put the signature of the macro implementation in the abstract trait
  def sproutMacro[T: c1.WeakTypeTag](c1: Context)(t: c1.Expr[T])(seed: c1.Expr[Seed]): c1.Expr[T]
}

/**
  * PataMorphism 
  */
trait PataMorphism extends Patatoid

// Custom patamorphism
object MyPataMorphism extends PataMorphism {
  type Seed = MyCustomSeed
  // the real sprout function expected for patatoid
  def sprout[T](t: T)(implicit seed: Seed): T = macro sproutMacro[T]

  // the real implementation of the macro and of the patatoid abstract operation
  def sproutMacro[T: c1.WeakTypeTag](c1: Context)(t: c1.Expr[T])(seed: c1.Expr[Seed]): c1.Expr[T] = {
    …
    // Your implementation here
    …
  }
}

Let’s implement a 1st sample of patamorphism called VerySeriousCompiler.

Sample #1: VerySeriousCompiler

What is it?

VerySeriousCompiler is a pure patamorphism which allows to change compiler behavior by :

• Choosing how long you want the compilation to last
• Displaying great messages at a given speed while compiling
• Returning the exact same code tree given in input

VerySeriousCompiler is an identity morphism returning your exact code without leaving any trace in AST after macro execution.

VerySeriousCompiler is implemented exactly using previous patamorphic pattern and the compiling phase can be configured using custom Seed:

1
2
3
4
5
6

/** Seed builder 
  * @param duration the duration of compiling in ms
  * @param speed the speed between each message display in ms
  * @param messages the messages to display
  */
def seed(duration: Long, speed: Long, messages: Seq[String])

When to use it?

VerySeriousCompiler is a useful tool when you want to have a coffee or discuss quietly at work with colleagues and fool your boss making him/her believe you’re waiting for the end of a very long compiling process.

To use it, you just have to modify your code using :

1
2
3
4
5
6
7
8
9

VerySeriousCompiler.sprout{
  …some code…
}

//or even 

val xxx = VerySeriousCompiler.sprout{
  …some code returning something…
}

Then you launch compilation for the duration you want, displaying meaningful messages in case your boss looks at your screen. Then, you have an excuse if your boss is not happy about your long pause, tell him/her: “Look, it’s compiling”.

Remember that this PataMorphism doesn’t pollute your code at runtime at all, it has only effects at compile-time and doesn’t inject any other code in the AST.