Never a “Hello, World!” was so real

As promised, here is our “Hello, World!” example (also in PDF). This kind of program is illustrative to show the very basics of a programming language. Usually, the program consists on showing the string “Hello World!” message in a console. However, Speech does not know much about consoles or strings. Instead, it is really good at representing social processes and the rules governing them. Therefore, we will be helping God in order to fulfill his functional requirement list: create the world, create the human being and empower him to say something. So please, be open-minded to meet Speech!

Before you leave: we hope to see you in the next delivery, where we will be showing you how to program Twitter in Speech, a real application that fits perfectly with our language.

Posted in Uncategorized | Leave a comment

Speech 0.1 released!

We are happy to announce the first release of the Speech interpreter. This is a beta release with the minimum functionality required to test significant application examples, demonstrate the virtues of the Speech DSL and … receive feedback from you! You can download the interpreter from the Speech Community Portal:

There you will also find instructions to run the interpreter, a short presentation of Speech, a first version of the user guide to Speech programming in Scala, and links to prototype applications. Parts of the Speech interpreter were already open sourced (particularly, the updatable package), and we aim at publishing other major modules of the interpreter in the near future.

Admittedly, documentation is far from being complete, so our purpose now is mainly showing you what a Speech program looks like. We expect to give you soon a full-fledge documentation including the Speech API and a design guide in order to allow you to become a proficient Speech programmer. In the meantime, we have planned a series of blog posts focusing in different application domains. And the first program that we will use to exemplify Speech will be … the “Hello, world!” program, of course. I hope you like it!

Posted in Uncategorized | Leave a comment

Macros and Reflective Calls to eliminate boilerplate

In our previous post, we told you about updatable, a library that empowers programmers to build and update immutable objects in generic contexts. We saw the builder macro as a main element in the library, but we did not explain in detail how it was implemented. We think it uses an interesting pattern to eliminate boilerplate, so we want to share it with you. Instead of showing the original updatable builder, we are going to use a reduced version, in order to keep the example small. We call it factory, because its unique aim is to instantiate traits. Now let’s get to work!

Even in Scala, there are situations where we can find annoying boilerplate. That could be the case of the following lines:

trait A {
  val a1: Int
  val a2: String
}

case class AImpl(a1: Int, a2: String) extends A

AImpl(a1 = 0, a2 = "")

trait B extends A {
  val b1: Double
}

case class BImpl(a1: Int, a2: String, b1: Double) extends B

BImpl(a1 = 3, a2 = "", b1 = 3.0)

The case class implements the trait and creates an object factory (among many other things). There is some boilerplate in this implementation that would be nice to eliminate: concretely, the redundant argument list that conforms the constructor. This might be a potential problem if the number of attributes grows excessively. Our approach to eliminate this boilerplate consists on using macros as follows:

trait A { ... }

val A = builder[A]

A(_a1 = 0, _a2 = "")

trait B extends A { ... }

val B = builder[B]

B(_a1 = 3, _a2 = "", _b1 = 3.0)

Scala 2.10 macros are limited in the creation of new types (a limitation that has been lifted with type macros in the macro paradise project), and can only return expressions. So in order to instantiate objects of types A and B we have to exploit anonymous classes. This instances will be returned by the apply method of the object returned by the builder macro.  But you may wonder how is it possible to get these invocations working, since the apply method signature seems to be variable in each case. In fact, there are at least two possible solutions to this problem: either returning a new object of a structural type that declares a custom apply method or simply returning an anonymous function of the proper type. We have chosen here the former alternative in analogy with the way in which the updatable builder is implemented (where you can find additional services besides the factory method). Thus, the code that should be generated by the macro is shown in the next snippet:

// builder[A]
new {
  def apply(_a1: Int, _a2: String): A = new A {
    val a1 = _a1
    val a2 = _a2
  }
}

// builder[B]
new {
  def apply(_a1: Int, _a2: String, _b1: Double): B = new B {
    val a1 = _a1
    val a2 = _a2
    val b1 = _b1
  }
}

Now, it is time for us to analyze the macro implementation. Since we are going to generate dynamic code – mainly in the apply’s argument list – it is not feasible to exploit the reify macro – which allows the programmer to create the returning expression in a natural way. So, one could choose to use either parse or manually create the AST. The latter one is discouraged by the macro author because the code turns pretty verbose. In the current case, if we had used the raw style, the number of lines would have grown remarkably. The reason why this huge growth happens is because creating trait instances produces very complex trees. Nevertheless, we should consider the AST version if we aim to optimize macro execution timings. Next, the macro implementation is shown:

  def builder[T] = macro builderImpl[T]

  def builderImpl[T: c.WeakTypeTag](c: Context) = {
    import c.universe._
    import c.mirror._

    implicit class SymbolHelper(sym: Symbol) {
      private def cross(t: Type): Type = t match {
        case NullaryMethodType(inner) => inner
        case _ => t
      }
      def name: String = sym.name.encoded
      def tpe: Type = cross(sym.typeSignature)
      def isAccessor: Boolean = sym.isTerm && sym.asTerm.isAccessor
    }

    implicit class TypeHelper(tpe: Type) {
      def name: String = (tpe.typeSymbol: SymbolHelper).name
      def accessors: List[Symbol] = tpe.members.toList.reverse.filter(sym =>
        sym.isAccessor)
    }

    def buildObject = {

      val tpe = weakTypeOf[T]

      def instanceTrait = {
        val vals = tpe.accessors.foldLeft("")(
          (s, sym) => s + s"val ${sym.name} = _${sym.name}\n")
	s"new ${tpe.name} { $vals }"
      }

      def applyArguments = tpe.accessors map { sym =>
	s"_${sym.name}: ${sym.tpe}"
      } mkString ","

      s"""
      new {
	def apply($applyArguments): ${tpe.name} = $instanceTrait
      }
      """
    }

    c.Expr[Object](c.parse(
      s"""
      { val aux = $buildObject; aux } // SI-6992
      """
    ))
  }

First, it is important to notice that neither the macro implementation nor the definition declare the result type. Also, note that the parameter type of the Expr object returned by the macro is simply Object. Thus, we let the compiler infer the proper type of the factory object returned by the macro. Concerning the implementation, we find two main areas in the previous code: reflection tasks and tree creation tasks. The first ones are owned by SymbolHelper and TypeHelper implicit classes, which extends Symbol and Type, respectively. The accessor concept makes reference to the methods that permit the programmer to access the trait values. In the factory’s case they have a direct correspondence with the apply’s argument list. With respect to the creation tasks, as we said before, the parse method seems to be the better alternative in this case to generate trees. To invoke it, we need a string containing the instructions to reify. That is the buildFactory‘s duty, which uses string interpolation to format the code that will be finally expanded. This Scala’s fresh feature notably improves the instruction’s readability.

Currently, we are experimenting with some new ideas to make the factory (and therefore updatable) better. Mainly, we would like to generate a parameterized apply method in those situations when the attribute type is abstract:

trait C {
  type C1
  val c1: C1
}

val C = builder[C]

C[Int](_c1 = 3)

We will tell you about this and other extensions in following posts.

To sum up, any programmer knows that boilerplate is not funny at all. In order to avoid lots of dangerous copy-paste actions, external pre-compilation tools may also be employed to solve the issue. However, by doing so, we are adding unnecessary complexity in the project. Today, we have shown how to use macros to face the problem, with a native feature! We have applied it to develop the Speech DSL. How are you planning to use it?

Posted in case classes, Embedded DSLs, immutability, Macros, Scala | Leave a comment

Updating immutable objects in generic contexts

Immutability is one of the hallmarks of functional design, and writing idiomatic programs in Scala highly relies on manipulating immutable objects. Now, if we don’t have mutable fields (aka vars) … how can we update objects in a convenient way? Scala provides so-called case classes which have a copy method with the required functionality. And we can also use lenses, a higher-level abstraction that you can find in popular Scala libraries such as scalaz and shapeless (you can find a macro-based implementation in the macrocosm project as well). Nevertheless, all these implementations build some way or another upon case classes as the basic updating mechanism.

Now, sometimes writing case classes for your specification traits is cumbersome, since it involves a lot of boilerplate. And this problem is specially exacerbated in the presence of inheritance hierarchies, where traits get also polluted with getters and setters. Wouldn’t it be nice if we found some way of automatically deriving case classes and eliminating all this boilerplate? Well, this is a question for macros, and type macros, in particular. But type macros are still a pre-release feature of Scala. So, what can be done with def macros alone? We have developed a library that exploits def macros in combination with reflective calls to eliminate the need of writing implementation classes. And it allows the programmer to update immutable objects in generic contexts with a minimum overhead. This library is called org.hablapps.updatable and you can find it on GitHub. Before explaining its functionality, though, let’s illustrate the problem with a simple example, and let’s solve it using case classes.

The problem …

We will illustrate the kind of updating problem we have to deal with by considering a design problem in the implementation of Speech itself. Among other things, our DSL offers to programmers an abstract layer which implements generic types and state transformations that can be reused across any kind of social domain. For instance, the layer includes interaction contexts and agent roles, and the play transformation which adds a new agent role within some context. We want to implement interaction contexts and agent roles as immutable objects and be able to reuse the play transformation as-is, across any application domain. For instance, think of Twitter: there you find accounts, followers, tweeters, and many other concepts. We can think of accounts as the contexts where tweeters interact with their followers; and following someone would involve playing a new follower role within their account. As another example, think of courses as contexts of interaction for student and teacher agents, and some student enrolling some course: this action can also be implemented with the help of the play action.

… Solved using case classes

Our design problem can be understood as a particular example of the family polymorphism problem, which can be easily solved in Scala using abstract types and the cake pattern. Accordingly, the Speech abstraction layer can be understood as a family of types which vary together covariantly in each application layer. In particular, our implementation will be structured in three basic layers:

  • An abstract layer (the Speech layer) which provides generic implementations of interactions contexts, agents, and generic transformations, in terms of traits and generic methods.
  • An application layer which provides specific implementations of domain-dependent concepts in terms of traits that extends the corresponding generic traits.
  • Another application layer which provides the implementation of domain-dependent traits, in terms of case classes.

The following snippet represents an implementation sketch of the first layer:

trait Speech {
  trait Interaction[This <: Interaction[This]] { self: This =>
    type Member <: Agent[Member]
    def member: Set[Member]
    def member_=(agent: Set[Member]): This
  }

  trait Agent[This <: Agent[This]] { self: This =>
  }

  def play[I <: Interaction[I]](i: I)(a: i.Member): I =
    i.member = i.member + a
}

Here, the Speech layer just implements two traits for the Interaction and Agent types, as well as the play transformation. Note that the play method must work for any type of interaction and agent, and we don’t want to forget the exact type of interaction once we call the method. Hence, the method is parameterized with respect to some interaction type I. Now, the agent to be played within that context must be compatible with the interaction type, i.e. we can play followers within Twitter accounts, but not students. To account for this constraint, we declare an abstract type Member in the Interaction trait and exploit dependent types in the play signature. How do we add the new member agent? We need a setter, of course. And this setter must also return the specific type of the interaction (again, to avoid type information loss). For that purpose, the trait is parameterized with the This parameter, following the standard solution to this problem. Last, note the updated sentence in the play method: it’s as if member was a var. But it’s not, it’s simply that we named the getter and setter according to the var convention.

How do we reuse this abstract layer? The following snippet uses the Twitter domain to illustrate reuse of the Speech layer.

trait Twitter extends Speech {

  trait Account extends Interaction[Account] {
    type Member = Follower
  }

  def Account(members: Set[Follower] = Set()): Account

  trait Follower extends Agent[Follower] {
  }

  def Follower(): Follower
}

The Twitter layer simply extends the Speech traits and sets the abstract members to the desired values. Of course, a real implementation will include additional domain-dependent attributes, methods, etc., to the Account and Follower traits (think of the Speech member attribute as a kind of standard attribute). Note that we also included factory methods for the Account and Follower types. In a real implementation, it is more than likely that we will need them. And we don’t want to commit to any specific implementation class, so we declare them abstract. The next portion of the cake will provide the implementations of the Twitter types – using case classes:

  trait TwitterImpl { self: Twitter =>

    private case class AccountClass(member: Set[Follower]) extends Account {
      def member_=(agent: Set[Follower]) = copy(member = agent)
    }

    def Account(members: Set[Follower] = Set()): Account = AccountClass(members)

    private case class FollowerClass() extends Follower {
    }

    def Follower(): Follower = FollowerClass()
  }

Now, this is the “ugliest” part: we had to provide case classes for all the application traits, and the getters/setters for all of their attributes (standard and non-standard). In this simple example, we just have the “member” attribute, but we may have dozens in a real implementation. This implementation layer must also provide implementations for factory methods, which happen to be the only way to create new entities (note the private declaration of case classes).

The following snippet exercises the above implementation:

  object s extends Twitter with TwitterImpl
  import s._

  val (a, f1, f2) = (Account(), Follower(), Follower())

  // test _=
  assert((a.member = Set()).member == Set())
  assert((a.member = Set(f1, f2)).member == Set(f1, f2))

  // test play
  val a1 = play(a)(f1)
  assert(a1.member == Set(f1))

 Solved using the org.hablapps.updatable package

The major structural change to the above implementation is that we don’t need the case class layer. Thus, we may qualify the following implementation as trait-oriented. Let’s see how the Speech and Twitter layers are modified:

trait Speech {
  trait Interaction {
    type Member <: Agent
    val member: Set[Member]
  }

  implicit val Interaction = weakBuilder[Interaction]

  trait Agent {
  }

  implicit val Agent = weakBuilder[Agent]

  def play[I <: Interaction: Builder](i: I)(a: i.Member): I =
    i.member := i.member + a
}

The first noticeable change is that … we don’t need getters and setters! We just declared our attributes using vals. And the implementation of the play method has not been excessively complicated: we just substituted the “=” operator for the new operator “:=”, and included through its signature evidence that the type parameter I has an implementation of the Builder type class. Instances of this type class can be understood as factories that allow programmers to instantiate and update objects of the specified type in a very convenient way. In particular, the Builder type class enables an implicit macro conversion which gives access to the “:=” operator. All this in a type-safe way.  In a sense, builders play the same role as case classes played in the previous implementation. But there is a crucial difference: builders are created automatically through the builder macro, as shown in the following snippet of the second layer:

  trait Twitter extends Speech {
    trait Account extends Interaction {
      type Member = Follower
    }

    implicit val Account = builder[Account]

    trait Follower extends Agent {
    }

    implicit val Follower = builder[Follower]
  }

The only difference in this layer with respect to the case class implementation is that no method factories are needed, since builders play that role. Now, if you come back to the previous snippet you will also notice weakBuilder invocations for types Interaction and Agent. Certainly, we don’t need strict builders for these types, since they are “abstract”. However, builders also provide attribute reifications, and we certainly want an unique reification for the member attribute. The weakBuilder macro generates the corresponding reifications. The following snippet shows how to access reified attributes, and mimic the functionality included in the case class implementation.

object s extends Twitter
import s._

// test reifications
assert(Account.attributes == List(Account._member))

// create instances
val (a, f1, f2) = (Account(), Follower(), Follower())

// test _=
assert(a.member == Set())
assert(((a.member += f2).member -= f2).member == Set())

// test play
val a1 = play(a)(f1)
assert(a1.member == Set(f1))

println("ok!")

Note that the factory method provided by the Account builder include default parameters as well. These default parameters are defined through the Default type class. The companion object of this type class comes equipped with default values for common Scala types, but you can also provide default values for your own specific types. As you can see, the default value defined for types Set[_] is the empty set.

Concerning the rest of the snippet, we also illustrated the use of the ‘+=’ and ‘-=’ operators. Basically, these operators allow the programmer to specify updates of multivalued attributes specifying only just the element to be added or removed to the collection. To be able to use these operators, the type constructor of the attribute type must implement the Modifiable type class. Currently, the updatable package offers modifiable instances for Option and any kind of Traversable.

But be careful with non-“final” attributes

Let’s suppose that we changed slightly the signature of the play method:

def playAll[I <: Interaction: Builder](i: I)(ags: Set[i.Member]): I =
  i.member := ags

Is this type-safe? Certainly not, since the actual type may have refined the member attribute to a proper Set subtype. For instance, actual type may have overridden the member declaration to a ListSet, while actual argument ags may be a HashSet. The source of this problem is that the member attribute is not “final”, in the sense that it can be overridden. We will consider an attribute as “final” if every component which is part of its declared type is a final class or refers to an abstract type.

We may have forbidden non-final attributes to be used as part of update sentences, but this would rule out the above implementation of the play method, which is perfectly safe: in that case, there was no problem because the ‘+’ operator is defined by the different subtypes of the trait Set. So, we ended up deciding to just emit a warning if non-final attributes are used by updating sentences in a generic context. If you want to eliminate that warning, you can always make the attribute declaration final with the help of new auxiliary abstract types. For instance, look at the following snippet: the member declaration now refers to a new abstract type MemberCol[_], which forces us to change the declaration of the playAll method in such a way that the actual type of the attribute must now be taken into account.

trait Interaction {
  type MemberCol[x] <: Set[x]
  type Member <: Agent
  val member: MemberCol[Member]
}

def playAll[I <: Interaction: Builder](i: I)(ags: i.MemberCol[i.Member]): I =
  i.member := ags

UPDATE: the above snippet has been changed to fix a mistake detected by Eugene Burmako. Thanks Eugene!

In hindsight …

We spent a considerable amount of time in the design and implementation of the updatable package, but it was worth it. The Speech layer is populated by several abstract types with dozens of standard attributes, and making the application programmer to provide getters and setters for them, for each of the application types, is a tough work.

But we also found the updatable library useful for other parts of the Speech platform: for instance, we exploit it to facilitate the serialization of JSON objects, so that we can automatically generate a serializer for buildable types (i.e. instances of the Builder type class). We will tell you about this and other applications of the updatable package in following posts, paying particular attention to macro issues.

But there is still a lot that could be done … besides fixing bugs, of course ;). For instance, we currently require traits to have all of its type members defined in order to generate a builder for it, and it would be nice to relax this constraint. Also, we may extend the updating operator := to cope with nested updates (similarly to what you can achieve with lenses). And we may add support for Union-like types, try to use type macros, etc. We warmly welcome any comment, suggestion for new functionality, corrections, … and any other kind of help. Enjoy it!

Posted in case classes, immutability, Macros, Scala, Speech, Type Class, Updatable | 1 Comment

Welcome message

Think of information systems, Web 2.0 apps, games, e-learning, e-commerce, and the rest of e-* applications. Certainly, these application domains differ significantly in several respects, but can we find some commonalities? We do think so: they deal directly with people; they are deeply concerned with their needs for communication, collaboration and coordination; they all can be regarded as social apps. If we have a look to their functional requirements, we will invariably find people playing different roles, saying things to one another, seeing what is happening, generating and consuming information, and so forth; moreover, we will also notice how the application must take into account different normative concerns: permissions to do something, commitments endorsed by particular role players, monitoring rules, etc.

How are these applications developed nowadays? Programmers mainly use object- and functional-oriented programming languages, such as Java, C#, Scala, Haskell, etc. Yet, functions and objects are quite apart from the kinds of social abstractions we just mentioned above. Similarly, factories, facades, monads, etc., are alien to domain experts and non-IT people. The likely result: a daunting amount of accidental complexity leaked into the code. Clearly, if we want to strive for purely functional code, i.e. software which directly encode the functional requirements of the application, we have to raise the level of abstraction supported by the programming language.

Towards social-oriented programming …

In Habla Computing we are building Speech, a domain-specific programming language for implementing the domain logic of social apps. Speech offers the programmer computational counterparts of social concepts such as roles, speech acts, interaction contexts, commitments, permissions, and the like, which aim at closing the gap between natural-language functional requirements and executable code. The net result of this higher-level of abstraction is a drastic reduction in lines-of-code, development times and cost, as well as a significant increase in quality. Of course, we are not alone in this quest for functional purity: the current landscape abounds with domain-specific proposals that aim at simplifying the implementation of business processes, vertical social networks, etc.

Two features, however, distinguish Speech from BPMS, social networking engines, and other domain-specific technologies. First, Speech is not confined to niche domains; do your functional requirements deal directly with people? If so, you can profit from Speech, regardless of the kind of social process supported by the application. Second, Speech is a language for programmers; it is neither a modeling language for business analysts, nor a suite of tools that cope with each functional concern separately. Certainly, we aim at bridging the gap between non-IT people and programmers, and we claim Speech designs to be directly understandable by functional experts. But programmers need a self-sufficient, cohesive and expressive language. Thus, besides being a domain-specific technology, Speech has been designed as a programming language which takes into account every functional concern in a modular and cohesive way. Moreover, in order to foster adoption of Speech in the programming language community, we’ve decided to offer Speech as an embedded DSL, rather than as a stand-alone implementation.

… embedded in Scala!

There are several very good languages which enable an embedded implementation strategy, but we finally chose Scala. We especially like its mix of functional and object-oriented features, and love its new experimental ones: macros turned out essential for us. This blog will allow us to unveil the major challenges we faced in implementing Speech, and how Scala helped us to solve them. We strive for functional-programming purity in our codebase, so the title of this blog also appeals to that leitmotiv. In this regard, topics that will eventually arise include: coping with updates of immutable objects within generic contexts; dealing with references to evolving immutable objects; etc. Some of these issues lead us to develop general-purpose libraries that we’ve planned to open-source. We are open to suggestions and recommendations for improvements, and any other form of collaboration. Your feedback is really important to us.

We will also use this blog to announce the release of our products. We are working against the clock to launch the beta release of the Speech interpreter in the following weeks. So, we expect to contact you again very soon. Finally, this blog will also give us the opportunity to advertise our participation in different events. One of these events will take place on 10th-13th April at Cambridge, UK: join us at Code Generation 2013!

Posted in Embedded DSLs, Habla Computing, Scala, Speech | Leave a comment