Slick: 5 seconds of hate

Simple reminder why I hate ORM. Some daily experience from trying to make Slick doing something more useful than the examples they generously provide in the very incomplete and hazy documentation (I just wanted to count the rows in some query):

Warning #1:

method count in class ColumnExtensionMethods is deprecated: Use Query.count instead

After some code changes and more esoteric responses from the compiler, I get this:

Warning #2:

method count in class Query is deprecated: Use .length instead of .count

LOL

++ to add more misery to insult, I can't print out the SQL statement that this thing generates to get the row count, because it's now a method of a query! (Of course, there are always the database logs...)

Argh.

Update:  as a result of this rant, I had a quick twitter conversation with @StefanZeiger, who explained that myQuery.length.run will give the result I want, and Query(myQuery.length).selectStatement would produce the SQL, and that in Slick 2.0.0, that's coming out somewhere in September, it will be possible to write myQuery.length.selectStatement directly. That is nice, except that Slick 2.0.0 won't be backwards compatible, which probably means that the ongoing projects might not be able to benefit from the new library, unless people are already using alpha version of 2.0.0 (which we don't).

Nevertheless, good to know that someone's working on the problem :-}

Scala, lists, vectors and mergesort

Today I noticed the following example of mergesort, quoted in the online book "Scala by example" (9.3 Example: Merge sort, page 69):

def msort[A](less: (A, A) => Boolean)(xs: List[A]): List[A] = {

    def merge(xs1: List[A], xs2: List[A]): List[A] =
        if (xs1.isEmpty) xs2
        else if (xs2.isEmpty) xs1
        else if (less(xs1.head, xs2.head)) xs1.head :: merge(xs1.tail, xs2)
        else xs2.head :: merge(xs1, xs2.tail)
    val n = xs.length/2
    if (n == 0) xs
    else merge(msort(less)(xs take n), msort(less)(xs drop n))
}

Further the authors say that the complexity of this algorithm is O(N log (N)), which makes it an attractive option for sorting lists.

Being in an inquisitive mood, I couldn't help wondering how big can in practice be a list sorted with such algorithm. (My current machine is 64-bit Ubuntu with 6 GB RAM and 8 cores)

So here we go.

scala> import scala.util.Random
scala> val randomNums = Seq.fill(1000)(Random.nextInt)
scala> msort((x:Int,y:Int)=>x<y)(randomNums.toList)

res27: List[Int] = List(-2145589793, -2143814602, -2143330861, -2142630038, -2136184047, -2135476301, -2132371275, -2129131922, -2123018872, -2120069375, -2118819114, -2117572993, -2112618055, -2102489605, -2096417279, -2095978655, -2095806565, -2087860343, -2087105558, -2085005596, -2083360810, -2077234330, -2065393243, -2058765966, -2056823240, -2053145149, -2047696716, -2044737011, -1847777706, -18259...

OK, for 1000 elements it works, for 10000 it works too. But let's push one more order of magnitude...

scala> val randomNums = Seq.fill(100000)(Random.nextInt).toList

randomNums: List[Int] = List(186865346, 638415825, -637620864, 220723809, -1536234831, 1710286185, 126091472, -1621728642, -1819749330, -294195052, -613979926, 1278841478, -111715804, -1953497441, 1891679544, 582175290, 1555531003, -430520072, 652471392, 1211722008, -112446234, -1900260621, 2058382521, 564201400, -1225275015, 2069052362, 797097978, 1077363576, 1469066877, -303059738, -166855116, -1876385701, 285630983, -1956550564, -1991336959, -1713232594, -868759609, -723403847, -282664963, 1965397484, 1563483549, -618177790, -297223307, -197365661, -703715983, 28207094, -1793590690, 374050582, 992041027, -1931269739, -932512120, -1551657371, 1523463808, -742246427, -1665172973, 50892779, -286029416, -1654054925, -874783455, -1825744857, -571856180, 289326103, -215127347, -1488600483,...

scala> msort((x:Int,y:Int)=>x<y)(randomNums)

java.lang.StackOverflowError

at .merge$1(<console>:20)

at .merge$1(<console>:22)

at .merge$1(<console>:20)

at .merge$1(<console>:20)

at .merge$1(<console>:20)

at .merge$1(<console>:20)

at .merge$1(<console>:22)

        ...

Oi wei, we ran out of stack!

The reasons are obvious: this line of code

xs2.head :: merge(xs1, xs2.tail)

means that all the time while you are busy merging the partial lists, there are orphan elements hanging around, waiting until they are allowed to be the head. And the stack isn't made of rubber!

Well... there is some sport, of course, could we make it at least work? Using the tail recursion, perhaps? (If you have forgotten, tail recursion happens when the recursive function ends with calling itself. Then the compiler can be smart enough to make a cycle instead of the sequence of the recursive calls, at least if you ask it politely :) )

import scala.annotation.tailrec
...
@tailrec
def merge(xs1: List[A], xs2: List[A]): List[A] =
   if (xs1.isEmpty) xs2
       else if (xs2.isEmpty) xs1
       else if (less(xs1.head, xs2.head)) xs1.head :: merge(xs1.tail, xs2)
       else xs2.head :: merge(xs1, xs2.tail)

However, this won't compile, and rightly so:

[error] 

<filename>: could not optimize @tailrec annotated method merge: it contains a recursive call not in tail position

[error] else xs2.head :: merge(xs1, xs2.tail)

Ah right. Operator :: is right associative, that means it's called after merge(arg1, arg2) was called, therefore tail recursion isn't possible.

Well, without further ado, that is the variant I have ended up with. It worked in the sense that it didn't break - there was now a cycle - but for the list with 100000 elements it took about 5 minutes to do the sorting!

import scala.math.Ordering.Implicits._ 
import scala.util.Random 
import scala.annotation.tailrec

object Sorting {

def less[T:Ordering](x:T,y:T) = x<y

@tailrec

def merge[A:Ordering ](xs0: List[A], xs1: List[A], xs2: List[A]): List[A] =

if (xs1.isEmpty) if (xs0 == Nil) xs2 else xs0 ::: xs2

else if (xs2.isEmpty) if (xs0 == Nil) xs1 else xs0 ::: xs1

else if (less(xs1.head, xs2.head)) merge(xs0 :+ xs1.head, xs1.tail, xs2)

else merge(xs0 :+ xs2.head, xs1, xs2.tail)

def msort[A:Ordering](xs: List[A]): List[A] = {

val n = xs.length/2

if (n == 0) xs

else merge(List[A](), msort(xs take n), msort(xs drop n))

}

def checkSorted[A:Ordering](x:A, xs:List[A]):Boolean = xs match {

case Nil => true

case head::tail => if (head < x) false else checkSorted(head, tail)

}

def test(size:Int, printAll:Boolean = true) = {

val randomNums = Seq.fill(size)(Random.nextInt).toList

val sorted = msort(randomNums)

if (printAll) println(sorted)

if (!checkSorted(sorted.head, sorted.tail)) println("Not sorted!!!")

}

}

The reason why this code is slow is in the fact that now we have added the accumulator, and we want to add elements to the end of it. However, List (as also mentioned here) only guarantees the quick access to the head - the complexity of accessing the last element would be proportional to the size of the list!

So is there anything to do to make it faster, without trying hard to imagine some other algorithm? My solution was to use Vectors instead of Lists.

object Sorting {
def less[T:Ordering](x:T,y:T) = x<y
@tailrec
def merge[A:Ordering ](xs0: Vector[A], xs1: Vector[A], xs2: Vector[A]): Vector[A] =
if (xs1.isEmpty) if (xs0 == Nil) xs2 else xs0 ++ xs2
else if (xs2.isEmpty) if (xs0 == Nil) xs1 else xs0 ++ xs1
else if (less(xs1.head, xs2.head)) merge(xs0 :+ xs1.head, xs1.tail, xs2)
else merge(xs0 :+ xs2.head, xs1, xs2.tail)

def msort[A:Ordering](xs: Vector[A]): Vector[A] = {
val n = xs.length/2
if (n == 0) xs
else merge(Vector[A](), msort(xs take n), msort(xs drop n))
}
def checkSorted[A:Ordering](x:A, xs:List[A]):Boolean = xs match {
case Nil => true
case head::tail => if (head < x) false else checkSorted(head, tail)
}
def test(size:Int, printAll:Boolean = true) = {
val randomNums = Seq.fill(size)(Random.nextInt).toVector
val sortedVec = msort(randomNums)
if (printAll) println(sortedVec)
val sorted = sortedVec.toList
if (!checkSorted(sorted.head, sorted.tail)) println("Not sorted!!!")
}

}

Vectors guarantee constant access to any element they contain. It is probably slightly slower than accessing the head of the list, but for the tasks like this one it seems to be a definitely better option. It took very little time to sort both 100000 and 1000000 vectors of random integers. With 10000000, however, it still takes quite some time and apparently more memory, but 10 millions is already more than the amount of rows in many decent tables :-)

I hope it was useful, or at least that you did rid to the end. I still want to write a C++ implementation and see which one finishes first. So may be there will be the part two...

A Voyage into the Scala Reflection land, part 2: case or no case

Let's make a little module (using :paste mode of the console or by compiling it separately and importing the corresponding module) - it looks a bit weird but suitable enough for what I want to demonstrate:

trait DemoStuff {
  lazy val i = 0 
  lazy val s = ""
  lazy val printMe = s"${s},${i}"
}

case object CaseEnglish extends DemoStuff {
  override lazy val i = 1 
  override lazy val s = "Hello"
}

case object CaseGerman extends DemoStuff {
  override lazy val i = 2 
  override lazy val s = "Guten TaG"
}

object NonCaseOne extends DemoStuff {
  override lazy val i = 3 
  override lazy val s = "Bye"
}

(Regarding the usage of lazy vals, I was inspired by this explanation).

In theory, both "normal" objects and case objects in Scala are supposed to be singletons. However, as far as serialization is concerned, it seems that case objects are more singletons than their non-case siblings

scala> val nco = NonCaseOne
nco: NonCaseOne.type = NonCaseOne$@74232853

scala> val ce = CaseEnglish
ce: CaseEnglish.type = CaseEnglish

This mimics the same conventions for the classes versus case classes.

Let's go on with reflection. Assume that, like in the previous case, we imported the universe, the current mirror and defined the helper to get the type tag for us.

scala> val ceType = getTypeTag(ce)

ceType: reflect.runtime.universe.Type = CaseEnglish.type

Let's look at the base classes for this type.

scala> val baseClasses = ceType.baseClasses
baseClasses: List[reflect.runtime.universe.Symbol] = List(object CaseEnglish, trait Serializable, trait Serializable, trait Product, trait Equals, trait DemoStuff, class Object, class Any)

I am still not sure why train Serializable is listed twice in these cases, but let's look at the symbol for DemoStuff, which is, as we know, the direct parent of our objects.

scala> val dsSymbol = baseClasses.filter(_.name.toString=="DemoStuff").head
res9: reflect.runtime.universe.Symbol = trait DemoStuff

Interesting to note that both ceType.typeSymbol and dsSymbol are seen as classes:

scala> ceType.typeSymbol.isClass
res12: Boolean = true

scala> dsSymbol.isClass

res14: Boolean = true

Apparently, there is a class implicitly defined for every object. We can get the symbols for the corresponding objects by asking for the companion of the original symbol, and see that one of them is considered to be a class and another isn't, also that one of them is described as "module" and another as "module class":

scala> val ceoSymbol = ceType.typeSymbol.companionSymbol

ceoSymbol: reflect.runtime.universe.Symbol = object CaseEnglish

scala> ceoSymbol.isClass

res16: Boolean = false

scala> ceoSymbol.isModule

res20: Boolean = true

scala> ceoSymbol.isModuleClass

res20: Boolean = false

scala> ceType.typeSymbol

res17: reflect.runtime.universe.Symbol = object CaseEnglish

scala> ceType.typeSymbol.isClass

res18: Boolean = true

scala> ceType.typeSymbol.isModule

res21: Boolean = false

scala> ceType.typeSymbol.isModuleClass

res21: Boolean = true

There is no companion symbol for a trait, however:

scala> dsSymbol.companionSymbol

res19: reflect.runtime.universe.Symbol = <none>

Now the question: if the dsSymbol is a class, can't we get its successors?

scala> val descendants = dsSymbol.asClass.knownDirectSubclasses

descendants: Set[reflect.runtime.universe.Symbol] = Set()

No luck? Now let's make the trait DemoStuff sealed, reload the example and do the same steps:

sealed trait DemoStuff {

  lazy val i = 0 

...(the rest of the code and the reflection steps are the same)...

scala> val dsSymbol = ru.typeOf[DemoStuff]

dsSymbol: reflect.runtime.universe.Type = DemoStuff

scala> val descendants = dsSymbol.asClass.knownDirectSubclasses

descendants: Set[reflect.runtime.universe.Symbol] = Set(object CaseEnglish, object CaseGerman, object NonCaseOne)

Yay, look what we've got! Just like with the good old enums, we can see them all, because for the sealed trait, all its direct descendants should be defined in the same module, so the parent class knows about them all.

Now suppose we want to instantiate all such objects. One way of using all this info would be to map the types and some inside information - e.g. val i in our case - so that we could mimic accessing the enums by value. Of course, if all elements are case objects, we can also just map the types to their string representations.

There is one right way and one wrong way to instantiate objects. Let's do the right way first.

In implies getting object companions and converting each of them from Symbol to ModuleSymbol:

scala> val descendantObjects = descendants map (m=> if (m.isModuleClass) m.companionSymbol else m) map (_.asModule)

descendantObjects: scala.collection.immutable.Set[reflect.runtime.universe.ModuleSymbol] = Set(object CaseEnglish, object CaseGerman, object NonCaseOne)

Finally, we can use the mirror to turn these symbols into instances:

scala> val reflectedObjects = descendantObjects map (m=>mirror.reflectModule(m).instance.asInstanceOf[DemoStuff])

reflectedObjects: scala.collection.immutable.Set[DemoStuff] = Set(CaseEnglish, CaseGerman, NonCaseOne$@74232853)

They do the right things, and they are equal to the "normally" created instances:

scala> reflectedObjects map (_.printMe)

res25: scala.collection.immutable.Set[String] = Set(Hello,1, Guten TaG,2, Bye,3)

scala> reflectedObjects.head == ce

res26: Boolean = true

Now, the wrong way! (DANGER!)

Suppose we go ahead with classes, not bothering to get the object companions, and try to instantiate them using constructors:

scala> val dsType = ru.typeOf[DemoStuff]

dsType: reflect.runtime.universe.Type = DemoStuff

scala> val descendants = dsType.typeSymbol.asClass.knownDirectSubclasses

descendants: Set[reflect.runtime.universe.Symbol] = Set(object CaseEnglish, object CaseGerman, object NonCaseOne)

scala> val ctors = descendants map (_.typeSignature.member(ru.nme.CONSTRUCTOR))

ctors: scala.collection.immutable.Set[reflect.runtime.universe.Symbol] = Set(constructor CaseEnglish, constructor CaseGerman, constructor NonCaseOne)

So let's reflect and instantiate these classes:

scala> val reflected = descendants map (m=>mirror.reflectClass(m.asClass))

reflected: scala.collection.immutable.Set[reflect.runtime.universe.ClassMirror] = Set(class mirror for CaseEnglish (bound to null), class mirror for CaseGerman (bound to null), class mirror for NonCaseOne (bound to null))

scala> val stuff = (reflected zip ctors) map (m=> m._1.reflectConstructor(m._2.asMethod))

stuff: scala.collection.immutable.Set[reflect.runtime.universe.MethodMirror] = Set(constructor mirror for CaseEnglish.<init>(): CaseEnglish.type (bound to null), constructor mirror for CaseGerman.<init>(): CaseGerman.type (bound to null), constructor mirror for NonCaseOne.<init>(): NonCaseOne.type (bound to null))

scala> val instances = stuff map (_.apply().asInstanceOf[DemoStuff])

instances: scala.collection.immutable.Set[DemoStuff] = Set(CaseEnglish, CaseGerman, NonCaseOne$@661a6677)

scala> instances map (_.printMe)

res1: scala.collection.immutable.Set[String] = Set(Hello,1, Guten TaG,2, Bye,3)

Looks about right? But... what if we compare the newly instantiated CaseEnglish with the our old friend val ce? They should be the same, right?

scala> instances.head == ce

res2: Boolean = false

OOPS. We managed to create those shadow classes for our, supposedly one and unique, case objects. Congratulations, what a bummer!

That concludes the story for now... but you'll never know what the future brings. Hope it was useful :)

Voyage into the Scala reflection land, part 1.

Recently I have played a little with Scala reflection capabilities. Hence I would like to share what I have learned.

Welcome to Scala version 2.10.2 (OpenJDK 64-Bit Server VM, Java 1.7.0_21).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import reflect.runtime.{universe=>ru}
import reflect.runtime.{universe=>ru}
scala> import reflect.runtime.{currentMirror=>mirror}

First step: import the "experimental" reflection library and assign a tag to it (this way it's easy to see in code where this library has been used. Also, import the "root" mirror through which you, like Alice, can enter the reflection world. Another way to obtain the "root" mirror would be by calling:

scala> val otherRootMirror = ru.runtimeMirror(getClass.getClassLoader)

otherRootMirror: reflect.runtime.universe.Mirror = JavaMirror with scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@59a10521 of type class scala.tools.nsc.interpreter.IMain$TranslatingClassLoader with classpath [(memory)] and parent being scala.tools.nsc.util.ScalaClassLoader$URLClassLoader@5a57e77f of type class scala.tools.nsc.util.ScalaClassLoader$URLClassLoader with classpath [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar,file:/usr/local/share/scala/scala-2.10.2/lib/akka-actors.jar,f...

Now, let's do an easy thing: define a case class and inspect its instance with reflection.

scala> case class ReflectDemo(intVal: Integer, stringVal:String) {

     | def multiply(byThat: Int): Int = intVal*byThat

     | def reverse(): String = stringVal.reverse

     | }

defined class ReflectDemo

scala> val rd = ReflectDemo(5, "five")

rd: ReflectDemo = ReflectDemo(5,five)

Let's get the type symbol for the object we are using. The docs about the Scala reflection suggest defining a helper in the following way:

scala> def getTypeTag[T:ru.TypeTag](obj:T) = ru.typeOf[T]

getTypeTag: [T](obj: T)(implicit evidence$1: reflect.runtime.universe.TypeTag[T])reflect.runtime.universe.Type

Now we can get the type tag of our object!

scala> val typeTag = getTypeTag(rd)

typeTag: reflect.runtime.universe.Type = ReflectDemo

It opens a lot of possibilities...

scala> typeTag.

=:=                 asInstanceOf        asSeenFrom          baseClasses         baseType            contains            

declaration         declarations        erasure             exists              find                foreach             

isInstanceOf        map                 member              members             normalize           substituteSymbols   

substituteTypes     takesTypeArgs       termSymbol          toString            typeConstructor     typeSymbol          

weak_<:<            widen               

For example, we can use it to get all the base classes for this type:

scala> typeTag.baseClasses

res10: List[reflect.runtime.universe.Symbol] = List(class ReflectDemo, trait Serializable, trait Serializable, trait Product, trait Equals, class Object, class Any)

We can get the type name:

scala> typeTag.typeSymbol.name

res18: reflect.runtime.universe.Name = ReflectDemo

And all the members (defined in all base classes up to the very generic ones:

scala> typeTag.members

res8: reflect.runtime.universe.MemberScope = Scopes(method equals, method toString, method hashCode, method canEqual, method productIterator, method productElement, method productArity, method productPrefix, method copy$default$2, method copy$default$1, method copy, method reverse, method multiply, constructor ReflectDemo, value stringVal, value stringVal, value intVal, value intVal, method $init$, method $asInstanceOf, method $isInstanceOf, method synchronized, method ##, method !=, method ==, method ne, method eq, constructor Object, method notifyAll, method notify, method clone, method getClass, method wait, method wait, method wait, method finalize, method asInstanceOf, method isInstanceOf, method !=, method ==)

The other method, declarations, seems to omit the generics:

scala> typeTag.declarations

res9: reflect.runtime.universe.MemberScope = SynchronizedOps(value intVal, value intVal, value stringVal, value stringVal, constructor ReflectDemo, method multiply, method reverse, method copy, method copy$default$1, method copy$default$2, method productPrefix, method productArity, method productElement, method productIterator, method canEqual, method hashCode, method toString, method equals)

In any case, we can use one of these to get the list of all the variables defined in the object of this type:

scala> val variables = typeTag.members.filter(!_.isMethod)

variables: Iterable[reflect.runtime.universe.Symbol] = SynchronizedOps(value stringVal, value intVal)

Now, back to the mirrors and stuff. Let's use the root mirror to get the InstanceMirror for the runtime object which we are inspecting.

scala> val instanceMirror = mirror.reflect(rd)

instanceMirror: reflect.runtime.universe.InstanceMirror = instance mirror for ReflectDemo(5,five)

We can use this mirror to reflect any of the variables belonging to that object, using the Symbols we obtained earlier:

scala> variables.map(m=>instanceMirror.reflectField(m.asTerm))

res12: Iterable[reflect.runtime.universe.FieldMirror] = List(field mirror for ReflectDemo.stringVal (bound to ReflectDemo(5,five)), field mirror for ReflectDemo.intVal (bound to ReflectDemo(5,five)))

Ultimately, we can obtain the values for these variables:

scala> val values = variables.map(m=>instanceMirror.reflectField(m.asTerm).get)

values: Iterable[Any] = List(five, 5)

Also, we can use the same Symbols to get access to the variables' types:

scala> val types = variables.map(_.typeSignature)

types: Iterable[reflect.runtime.universe.Type] = List(String, java.lang.Integer)

Now, if we combine all this, we get the whole definition of the variable's properties:

scala> (variables zip( types zip values)).toMap

res17: scala.collection.immutable.Map[reflect.runtime.universe.Symbol,(reflect.runtime.universe.Type, Any)] = Map(value stringVal -> (String,five), value intVal -> (java.lang.Integer,5))

This can be used, for example, if you get a fancy to create a wrapper for persisting the given object into the database based solely on its structure. You can use the type information to create the table structure and then use the values of the given objects to populate the rows. (I did something like that for someone recently, as an experimental prototype. It did work :) ).

Update: How about creating new instances of classes during reflection? Well, let's try to create a new instance of our ReflectDemo class.

The original input, then, would be the type and the map of variables, where the names point to the values (in principle, we could also reflect the type of every variable and use the type info to check if the variables match the expectations of the constructor...)

scala> val values = Map("intVal"->5, "stringVal"->"five")
values: scala.collection.immutable.Map[String,Any] = Map(intVal -> 5, stringVal -> five)

scala> val rdType = ru.typeOf[ReflectDemo]
rdType: reflect.runtime.universe.Type = ReflectDemo

Let's reflect the class:

scala> val rdClass = mirror.reflectClass(rdType.typeSymbol.asClass)
rdClass: reflect.runtime.universe.ClassMirror = class mirror for ReflectDemo (bound to null)

The result is of the type ClassMirror and it can be used to invoke the constructor for the given class.

We can get all constructors for a class from its Type information:

scala> val constructors = rdType.members.filter(m=>m.isMethod && m.asMethod.isConstructor)

constructors: Iterable[reflect.runtime.universe.Symbol] = SynchronizedOps(constructor ReflectDemo, method $init$, constructor Object)

In our case, we know we only have one constructor, and we can also access it using the predefined name tag:

scala> val defaultCtor = rdType.member(ru.nme.CONSTRUCTOR)
defaultCtor: reflect.runtime.universe.Symbol = constructor ReflectDemo

Let's have a look which parameters it expects, now that's the piece of magic you just have to know (took me some research :) ) :

scala> val paramsList = defaultCtor.asMethod.paramss
paramsList: List[List[reflect.runtime.universe.Symbol]] = List(List(value intVal, value stringVal))

I suppose that this reflects the fact that one can invoke the constructor with various sets of parameters.
In our case we only have one constructor, so let's to the easy way. If there would be more than one constructor, we would have to use the name and type information from the available values in order to find out the suitable one, which is the exercise I leave for the curious reader (as such things go :) )

scala> val params = paramsList.head

params: List[reflect.runtime.universe.Symbol] = List(value intVal, value stringVal)
scala> val mappedValues = params map (m=>values(m.name.toString))
mappedValues: List[Any] = List(5, five)

Now we are almost there. Use the ClassMirror to reflect the constructor:

scala> val runtimeCtor = rdClass.reflectConstructor(defaultCtor.asMethod)

runtimeCtor: reflect.runtime.universe.MethodMirror = constructor mirror for ReflectDemo.<init>(intVal: java.lang.Integer, stringVal: String): ReflectDemo (bound to null)

MethodMirror provides the method apply(args: Any*) which allows to supply a sequence of any kind of values. However, it won't accept the List, because it expects a Seq, so it will consider the List to be a single parameter, and complain loudly:

scala> runtimeCtor.apply(mappedValues)

java.lang.IllegalArgumentException: wrong number of arguments

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaConstructorMirror.apply(JavaMirrors.scala:444)

....

But (that's another little piece of magic) one can easily make a sequence from the List:

scala> val newRD = runtimeCtor.apply(mappedValues:_*)
newRD: Any = ReflectDemo(5,five)

Remember the instance of the same class we had in the beginning? Let's compare them and prove that we could recreate the same instance!

scala> rd == newRD

res24: Boolean = true

Yay! 

Well, that's enough for now. Next topic (if I get to that): the specifics of case classes and objects, and the dangers of their instantiating via reflection :) By the way, in this post, the case class could also be the normal class - nothing would have changed except that you'd create it with New. The ways you instantiate (case) objects and (case) classes via reflection, though, are very different, and should be used with some care! To be continued :)

Customizing dict, or Offline dictionary from command line in Ubuntu

This weekend I spent quite some time setting up the dictionary in Ubuntu. My goal was: being able to easily get a translation between the languages I want, and if possible, offline.

I already knew that there is a Dictionary program, which is available in Ubuntu by default or can be easily installed, plus a gnome-dictionary plugin to easily invoke it from the top panel. I had two main problems with this:

1. My native language, Russian, was not available by default as a target language.
2. I wanted to be able to use the dictionary also when I am offline (which can happen when one has a laptop).

In addition, I would really love to use the dictionary without a client, just from command line.

How I went solving this problem:

1. It appears there are better clients for Ubuntu, supporting both dict and other formats. I used StarDict before (plugging in extra dictionaries found elsewhere in the internet) but its interface is rather messed-up, at least for me (while I was on Windows, I used Babylon and it was practically just what I needed). On Ubuntu, recently I discoveblue Fantasdic and it seems to be at least much neater than Stardict; plus, it can itself import dictionaries in other formats (StarDict among them), so it was already an improvement.
2. Then I found out that it's possible to install local dictd server and let it provide the dictionaries.

After some experiments and poking around the net, I have got dictd up and running and could import some extra dictionaries to it. Here are the steps, meant more as an inspiration than as a cookbook :)

• You can get an idea what is available from the repos you use by typing something like:
apt-cache search dict- As a result, you will see something like:

dict-jargon - dict package for The Jargon Lexicon
dict-freedict-afr-deu - Dict package for Afrikaans-German Freedict dictionary
dict-freedict-iri-eng - Dict package for Irish-English Freedict dictionary
[...skipped...]
dict-freedict-tur-eng - Dict package for Turkish-English Freedict dictionary
dict-freedict-wel-eng - Dict package for Welsh-English Freedict dictionary
stardict-common - International dictionary - data files
stardict-czech - Stardict package for Czech dictionary of foreign words
stardict-english-czech - Stardict package for English-Czech dictionary

Most packages starting with "dict-" will be the dictionaries in the dictd format. The naming scheme, though, is not strict (for example, English-Russian Mueller dictionary is called mueller7-dict and Moby Thesaurus is called dict-moby-thesaurus) but you get the idea. Otherwise, you can just look them up in Synaptic package manager.
• To install dictd and the additional packages, you can either add them via Synaptic package manager or just apt-get them:

apt-get install dictd dict-gcide dict-wn dict-moby-thesaurus [whatever else dictionaries you want]

Additional dict packages can always be added later.
• If installation succeeded, you will have your dictd service up an running locally on port 2628! You can check it by typing:

/etc/init.d/dictd status

You should get:

* dictd is running

Also, now you can type something like:

dict athwart

and get results:

6 definitions found

From The Collaborative International Dictionary of English v.0.48 [gcide]:

Athwart \A*thwart"\, prep. [Pref. a- + thwart.]
1. Across; from side to side of.
[1913 Webster]

Athwart the thicket lone.             --Tennyson.
[1913 Webster]

2. (Naut.) Across the direction or course of; as, a fleet
standing athwart our course.
[...skipped...]
From Mueller English-Russian Dictionary [mueller7]:

athwart
[ɜ↗θwɘ:t]
1. _adv.
1) косо; поперёк; перпендикулярно
2) против; наперекор
2. _prep.
1) поперёк; через; to run athwart a ship врезаться в борт другого судна;
to throw a bridge athwart a river перебросить мост через реку
2) против; вопреки; athwart his plans вопреки его планам

If you want to use the client (like Dictionary or Fantasdic) you can set up your local dictd server as the source there: in preferences, add new source of type "DICT dictionary server", specify "127.0.0.1" as the server address and leave the port number unchanged (2628).
• In the previous example, I have cheated a bit: you will get less results, because I have put in a couple of additional dictionaries already, converted into dictd format. The reason for hat was that not all dictionaries I needed were available in dictd format, but they could be found in other formats (stardict, sdict, dsl): for example, look here or here (I suspect that the first list is just the combination of all entries from the second one, not sure).
The second link also points to the home of XDXF project, where you can get a program called makedict to convert the dictionaries between different formats. This program is not available in the binary form to install, so you can clone the source and build it yourself with standard steps:

cd [someplace]
mkdir xdxf
svn co https://xdxf.svn.sourceforge.net/svnroot/xdxf xdxf
mkdir makedict-out
cd makedict-out
cmake ../xdxf/trunk
make
make install 

After this, you can convert the dictionaries (at least in sdict, stardict and xdxf formats - haven't tried the others) to dictd format using

makedict -o dictd file-name

• Finally, a couple of import examples.
1. For example, suppose you have downloaded English-German dictionary.
You will get a file comn_sdict_axm05_English_German.tar.bz2 in bzip format, and can proceed as follows:

tar -xvjf comn_sdict_axm05_English_German.tar.bz2

English_German/
English_German/icon16.png
English_German/dict.xdxf

So, this is an xdxf format. We don't have to specify it explicitly, specifying output format is enough:

makedict -o dictd English_German/dict.xdxf 

Write index to English_German/English_German/English_German.index
Write data to English_German/English_German/English_German.dict

The resulting two files have to be put together with other dictd files (on my machine they dwell in /usr/share/dictd folder by default), the dictd config should be updated and the dictd service should be restarted: mv English_German/English_German/*.* /usr/share/dictd /usr/sbin/dictdconfig --write /etc/init.d/dictd restart Now you should be able to see new dictionary in your client or just check its availability from the terminal:

dict --dbs

It will provide a list which should contain the new source (usually named after the file name).

Databases available:
 gcide           The Collaborative International Dictionary of English v.0.48
 wn              WordNet (r) 3.0 (2006)
[...skipped...]
 English_German  English_German
 fd-eng-fra      English-French Freedict dictionary
 rus_eng_full    rus_eng_full

And it should just work:

dict athwart

[...skipped...]
From English_German [English_German]:

  <k>athwart<k>
  quer

(Yes, there might be some specific tags which don't look pretty from terminal; they can be removed if needed - the file is just plain text - but that's outside of the current topic). According to the Wiki article about Dict, there is another program formatting text files into .dict and .index files, called dictfmt. I tried using it to format a file in text format generated from dicts.info page, but the format of these text files does not seem to be what dictfmt expects. I didn't spent much time on it yet.
2. The procedure has an additional extra step for the files in stardict format, for example Dutch-English one.
After unpacking the file, we get the following structure:

dutch-english.dict.dz  dutch-english.idx  dutch-english.ifo

The converter will complain, because it expects non-compmressed dict file. The additional step is uncompressing:

dictzip -d dutch-english.dict.dz

Which will give us:

ls stardict-dutch-english-2.4.2
dutch-english.dict  dutch-english.idx  dutch-english.ifo
makedict -o dictd stardict-dutch-english-2.4.2/dutch-english.ifo
Write index to stardict-dutch-english-2.4.2/dutch-english/dutch-english.index
Write data to stardict-dutch-english-2.4.2/dutch-english/dutch-english.dict

3. Another caveat is the index. If the index entries contain anything else than words (lexical definitions, hyphens, etc), then these entries won't be matched with a default search, but can be matched using a different search stragegy.

dict -d English_German ceiling
1 definition found

From English_German [English_German]:

  <k>ceiling<k>
  Höchstbetrag {m}, Obergrenze {f}, Zimmerdecke {f}

dict -d English_German -s suffix ceiling

From English_German [English_German]:

  <k>(absolute) ceiling<k>
  Gipfelhöhe {f} (Luftfahrt)

From English_German [English_German]:

  <k>asset ceiling<k>
  Höchstgrenze {f}

From English_German [English_German]:

  <k>ceiling<k>
  Höchstbetrag {m}, Obergrenze {f}, Zimmerdecke {f}

[...skipped...]

That's it for the start! Might not look extremely fancy, but... it's a free horse after all :)

UPDATE: if you have Babylon dictionaries (.BGL), you can convert them into dictd format using (available from Ubuntu distro) program called dictconv.

Ubuntu, renaming files recursively

As a result of researching how one can quickly (in one-liner) rename, on Linux (=Ubuntu in my case) all .JPG files in a directory tree to .jpg files, the following solution was found (borrowed from a discussion here after fixing the typo's :) ):

 find . -name *.JPG -exec rename 's/\.JPG$/\.jpg/i' {} +

It won't work on Mac though (a Mac user offered this one):

for i in `find . -name '*.JPG'`; do mv $i ${i/%JPG/jpg}; done

Thought it might be worth remembering :)

Ubuntu 11.04, VMWare Player and coolah scrollbars

After feeling myself very miserable, because VMWare Player was "crashing" after trying to play any virtual machine on Ubuntu 11.04 ("crashing in the sense that it was still running in the background, but the screen was gone), I accidentally found some helpful info where somebody with the similar problem mentioned that he had to downgrade overlay-scrollbar package from version 0.1.9 to 0.1.7.

This solution didn't work for me (because I had fresh Ubuntu 11.04 install), but after starting Synaptic Package Manager, I could see that there is a newer version of overlay-scrollbar package available (0.1.12) and decided to give it a try. Miracle happened - VMWare Player started to work normally after I performed the upgrade. (It took several hours of trying to find out what was happening, and I strongly dislike those new fancy scrollbars - now even more so!.. :) )

PS I have also found one other mentioning of VMWare problem with the latest Ubuntu here, but their solution didn't seem to help.