Getting started with scala-native

This is just a quick recollection of the steps I've taken to get my own helloworld running using scala-native. I haven't really started looking at scala-native in detail, but the idea behind it is really nice. Some general input about scala-native can be found here:

• The Github repository: https://github.com/scala-native/scala-native
• Presentation from Scaladays NY: https://github.com/densh/talks/blob/517b20c30dd4aaf390785039cdd002f623eaa91e/2016-05-11-scala-goes-native.pdf
• Twitter to follow: @scala_native

Since scala-native relies on the avalability of LLVM and Clang++ this setup might be different in your environment, than it was in mine. I'm still on OS-X Yosemite:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.10.3
BuildVersion:	14D136

But for Mac users the steps will be pretty much the same. Now lets get started setting up the environment so you can run the sample project provided by scala-native, and we'll create a basic helloworld project as well. Once again, these steps worked in my environment, and yours might be different. There are a couple of github issues you can follow or check for more information on how to set up your environment.

The steps that worked for me were the following:

Checking out the code

We of course first need to get the code, from the command line do the following:

$ git clone --recursive https://github.com/scala-native/scala-native
Cloning into 'scala-native'...
remote: Counting objects: 8295, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 8295 (delta 3), reused 0 (delta 0), pack-reused 8278
Receiving objects: 100% (8295/8295), 1.31 MiB | 94.00 KiB/s, done.
Resolving deltas: 100% (2955/2955), done.

This should, in theory, get you all the required sources. I, however, ran into the fist issue, that the submodele scala wasn't pulled correctly. Somehow git wants to use public-key authentication, instead of using https. The easiest way to quickly solve this, is just do this:

$ cd scala-native/submodules
$ git clone https://github.com/scala/scala
Cloning into 'scala'...
remote: Counting objects: 333614, done.
remote: Total 333614 (delta 0), reused 0 (delta 0), pack-reused 333613
Receiving objects: 100% (333614/333614), 83.31 MiB | 5.63 MiB/s, done.
Resolving deltas: 100% (226320/226320), done.
Checking connectivity... done.
Checking out files: 100% (10492/10492), done.

$ cd scala
$ git checkout 2.11.x

Publish the libraries locally

At this point you'll have all the required sources, but when you try to build it you can possibly run into the following issues:

# from the scala-native directory
$ sbt
> publish-local
... lots of output
[trace] Stack trace suppressed: run last scalalib/compile:compileIncremental for the full output.
[trace] Stack trace suppressed: run last javalib/compile:doc for the full output.
[error] (scalalib/compile:compileIncremental) java.lang.NoClassDefFoundError: scala/scalanative/nscplugin/NirGlobalAddons$nirPrimitives$
[error] (javalib/compile:doc) java.nio.charset.MalformedInputException: Input length = 1

Somehow there seems to be something wrong with some scaladoc stuff which causes our project to fail. Open the build.sbt file and change the javalib project. To solve this we can just disable the scaladoc stuff like by adding the following two lines:

sources in (Compile,doc) := Seq.empty,
publishArtifact in packageDoc := false

To here:

83 lazy val javalib =
84   project.in(file("javalib")).
85     settings(libSettings,
86       sources in (Compile,doc) := Seq.empty,
87       publishArtifact in packageDoc := false
88     ).
89     dependsOn(nativelib)

Now do a local publish again, and that should at least succeed:

> reload
> publish-local
... again, lots of output
[info] 	published ivy to /Users/jos/.ivy2/local/org.scala-native/scalalib_2.11/0.1-SNAPSHOT/ivys/ivy.xml
[success] Total time: 17 s, completed May 14, 2016 4:06:31 PM

At this point, we can try to run the demo application, but you'll probably run into the following two errors: (at least I did)

# This one:
/Users/jos/dev/git/scala-native-clean/scala-native/demo-native/target/scala-2.11/demonative-out.ll:152:40: error: expected '{' in function body
declare double @"llvm.sqrt.f64"(double)
                                       ^

# And this one
rt.cpp:3:10: fatal error: 'gc.h' file not found
#include <gc.h>

Ah... to bad. Something else went wrong. The first error is caused by having an incorrect version of LLVM (3.7 is required, the version I have is 3.6), and the second one is because the Boehm GC libraries aren't available.

Setup LLVM and Clang++ correctly

I don't know what the version is on newer OS-X installations, but mine still had 3.6:

$ clang++ -v
Apple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.3.0
Thread model: posix

To fix this, without messing with possibly other tools depending on 3.6, I used brew to install the 3.7 version of LLVM and Clang++. You might need to do **brew tap homebrew/versions** first, if you can't find llvm37

$ brew install llvm37 --with-clang
==> Installing llvm37 from homebrew/versions
==> Downloading https://homebrew.bintray.com/bottles-versions/llvm37-3.7.1.yosemite.bottle.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/llvm37-3.7.1.yosemite.bottle.1.tar.gz
==> Pouring llvm37-3.7.1.yosemite.bottle.1.tar.gz
==> Caveats
Extra tools are installed in /usr/local/opt/llvm37/share/clang-3.7

To link to libc++, something like the following is required:
  CXX="clang++-3.7 -stdlib=libc++"
  CXXFLAGS="$CXXFLAGS -nostdinc++ -I/usr/local/opt/llvm37/lib/llvm-3.7/include/c++/v1"
  LDFLAGS="$LDFLAGS -L/usr/local/opt/llvm37/lib/llvm-3.7/lib"

You can see the directory where Clang and LLVM are installed. Before we start SBT again, we need to make a symlink so that the plugin can find the correct clang executable.

$ cd /usr/local/opt/llvm37/bin
$ ln -s clang++-3.7 clang++

And finally we just need to update our PATH variable to point to the installation directory before starting SBT:

$ export PATH=/usr/local/opt/llvm37/bin:$PATH
$ clang++ -v
clang version 3.7.1 (tags/RELEASE_371/final)
Target: x86_64-apple-darwin14.3.0
Thread model: posix

At this point we should have one of our problems, so lets just check what happens if we try to run the demo:

# go back to the directory where you cloned scala-native
$ cd ~/dev/git/scala-native-clean/scala-native
$ sbt
> demoNative/run
[info] Updating {file:/Users/jos/dev/git/scala-native-clean/scala-native/}demoNative...
[info] Resolving org.scala-lang#scalap;2.11.8 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jos/dev/git/scala-native-clean/scala-native/demo-native/target/scala-2.11/classes...
warning: overriding the module target triple with x86_64-apple-macosx10.10.0 [-Woverride-module]
1 warning generated.
/Users/jos/.scalanative/rtlib-0.1-SNAPSHOT/rt.cpp:3:10: fatal error: 'gc.h' file not found
#include <gc.h>
        ^
1 error generated.

As you can see, we've got one error less, but there is still something wrong

Install Boehm GC Library

What is missing, and what the error is pointing to, is the Boehm GC library. There are a couple of different ways we can install this. We can use a prepackaged brew instance, or we can just download the sources directly and install Boehm in that way. In this example we'll do the latter. Which means following the instructions from here: http://hboehm.info/gc/

$ mkdir boehm
$ cd boehm
$ git clone https://github.com/ivmai/libatomic_ops.git
$ git clone https://github.com/ivmai/bdwgc.git
$ cd bdgwc
$ ln -s ../libatomic_ops
$ autoreconf -vif
$ automake --add-missing
$ ./configure
$ make
$ sudo make install

Alternatively, you can also install the bdw-gc package in brew.

$ brew install bdw-gc

And after that add the following build.sbt to the demo-native project with additional clang settings:

$ cat build.sbt
nativeClangOptions := Seq("-I/usr/local/Cellar/bdw-gc/7.4.2/include", "-L/usr/local/Cellar/bdw-gc/7.4.2/lib")

I, however, prefer the installing from source* option. At this point we should actually be able to run the demo.

> demoNative/run
warning: overriding the module target triple with x86_64-apple-macosx10.10.0 [-Woverride-module]
1 warning generated.
Rendering (8 spp) 100.00%[success] Total time: 13 s, completed May 14, 2016 6:30:42 PM

Woohoo... success!!! To see that we're actually running the code, we'll make a small change to the source in the demoNative directory. In the file smallpt.scala change the number of samples to 8 like this:

153   final val W = 800
154   final val H = 600
155   final val SAMPLES = 8
156   def main(args: Array[String]): Unit = {

Now run the program again (which should take a lot more time):

> demoNative/run
[info] Compiling 1 Scala source to /Users/jos/dev/git/scala-native-clean/scala-native/demo-native/target/scala-2.11/classes...
warning: overriding the module target triple with x86_64-apple-macosx10.10.0 [-Woverride-module]
1 warning generated.
Rendering (32 spp) 100.00%[success] Total time: 52 s, completed May 14, 2016 6:34:11 PM

It seems to be working. By the way, the result of this program is a 3D rendered scene (the sample program is a raytracer). If you set the samples to a high number, you actually get quite a nice render:

<a href=”/sites/www.smartjava.org/files/scala-native.png” width=700></a>

We can also check if the output is a real native program:

otool -L demonative-out
demonative-out:
	/usr/local/lib/libgc.1.dylib (compatibility version 2.0.0, current version 2.3.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Now lets create an empty helloworld project, where we create our own minimal native-scala program.

Helloworld scala-native

In the scala-native project create a new directory with the name helloworld-native. In that directory create the following file (**helloworld.scala**):

package helloworld

import scalanative.native._, stdlib._

object Main {

  def main(args: Array[String]): Unit = {
    fprintf(__stdoutp, c"Hello World: Scala Native!!!")
  }
}

Before we can run it, we also need to update the main build.sbt file and define this project. To do this, add the following:

lazy val helloworldNative =
  project.in(file("helloworld-native")).
    settings(libSettings).
    settings(
      nativeVerbose := true,
      nativeClangOptions := Seq("-O2")
    ).
    dependsOn(scalalib)

Don't forget the Seq("-I/usr/local/Cellar/bdw-gc/7.4.2/include", "-L/usr/local/Cellar/bdw-gc/7.4.2/lib") for your helloworld-native build definition if you don't install Boehm GC from scratch.

Now reload sbt, and you should be able to run our own scala-native application:

> helloworldNative/run
[info] Compiling 1 Scala source to /Users/jos/dev/git/scala-native-clean/scala-native/helloworld-native/target/scala-2.11/classes...
warning: overriding the module target triple with x86_64-apple-macosx10.10.0 [-Woverride-module]
1 warning generated.
Hello World: Scala Native!!!

Or we can just run it from the command line directly:

$ pwd
/Users/jos/dev/git/scala-native-clean/scala-native/helloworld-native/target/scala-2.11
$ ./helloworldnative-out
Hello World: Scala Native!!!

I hope you can make a start as well with this short introduction. If time permits I've planned to dive a little bit deeper into scala-native, and explore some of its features.