C Foreign Function Interface

Introduction

This tutorial demonstrates how to call external C/C++ code from a haxe application.
In many ways, the system is similar to the neko C Foreign Function Interface. The main difference is that "value" types are treated as "opaque".
An example can be found at the bottom of this page.

The "value" Type

All arguments and return data used for communicating between the Haxe compiled code and the native library code are declared as type "value". On the Neko target, this is a very particular data structure, and the type of element that is referred to by it can be determined by looking directly at the bits in the value variable. This means that in order for the C++ traget to be able to use the Neko ndll files directly, it would have to convert to and from these Neko values for each CFFI call. For efficiency reasons, the C++ target does not do this. Instead, as far as the external library is concerned, the value type is a generic pointer to an unknown type.

The Haxe C++ code actually knows that the value type is a pointer to its "hx::Object" structure, from which all classes derive, so the conversion for classes is very simple and fast. For the "non-class" types (Int, Bool, Float, String) the conversion requires the value to be "boxed" inside an object in order to be passed to/from the CFFI. This is exactly the same procedure that is used for passing "Dynamic" objects within the Haxe C++ code. As a consequence of this, passing non-class objects to C++ CFFI incurs a small overhead per call. The special case is the "null" value, which is encoded as the null pointer and is very fast - if your functions have nothing better to return, your should return the null value (alloc_null()).

Since the actual type of "value" variables is not known by the external code, it is very easy for this external code work with neko too. When the external code requests the "integer value of this variable", val_int(x), the external code calls a function that will ultimately call the Neko version of val_int, if the ndll is loaded into a Neko VM, or the C++ version if it is loaded into a C++ program. This extra function call represents a small overhead when running Neko code like this, however the benefits of being able to use the same ndll on both Neko and C++ targets usually makes this trade-off worth while.

One ndll to bring them all and in the darkness bind them

The C++ ndll files use a technique similar to the windows "delay load" dll binding, along with some macro magic, to allow them to be built without needing to linked against a specific dll or library file. The host application (neko or the hxcpp binary) must provide a single export symbol, "hx_cffi", which the ndll file will query for function pointers (this is done by all hxcpp applications). If this is not found, then it is assumed that the executable is neko, and tries to load the dll "nekoapi.dll". This dll exports the "hx_cffi" symbol and is linked against the standard neko cffi library, which requires the neko.dll file to be in the process, This nekoapi library provides the translation from the opaque pointers to the neko values. There is no complex logic the determine the location of this nekoapi file, so it generally must be in your library search path (executable search path on windows). The best way to ensure this is to place it next the neko.dso (neko.dll) that is being used.

The C++ FFI is available when you "#include <hx/CFFI.h>", which is distributed with the hxcpp haxelib module. If you are using a makefile, you can find the current location using something like "haxelib path hxcpp". If you are using Visual Studio, it is probably easiest just to hardwire the location in. Because the system uses opaque values, you don't necessarily need to update your ndll every time you update haxe or the hxcpp module. You may also consider using the hxcpp build-tool to build your ndlls - more on this later.

When you include the header file, the CFFI API functions get defined as function pointers. In exactly one of your library files, you should "#define IMPLEMENT_API" before you "#include <hx/CFFI.h>. This initializes each function pointer with a "bootstrap" function that will attempt to find the actual function via querying the "hx_cffi" function from the host application. Once found, the function pointer is moved from the bootstrap function to the actual function, meaning that subsequent calls will go straight to the actual function. This magic is mostly handled by the code in hx/CFFILoader.h, and in conjunction with hx/CFFIAPI.h, which defined the whole API. Using macros like this means that no link-library is required - it is done in the header files.

Static Linking

When generating a static library (as required by iPhone), instead of a dynamic ndll, some care must be taken to ensure the "DEFINE_PRIM" macro works correctly. Most linkers today will only pull symbols from a static library if there exist in an obj file (ie, c++ file) that has other symbols explicitly required by the application. Since the CFFI functions are located by name, the linker does not explicitly know they are required, so the exporting DEFINE_PRIM macros will not be linked, and will not export their functions. To fix this, ensure that there is at least 1 real symbols in each c++ file that is referred to externally. This can simply be a dummy function, or an "extern int". Putting all the CFFI "glue" functions in a single file is not such a bad idea, and reduces this problem to a single symbol.

Array Access

One of the differences between neko CFFI and C++ CFFI is how arrays are treated. The neko implementation of arrays naturally allows it to provide and "value *" - pointer to series of elements. However, this is only available on a C++ executable if the array is an array of Dynamic. If the array is an array of Int, Bool or Double, then it may be possible to get an array of "double *" etc, which is a very efficient way of accessing the data (it is not possible to access strings like this). To write efficient code, your ndll should try the val_array_value function (if you are loaded in neko), then one or more of the specific array functions (eg, val_array_double if your haxe code takes an Array<Float>), before falling back to val_array_i if the other options are not available.

ByteArray Access

Neko uses "strings" as byte buffers, whereas C++ uses "Array of unsigned char", since its native string representation is an immutable string of wchar_t. To access the data as a byte-buffer, you must first use "val_to_buffer" to convert the value handle to a buffer handle. Using this, you can then call the various buffer manipulation functions. The different type of handle ensures type checking at compile time.

Neko vs Haxe

When haxe runs on the neko vm, it "wraps" the neko string and array values in objects. The nekoapi.dll file attempts to make this transparent. In this sense, it is more like a "haxe" CFFI than a "neko" CFFI.

Playing Nice with Garbage Collection

The C++ runtime uses built-in garbage collection that requires cooperation from all haxe threads to synchronize when a garbage collection is performed. In a single threaded application, it is not really necessary to worry about this, however it is best to do the right thing in case someone wants to use you library in their application.

The requirements are that before you enter a potentially blocking call (eg, socket select) or perform a long calculation that does not require any interaction with the haxe allocation system, you call gc_enter_blocking and before you call another haxe allocation, you call gc_exit_blocking. If you are performing a long calculation that may or may not use haxe allocation/callbacks, then you should call gc_safe_point inside your loop. You can use the C++ "AutoGCBlocking" object to ensure your blocks match your unblocks.

You do not need to do this for neko, but is does not hurt to leave to code in there.

Building Using the HXCPP Build-tool

There are a several advantages of using the hxcpp built-tool to build your ndll. The most obvious ones are that it is cross-platform and low-dependency. Since it uses neko and haxelib to do the work, you do not have to worry about installers etc, and you can be reasonably sure that anyone who is interested in building a haxe ndll is going to have the required files right there on their machine. This even applies if you are primarily interested in neko - all you have to do is "haxelib install hxcpp" to get the build tool, and avoids the need to get "make" for windows, or "Visual Studio" for Linux. The cross-platform library stuff is used for the C++ backend, so you will benefit from any improvements performed for that project.

Currently, Windows, Linux, Mac iPhoneOS and iPhoneSimulator are supported, and 64 bit versions of Windows, Linux and Mac are planned.

To use the tool, you need to write a "Build.xml" file to specify the files, extra compile options and extra link options. This is currently not well documented, but you can see examples in http://code.google.com/p/waxe/source/browse/trunk/src/Build.xml and http://code.google.com/p/nekonme/source/browse/trunk/version2/project/Build.xml.

To run the tool, you can use "haxelib run hxcpp Build.xml". You can provide additional "defines" to the command line, using -D, eg: "-Ddebug", "-Diphoneos", "-Diphonesim", "-Diphone", are the most useful and have built-in meaning to the tool.

When you use haxelib in this way, it tries to find a file called "run.n" in your library (ie, hxcpp in this case) install directory. You can modify the build tool by compiling the source in the hxcpp/built-tool directory.

Example


This example shows how to call external C/C++ code from a haxe (hxcpp) application.

1. Create the following directory structure.


-A folder named include. (This will contain the hx folder with the header files.)
-A folder named src. (This will contain the cpp and haxe source code.)
-A folder named bin. (This will contain the final application.)

2. Copy the hx folder from hxcpp to the include folder.


Navigate to your haxelib directory, open the hxcpp folder and copy the hx folder to the include folder from this example.
The hx folder can for instance be found in lib\hxcpp\2,10\include\.
(If you don't have hxcpp installed yet, first run "haxelib install hxcpp" from command line.)

3. Write the cpp and haxe source code.


Create a file named test.cpp inside the src folder. The contents of this file has been explained in the introduction of this page.
#define IMPLEMENT_API
#include <hx/CFFI.h>

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
extern "C" {
    value sum(value a, value b)
    {

        if( !val_is_int(a) || !val_is_int(b) ) return val_null;
        return alloc_int(val_int(a) + val_int(b));
    }
}
DEFINE_PRIM( sum, 2 );

Create a file named Main.hx inside the src folder.

class Main 
{
    //This loads the test.ndll and 'binds' the Haxe function 'sum' to the C function 'sum'.
    //Int->Int->Int is Haxe syntax for typing a function that takes 2 integers as arguments and that returns an integer.
    static var sum:Int->Int->Int = cpp.Lib.load("test","sum",2);
    static function main()
    {
        trace(sum(1,2));
    }
}

4. Compile the source code.


Let's compile test.cpp to test.ndll with:
g++ ./src/test.cpp -shared -o ./bin/test.ndll -I./include

and Main.hx to Main.exe with:
haxe -cp src -main Main -cpp bin

5. Test the final application.


If you navigate to the bin folder and execute Main(.exe) you should see printed to your console:
Main.hx:8: 3

6. Automate the build process.


You can store the compile and test commands in a shell script.
For windows create a file named build.bat next to your src folder:
@echo off
g++ ./src/test.cpp -shared -o ./bin/test.ndll -I./include
haxe -cp src -main Main -cpp bin
cd bin
Main.exe
pause

For linux or mac create a file named build.sh next to your src folder:
#!/bin/bash
g++ ./src/test.cpp -shared -o ./bin/test.ndll -I./include
haxe -cp src -main Main -cpp bin
bin/Main

7. Troubleshooting

  • 'Error : Could not load module test@sum__2'

You are on a 64-bit machine, add ' -D HXCPP_M64' to the end of your build command.

haxe -cp src -main Main -cpp bin -D HXCPP_M64
  • 'g++' is not recognized as an internal or external command,operable program or batch file.

In windows you'll see this error message when you have no c++ compiler installed or in PATH.
(You can for example use mingw but there are many others as well.)

  • In file included from ./include/hx/CFFI.h:73:0, from ./src/test.cpp:2:
    ./include/hx/CFFILoader.h: In function 'void* LoadFunc(const char*)':
    ./include/hx/CFFILoader.h:592:57: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
    ./include/hx/CFFILoader.h:592:57: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]

You can safely ignore these warnings or add -w to the build command to suppress any warnings:

g++ ./src/test.cpp -shared -o ./bin/test.ndll -I./include -w

version #20049, modified 2014-04-09 07:43:31 by indolering