CSC/ECE 517 Spring 2014/ch1 1w1h jg: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
 
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== '''Ruby libraries to load objects of other languages at run time''' ==
== '''Ruby libraries to load objects of other languages at run time''' ==
[http://en.wikipedia.org/wiki/Ruby_(programming_language) Ruby] is a dynamic, reflective, object-oriented, general-purpose programming language. It is easy to extend Ruby with new features by writing code in Ruby. But every now and then extending ruby with low-level languages, such as[http://en.wikipedia.org/wiki/C_(programming_language) C]/[http://en.wikipedia.org/wiki/C%2B%2B C++]/[http://en.wikipedia.org/wiki/Java_(programming_language) Java] is also necessary.


Currently, various kinds of languages codes could be invoked from within ruby. The extension for C/C++ and Java are focused here.
This is an advanced topic. Libraries are being developed where Ruby gems can load libraries (and even native code) of other languages.A few open-source projects like FFI are currently working on this. Previous version is available [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2013/ch1_1w21_w here]
 


__TOC__
__TOC__
==Introduction==
===Runtime Library===
In computer programming, a runtime library is the API used by a compiler to invoke some of the behaviors of a runtime system. The runtime system implements the execution model and other fundamental behaviors of a programming language. The compiler inserts calls to the runtime library into the executable binary. During execution (run time) of that computer program, execution of those calls to the runtime library cause communication between the application and the runtime system. This often includes functions for input and output, or for memory management.
The runtime library may implement a portion of the runtime system's behavior, but if one reads the code of the calls available, they typically are thin wrappers that simply package information and send it to the runtime system. However, sometimes the term runtime library is meant to include the code of the runtime system itself, even though much of that code cannot be directly reached via a library call.
For example, some language features that can be performed only (or are more efficient or accurate) at runtime are implemented in the runtime system and may be invoked via the runtime library API, e.g. some logic errors, array bounds checking, dynamic type checking, exception handling and possibly debugging functionality. For this reason, some programming bugs are not discovered until the program is tested in a "live" environment with real data, despite sophisticated compile-time checking and pre-release testing. In this case, the end user may encounter a runtime error message.
The concept of a runtime library should not be confused with an ordinary program library like that created by an application programmer or delivered by a third party, nor with a dynamic library, meaning a program library linked at run time.
===libffi===
FFI stands for Foreign Function Interface. A foreign function interface is the popular name for the interface that allows code written in one language to call code written in another language. The libffi was originally written by Anthony Green<ref>http://moxielogic.org/blog</ref>. The libffi library really only provides the lowest, machine dependent layer of a fully featured foreign function interface. A layer must exist above libffi that handles type conversions for values passed between the two languages.<ref>http://sourceware.org/libffi/</ref>
libffi is a foreign function interface library. It provides a C programming language interface for calling natively compiled functions given information about the target function at run time instead of compile time. It also implements the opposite functionality: libffi can produce a pointer to a function that can accept and decode any combination of arguments defined at run time.
libffi is most often used as a bridging technology between compiled and interpreted language implementations. libffi may also be used to implement plug-ins, where the plug-in's function signatures are not known at the time of creating the host application.
Notable users include Python, Haskell, Dalvik, F-Script, PyPy, PyObjC, RubyCocoa, JRuby, Rubinius, MacRuby, gcj, GNU Smalltalk, IcedTea, Cycript, Pawn, Squeak, Java Native Access, Racket,<ref>http://en.wikipedia.org/wiki/Libffi#cite_note-2</ref> Embeddable Common Lisp and Mozilla.<ref>http://en.wikipedia.org/wiki/Libffi#cite_note-3</ref>
On Mac OS X libffi is commonly used with BridgeSupport, which provides programming language neutral descriptions of framework interfaces, and Nu which binds direct Objective-C access from Lisp.
libffi has been widely ported and is released under a MIT license.
===Ruby-FFI===
Ruby-FFI<ref>https://github.com/ffi/ffi/wiki</ref> is a Foreign Function Interface extension for Ruby. It is an FFI extension doesn't need compilation and can be used on multi-platform and multi-implementation. Ruby-FFI extension is easy to read,write and maintain. In addition, it is impervious to changes in the internal Ruby API.
====Overall Architecture<ref>https://github.com/ffi/ffi/wiki/Core-Concepts#wiki-arch</ref>====
Using Ruby-FFI, you can use native libraries from Ruby without writing a single line of native code. Ruby-FFI takes care of all of the cross Ruby implementation (MRI, JRuby, Rubinius, MacRuby, etc) and platform specific issues so that you can focus on writing and testing your Ruby code.
As FFI is effectively a bridge between the multiple worlds of Ruby implementations and multiple platform types, you might suffer a bit of cognitive dissonance trying to pull all the pieces together. When we develop in Ruby we tend to think in higher level terms, and don't so much concern ourselves with the lower level issues.
====Core Components<ref>https://github.com/ffi/ffi/wiki/Core-Concepts#wiki-core-components</ref>====
Ruby-FFI has a number of useful components. Investing the time to understand FFI's components and capabilities will pay off as you begin using FFI. That said, it's nice to have an idea which components you should look at first. Understanding the following core modules and classes is a great way to start getting FFI's capabilities.
*'''FFI::Library'''
Along with require 'ffi', this module brings FFI's powerful native library interfacing capabilities into your Ruby code as a DSL. Typically you extend your custom module with this one, specify the native libraries and their calling conventions, prototype the native library's functions and structs in Ruby, and then start using the native library's API from Ruby.
*'''FFI::Pointer'''
Wraps native memory allocated by a third party library. It provides a number of methods for transferring data from unmanaged native memory to Ruby-managed native memory (FFI::MemoryPointer). The native memory wrapped by this class is not freed during garbage collection runs.
*'''FFI::MemoryPointer'''
Allows for Ruby code to allocate native memory and pass it to non-Ruby libraries. Lifecycle management (allocation and deallocation) are handled by this class, so when it gets garbage collected the native memory is also freed.
*'''FFI::Struct and FFI::Union'''
==How does it work?<ref>http://www.rubyinside.com/ruby-ffi-library-calling-external-libraries-now-easier-1293.html</ref>==
The many compiled bridges between external libraries and Ruby pose a problem for alternate implementations like JRuby, because of the complexity involved in exposing internals of the implementation or expensive serialization in both directions. Instead, an interface is necessary so that instead of developing libraries that act only as unique bridges to others, we can just have one library that provides the interface to any arbitrary library of our choice.
Ruby already has a library called "dl" that makes it possible to dynamically link external libraries with Ruby. It's a bit arcane though, and Charles points out that it's not widely used "because of real or perceived bugs." Given this, and given the need for an implementation that can be compatible with JRuby, Wayne Meissner has developed "FFI", a new Ruby library that provides "Foreign Function Interface" features to Ruby.
===Progress of FFI<ref>http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html</ref>===
One of the largest problems plaguing Ruby implementations is the ever-painful story of "extensions". In general, these take the form of a dynamic library, usually written in C, that plugs into and calls Ruby's native API as exposed through ruby.h and libruby. Ignoring for the moment the fact that this API exposes way more of Ruby's internals than it should, extensions present a very difficult problem for other implementations:
Do we support them or not?


== '''Ruby C/C++ extensions<ref>http://java.ociweb.com/mark/NFJS/RubyCExtensions.pdf</ref>''' ==
In many cases, this question is answered for us; most extensions require access to object internals we can't expose, or can't expose without extremely expensive copying back and forth. But there's also a silver lining: the vast majority of C-based extensions exist solely to wrap another library.
By extending Ruby with C. The C libraries could be used directly in Ruby applications.
Ruby could call C codes in three ways: interpreter API, RubyInline, SWIG.
=== interpreter API ===
Ruby interpreter is implemented in C, its API can be used and no special API added for interacting with C like Java’s JNI is needed.


The files needed include:
Isn't it obvious what's needed here?
*'''extconf.rb'''
Used to generate Makefile


Platform-specific Makefiles for compiling C extensions to Ruby is needed to be generated firstly.
This problem has been tackled by a number of libraries on a number of platforms. On the JVM, there's Java Native Access (JNA). On Python, there's ctypes. And even on Ruby, there's the "dl" stdlib, wrapping libdl for programmatic access to dynamic libraries. But dl is not widely used, because of real or perceived bugs and a rather arcane API. Something better is needed.


A simple sample is like below:
Wayne is one of the primary maintainers of JNA, and has recently spent time on a new higher-performance version of it called JFFI. Wayne also became a JRuby committer this spring, and perhaps his most impressive contribution to date is a full FFI library for JRuby, based on JNA (eventually JFFI, once we migrate fully) and implementing the full set of what we and Evan agreed would be "FFI API 1.0". We shipped the completed FFI support in JRuby 1.1.4.


create a file containing the following,
 
named extconf.rb by convention
The "Passwd" and "Group" structures for functions like 'getpwuid':
<pre>
<pre>
require 'mkmf'
module Etc
extension_name = 'name'
  class Passwd < FFI::Struct
dir_config(extension_name)
    layout :pw_name, :string, 0,
create_makefile(extension_name)
          :pw_passwd, :string, 4,
          :pw_uid, :uint, 8,
          :pw_gid, :uint, 12,
          :pw_dir, :string, 20,
          :pw_shell, :string, 24
  end
  class Group < FFI::Struct
    layout :gr_name, :string, 0,
          :gr_gid, :uint, 8
  end
end
</pre>
</pre>
And use by running
 
In JRuby 1.1.5, we've taken another step forward with the API, adding support for callbacks. How would we represent a callback you pass into a C function from Ruby?
 
Binding and calling "qsort" with an array of integers:
<pre>
<pre>
ruby extconf.rb
require 'ffi'
make
 
module LibC
  extend FFI::Library
  callback :qsort_cmp, [ :pointer, :pointer ], :int
  attach_function :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end
 
p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
  i1 = p1.get_int32(0)
  i2 = p2.get_int32(0)
  i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"
</pre>
</pre>
Then generates
 
What good is having such a library if it doesn't run everywhere? Up until recently, only Rubinius and JRuby supported FFI, which made our case for cross-implementation use pretty weak. Even though we were getting good use out of FFI, there was no motivation for anyone to use it in general, since the standard Ruby implementation had no support.
 
That is, until Wayne pulled another rabbit out of his hat and implemented FFI for C Ruby as well. The JRuby team is proud to announce a wholly non-JRuby library: FFI is now available on Ruby 1.9 and Ruby 1.8.6/7, in addition to JRuby 1.1.4+ and Rubinius (though Rubinius does not yet support callbacks).
 
Session showing installation and use of FFI in C Ruby:
<pre>
<pre>
.so under UNIX/Linux
$ sudo gem install ffi
.so under Windows when building with Cygwin
Password:
.bundle under Mac OS X
Building native extensions.  This could take a while...
Successfully installed ffi-0.1.1
1 gem installed
Installing ri documentation for ffi-0.1.1...
Installing RDoc documentation for ffi-0.1.1...
[headius @ cnutter:~]
$ irb
>> require 'ffi'
=> true
>> module RubyFFI
>> extend FFI::Library
>> attach_function :getuid, [], :uint
>> end
=> #<FFI::Invoker:0x1fe8c>
>> puts RubyFFI.getuid
501
=> nil
>>
</pre>
</pre>


*'''C File'''
C codes to be invoked from Ruby


An example Hello.c is shown below
Our hope with JRuby's support of FFI and our release of FFI for C Ruby is that we may finally escape the hell of C extensions.  
 
A key feature that's not well documented is the use of FFI's templating system to generate bindings based on the current platform's header files. Here's a sample from the "Etc" module above.
 
Etc module template, showing how to pull in header files and inspect a struct definition:
 
<pre>
<pre>
#include <ruby.h>
module Etc
#include <stdio.h>
  class Passwd < FFI::Struct
// These C functions will be associated with
    @@@
// methods of a Ruby class on the next page.
    struct do |s|
static VALUE hello(VALUE self, VALUE arg) {
      s.include "sys/types.h"
char* name = RSTRING(arg)->ptr;
      s.include "pwd.h"
printf("Hello %s!\n", name);
 
return Qnil;
      s.name "struct passwd"
}
      s.field :pw_name, :string
static VALUE goodbye(VALUE class) {
      s.field :pw_passwd, :string
printf("Later dude!\n");
      s.field :pw_uid, :uint
return Qnil;
      s.field :pw_gid, :uint
}
      s.field :pw_dir, :string
// This is called when the Ruby interpreter loads this C extension.
      s.field :pw_shell, :string
// The part after "Init_" is the name of the C extension specified
    end
// in extconf.rb, not the name of the C source file.
    @@@
void Init_hello() {
  end
// Create a Ruby module.
  class Group < FFI::Struct
VALUE myModule = rb_define_module("MyModule");
    @@@
// Create a Ruby class in this module.
    struct do |s|
// rb_cObject is defined in ruby.h
      s.include "sys/types.h"
VALUE myClass =
      s.include "grp.h"
rb_define_class_under(myModule, "MyClass", rb_cObject);
// Add an instance method to the Ruby class.
int arg_count = 1;
rb_define_method(myClass, "hello", hello, arg_count);
// Add a class method to the Ruby class.
arg_count = 0;
rb_define_module_function(myClass, "goodbye", goodbye, arg_count);
}
</pre>
*'''Ruby File'''
Ruby code that invokes C code


An example client.rb is shown as below
      s.name "struct group"
<pre>
      s.field :gr_name, :string
require 'hello'
      s.field :gr_gid, :uint
include MyModule # so MyClass doesn't need MyModule:: prefix
    end
obj = MyClass.new # MyClass is defined in C
    @@@
obj.hello('Mark') # calling an object method
  end
MyClass.goodbye # calling a class method
end
</pre>
</pre>


Finally we could build and run
===Basic Usage of Ruby-FFI<ref>https://github.com/ffi/ffi/wiki/Basic-Usage</ref>===
<pre>
Here is a  "Hello, World" example how to use Ruby-FFI.
ruby extconf.rb
make
ruby client.rb
</pre>


Then the output will display as
<pre>
<pre>
Hello Mark!
require 'ffi'
Later dude!
</pre>


=== RubyInline ===
module Hello
[http://www.zenspider.com/ZSS/Products/RubyInline/ RubyInline] allows C code to be imbedded in Ruby code. It automatically determines if the code in question has changed and builds it only when necessary. The extensions are then automatically loaded into the class/module that defines it.
  extend FFI::Library
  ffi_lib FFI::Library::LIBC
  attach_function :puts, [ :string ], :int
end


To setup RubyInline, just use
Hello.puts("Hello, World")
<pre>
gem install rubyinline
</pre>
</pre>


=== SWIG ===
The interesting part above is the attach_function call. It requests that a C function named puts which takes a :string argument and returns an :int, be attached to the Hello module.
[http://en.wikipedia.org/wiki/SWIG SWIG] is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of target languages including common scripting languages such as Perl, PHP, Python, Tcl and Ruby.  


As an example, building Ruby Extensions under Windows 95/NT is shown as below
The attach_function method locates the C function in a specific external library (i.e. libc in the example). The name of the function, the parameter types and the return type are needed. In the above example, attach_function locates a function named puts, which papameter type and return type are :string and :int respectively.
Building a SWIG extension to Ruby under Windows 95/NT is roughly similar to the process used with Unix.  
<pre>
C:\swigtest> ruby extconf.rb
C:\swigtest> nmake
C:\swigtest> nmake install
</pre>


== '''Ruby JAVA extensions''' ==
The typical parameter and return types<ref>https://github.com/ffi/ffi/wiki/Types</ref> are listed here:
To extend Ruby with JAVA, the [http://jruby.org/ JRuby] is a good tool to use. JRuby is a high performance, stable, fully threaded Java implementation of the Ruby programming language. It is a Ruby interpreter written entirely in Java and can use Java capabilities from Ruby as well as can use Ruby capabilities from Java.
=== Using JRuby From Command-Line ===
*'''Steps to install'''
– download a binary release [www.jruby.org here]


– unzip/untar the downloaded archive
* :char and :uchar - 8 bit signed and unsigned values
* :short and :ushort - 16 bit signed and unsigned values
* :int and :uint - 32 bit signed and unsigned values
* :long_long and :ulong_long - 64 bit signed and unsigned values
* :long and :ulong - native cpu word (32 bit or 64bit) signed and unsigned values. Equivalent to C 'long' type.
* :float and :double
* :string - C string, NULL terminated.
* :pointer - a C pointer


– set JRUBY_HOME environment variable to point to resulting directory
You can also pass the struct as parameter. For example:


– add $JRUBY_HOME/bin to PATH
*'''Steps to use'''
– jruby {script-name}
*'''Example'''
– hello.rb
<pre>
<pre>
name = ARGV[0] || "you"
typedef struct _WHAT {
puts "Hello #{name}!"
  int d;
} WHAT;
 
int doit(WHAT w) {
  printf("%d\n", w.d);
  return w.d;
}
</pre>
</pre>
– run with
 
<pre>
<pre>
jruby hello.rb
class WHAT < FFI::Struct
</pre>
  layout :d, :int
outputs
end
<pre>
attach_function 'doit', [ WHAT.by_value ], :int
Hello you!
</pre>
– run with
<pre>
jruby hello.rb Mark
</pre>
outputs
<pre>
Hello Mark
</pre>
</pre>
In the above 2 pieces of code, attach_function uses struct WHAT as a parameter type.
==Who uses it?==
The libffi library is useful to anyone trying to build a bridge between interpreted and natively compiled code. Some notable users include:
*CPython - the default, most-widely used implementation of the Python programming language uses libffi in the standard ctypes library.
*OpenJDK - the open-source implementation of the Java Platform Standard Edition uses libffi to bridge between the interpreter and native code for some platforms.
*js-ctypes - a foreign function interface for javascript that Mozilla will be shipping in Firefox 3.6.
*Dalvik - Dalvik is the virtual machine which runs the Java platform on Android mobile devices. libffi is used on Android ports for which no custom bridging code has been written.
*Java Native Access (JNA) - the JNI-free way to call native code from Java.
*Ruby-FFI - a Foreign Function Interface extension for Ruby.
*fsbv - Foreign Structure By Value is a foreign function interface library for Common Lisp that extends the standard CFFI package to include support for passing structure arguments by value.
*JSCocoa - call Objective-C code from javascript on Mac OSX and the iPhone (via the libffi-iphone port).
*PyObjC - call Objective-C code from Python on Mac OSX.
*RubyCocoa - call Objective-C code from Ruby on Mac OSX.
*PLT Scheme - call C code from this popular Scheme implementation (also used as the implementation platform for Paul Graham's new Lisp, Arc).
*gcj - the runtime library for the GNU Compiler for the Java Programming Language uses libffi to handle calls back and forth between interpreted and natively compiled code. gcj is part of the GCC, the GNU Compiler Collection.
==Supported Platforms==
Libffi has been ported to many different platforms. For specific configuration details and testing status, please refer to the wiki page [http://www.moxielogic.org/wiki/index.php?title=Libffi_3.0.13 here]
==Adoption==
[http://ruby-doc.org/stdlib-2.0/libdoc/fiddle/rdoc/index.html Fiddle]<br/>
A libffi wrapper in the Ruby Standard Library<br/>
[http://github.com/ffi/ffi Ruby-FFI]<br/>
A Foreign Function Interface extension for Ruby.<br/>
[http://en.wikipedia.org/wiki/RubyCocoa  RubyCocoa]<br/>
Call Objective-C code from Ruby on Mac OSX.<br/>
==See Also==
[http://en.wikipedia.org/wiki/Libffi Wikipedia page for Libffi]<br/>
[http://wiki.jruby.org/wiki/Calling_C_from_JRuby Calling C from JRuby]<br/>
[http://blog.segment7.net/articles/2008/01/15/rubinius-foreign-function-interfaceRubinius's Foreign Function Interface]<br/>
[http://pluskid.lifegoo.com/?p=370On the Rubinius FFI]<br/>
=='''References''' ==
<references/>
*http://github.com/atgreen/libffi/blob/master/LICENSE
*http://repository.readscheme.org/ftp/papers/sw2004/barzilay.pdf
*http://hg.mozilla.org/mozilla-central/file/2dc00d4b379a/js/ctypes/libffi
*https://developer.mozilla.org/en/js-ctypes
*https://developer.mozilla.org/en/js-ctypes/js-ctypes_reference
*http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html
*http://www.rubyinside.com/ruby-ffi-library-calling-external-libraries-now-easier-1293.html
*http://sourceware.org/libffi/

Latest revision as of 21:39, 10 February 2014

Ruby libraries to load objects of other languages at run time

This is an advanced topic. Libraries are being developed where Ruby gems can load libraries (and even native code) of other languages.A few open-source projects like FFI are currently working on this. Previous version is available here


Introduction

Runtime Library

In computer programming, a runtime library is the API used by a compiler to invoke some of the behaviors of a runtime system. The runtime system implements the execution model and other fundamental behaviors of a programming language. The compiler inserts calls to the runtime library into the executable binary. During execution (run time) of that computer program, execution of those calls to the runtime library cause communication between the application and the runtime system. This often includes functions for input and output, or for memory management. The runtime library may implement a portion of the runtime system's behavior, but if one reads the code of the calls available, they typically are thin wrappers that simply package information and send it to the runtime system. However, sometimes the term runtime library is meant to include the code of the runtime system itself, even though much of that code cannot be directly reached via a library call.

For example, some language features that can be performed only (or are more efficient or accurate) at runtime are implemented in the runtime system and may be invoked via the runtime library API, e.g. some logic errors, array bounds checking, dynamic type checking, exception handling and possibly debugging functionality. For this reason, some programming bugs are not discovered until the program is tested in a "live" environment with real data, despite sophisticated compile-time checking and pre-release testing. In this case, the end user may encounter a runtime error message.

The concept of a runtime library should not be confused with an ordinary program library like that created by an application programmer or delivered by a third party, nor with a dynamic library, meaning a program library linked at run time.

libffi

FFI stands for Foreign Function Interface. A foreign function interface is the popular name for the interface that allows code written in one language to call code written in another language. The libffi was originally written by Anthony Green<ref>http://moxielogic.org/blog</ref>. The libffi library really only provides the lowest, machine dependent layer of a fully featured foreign function interface. A layer must exist above libffi that handles type conversions for values passed between the two languages.<ref>http://sourceware.org/libffi/</ref> libffi is a foreign function interface library. It provides a C programming language interface for calling natively compiled functions given information about the target function at run time instead of compile time. It also implements the opposite functionality: libffi can produce a pointer to a function that can accept and decode any combination of arguments defined at run time. libffi is most often used as a bridging technology between compiled and interpreted language implementations. libffi may also be used to implement plug-ins, where the plug-in's function signatures are not known at the time of creating the host application. Notable users include Python, Haskell, Dalvik, F-Script, PyPy, PyObjC, RubyCocoa, JRuby, Rubinius, MacRuby, gcj, GNU Smalltalk, IcedTea, Cycript, Pawn, Squeak, Java Native Access, Racket,<ref>http://en.wikipedia.org/wiki/Libffi#cite_note-2</ref> Embeddable Common Lisp and Mozilla.<ref>http://en.wikipedia.org/wiki/Libffi#cite_note-3</ref> On Mac OS X libffi is commonly used with BridgeSupport, which provides programming language neutral descriptions of framework interfaces, and Nu which binds direct Objective-C access from Lisp. libffi has been widely ported and is released under a MIT license.

Ruby-FFI

Ruby-FFI<ref>https://github.com/ffi/ffi/wiki</ref> is a Foreign Function Interface extension for Ruby. It is an FFI extension doesn't need compilation and can be used on multi-platform and multi-implementation. Ruby-FFI extension is easy to read,write and maintain. In addition, it is impervious to changes in the internal Ruby API.

Overall Architecture<ref>https://github.com/ffi/ffi/wiki/Core-Concepts#wiki-arch</ref>

Using Ruby-FFI, you can use native libraries from Ruby without writing a single line of native code. Ruby-FFI takes care of all of the cross Ruby implementation (MRI, JRuby, Rubinius, MacRuby, etc) and platform specific issues so that you can focus on writing and testing your Ruby code.

As FFI is effectively a bridge between the multiple worlds of Ruby implementations and multiple platform types, you might suffer a bit of cognitive dissonance trying to pull all the pieces together. When we develop in Ruby we tend to think in higher level terms, and don't so much concern ourselves with the lower level issues.

Core Components<ref>https://github.com/ffi/ffi/wiki/Core-Concepts#wiki-core-components</ref>

Ruby-FFI has a number of useful components. Investing the time to understand FFI's components and capabilities will pay off as you begin using FFI. That said, it's nice to have an idea which components you should look at first. Understanding the following core modules and classes is a great way to start getting FFI's capabilities.

  • FFI::Library

Along with require 'ffi', this module brings FFI's powerful native library interfacing capabilities into your Ruby code as a DSL. Typically you extend your custom module with this one, specify the native libraries and their calling conventions, prototype the native library's functions and structs in Ruby, and then start using the native library's API from Ruby.

  • FFI::Pointer

Wraps native memory allocated by a third party library. It provides a number of methods for transferring data from unmanaged native memory to Ruby-managed native memory (FFI::MemoryPointer). The native memory wrapped by this class is not freed during garbage collection runs.

  • FFI::MemoryPointer

Allows for Ruby code to allocate native memory and pass it to non-Ruby libraries. Lifecycle management (allocation and deallocation) are handled by this class, so when it gets garbage collected the native memory is also freed.

  • FFI::Struct and FFI::Union

How does it work?<ref>http://www.rubyinside.com/ruby-ffi-library-calling-external-libraries-now-easier-1293.html</ref>

The many compiled bridges between external libraries and Ruby pose a problem for alternate implementations like JRuby, because of the complexity involved in exposing internals of the implementation or expensive serialization in both directions. Instead, an interface is necessary so that instead of developing libraries that act only as unique bridges to others, we can just have one library that provides the interface to any arbitrary library of our choice.

Ruby already has a library called "dl" that makes it possible to dynamically link external libraries with Ruby. It's a bit arcane though, and Charles points out that it's not widely used "because of real or perceived bugs." Given this, and given the need for an implementation that can be compatible with JRuby, Wayne Meissner has developed "FFI", a new Ruby library that provides "Foreign Function Interface" features to Ruby.

Progress of FFI<ref>http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html</ref>

One of the largest problems plaguing Ruby implementations is the ever-painful story of "extensions". In general, these take the form of a dynamic library, usually written in C, that plugs into and calls Ruby's native API as exposed through ruby.h and libruby. Ignoring for the moment the fact that this API exposes way more of Ruby's internals than it should, extensions present a very difficult problem for other implementations:

Do we support them or not?

In many cases, this question is answered for us; most extensions require access to object internals we can't expose, or can't expose without extremely expensive copying back and forth. But there's also a silver lining: the vast majority of C-based extensions exist solely to wrap another library.

Isn't it obvious what's needed here?

This problem has been tackled by a number of libraries on a number of platforms. On the JVM, there's Java Native Access (JNA). On Python, there's ctypes. And even on Ruby, there's the "dl" stdlib, wrapping libdl for programmatic access to dynamic libraries. But dl is not widely used, because of real or perceived bugs and a rather arcane API. Something better is needed.

Wayne is one of the primary maintainers of JNA, and has recently spent time on a new higher-performance version of it called JFFI. Wayne also became a JRuby committer this spring, and perhaps his most impressive contribution to date is a full FFI library for JRuby, based on JNA (eventually JFFI, once we migrate fully) and implementing the full set of what we and Evan agreed would be "FFI API 1.0". We shipped the completed FFI support in JRuby 1.1.4.


The "Passwd" and "Group" structures for functions like 'getpwuid':

module Etc
  class Passwd < FFI::Struct
    layout :pw_name, :string, 0,
           :pw_passwd, :string, 4,
           :pw_uid, :uint, 8,
           :pw_gid, :uint, 12,
           :pw_dir, :string, 20,
           :pw_shell, :string, 24
  end
  class Group < FFI::Struct
    layout :gr_name, :string, 0,
           :gr_gid, :uint, 8
  end
end

In JRuby 1.1.5, we've taken another step forward with the API, adding support for callbacks. How would we represent a callback you pass into a C function from Ruby?

Binding and calling "qsort" with an array of integers:

require 'ffi'

module LibC
  extend FFI::Library
  callback :qsort_cmp, [ :pointer, :pointer ], :int
  attach_function :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
  i1 = p1.get_int32(0)
  i2 = p2.get_int32(0)
  i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"

What good is having such a library if it doesn't run everywhere? Up until recently, only Rubinius and JRuby supported FFI, which made our case for cross-implementation use pretty weak. Even though we were getting good use out of FFI, there was no motivation for anyone to use it in general, since the standard Ruby implementation had no support.

That is, until Wayne pulled another rabbit out of his hat and implemented FFI for C Ruby as well. The JRuby team is proud to announce a wholly non-JRuby library: FFI is now available on Ruby 1.9 and Ruby 1.8.6/7, in addition to JRuby 1.1.4+ and Rubinius (though Rubinius does not yet support callbacks).

Session showing installation and use of FFI in C Ruby:

$ sudo gem install ffi
Password:
Building native extensions.  This could take a while...
Successfully installed ffi-0.1.1
1 gem installed
Installing ri documentation for ffi-0.1.1...
Installing RDoc documentation for ffi-0.1.1...
[headius @ cnutter:~]
$ irb
>> require 'ffi'
=> true
>> module RubyFFI
>> extend FFI::Library
>> attach_function :getuid, [], :uint
>> end
=> #<FFI::Invoker:0x1fe8c>
>> puts RubyFFI.getuid
501
=> nil
>> 


Our hope with JRuby's support of FFI and our release of FFI for C Ruby is that we may finally escape the hell of C extensions.

A key feature that's not well documented is the use of FFI's templating system to generate bindings based on the current platform's header files. Here's a sample from the "Etc" module above.

Etc module template, showing how to pull in header files and inspect a struct definition:

module Etc
  class Passwd < FFI::Struct
    @@@
    struct do |s|
      s.include "sys/types.h"
      s.include "pwd.h"

      s.name "struct passwd"
      s.field :pw_name, :string
      s.field :pw_passwd, :string
      s.field :pw_uid, :uint
      s.field :pw_gid, :uint
      s.field :pw_dir, :string
      s.field :pw_shell, :string
    end
    @@@
  end
  class Group < FFI::Struct
    @@@
    struct do |s|
      s.include "sys/types.h"
      s.include "grp.h"

      s.name "struct group"
      s.field :gr_name, :string
      s.field :gr_gid, :uint
    end
    @@@
  end
end

Basic Usage of Ruby-FFI<ref>https://github.com/ffi/ffi/wiki/Basic-Usage</ref>

Here is a "Hello, World" example how to use Ruby-FFI.

require 'ffi'

module Hello
  extend FFI::Library
  ffi_lib FFI::Library::LIBC
  attach_function :puts, [ :string ], :int
end

Hello.puts("Hello, World")

The interesting part above is the attach_function call. It requests that a C function named puts which takes a :string argument and returns an :int, be attached to the Hello module.

The attach_function method locates the C function in a specific external library (i.e. libc in the example). The name of the function, the parameter types and the return type are needed. In the above example, attach_function locates a function named puts, which papameter type and return type are :string and :int respectively.

The typical parameter and return types<ref>https://github.com/ffi/ffi/wiki/Types</ref> are listed here:

  • :char and :uchar - 8 bit signed and unsigned values
  • :short and :ushort - 16 bit signed and unsigned values
  • :int and :uint - 32 bit signed and unsigned values
  • :long_long and :ulong_long - 64 bit signed and unsigned values
  • :long and :ulong - native cpu word (32 bit or 64bit) signed and unsigned values. Equivalent to C 'long' type.
  • :float and :double
  • :string - C string, NULL terminated.
  • :pointer - a C pointer

You can also pass the struct as parameter. For example:

typedef struct _WHAT {
  int d;
} WHAT;

int doit(WHAT w) {
  printf("%d\n", w.d);
  return w.d;
}
class WHAT < FFI::Struct
  layout :d, :int
end
attach_function 'doit', [ WHAT.by_value ], :int

In the above 2 pieces of code, attach_function uses struct WHAT as a parameter type.

Who uses it?

The libffi library is useful to anyone trying to build a bridge between interpreted and natively compiled code. Some notable users include:

  • CPython - the default, most-widely used implementation of the Python programming language uses libffi in the standard ctypes library.
  • OpenJDK - the open-source implementation of the Java Platform Standard Edition uses libffi to bridge between the interpreter and native code for some platforms.
  • js-ctypes - a foreign function interface for javascript that Mozilla will be shipping in Firefox 3.6.
  • Dalvik - Dalvik is the virtual machine which runs the Java platform on Android mobile devices. libffi is used on Android ports for which no custom bridging code has been written.
  • Java Native Access (JNA) - the JNI-free way to call native code from Java.
  • Ruby-FFI - a Foreign Function Interface extension for Ruby.
  • fsbv - Foreign Structure By Value is a foreign function interface library for Common Lisp that extends the standard CFFI package to include support for passing structure arguments by value.
  • JSCocoa - call Objective-C code from javascript on Mac OSX and the iPhone (via the libffi-iphone port).
  • PyObjC - call Objective-C code from Python on Mac OSX.
  • RubyCocoa - call Objective-C code from Ruby on Mac OSX.
  • PLT Scheme - call C code from this popular Scheme implementation (also used as the implementation platform for Paul Graham's new Lisp, Arc).
  • gcj - the runtime library for the GNU Compiler for the Java Programming Language uses libffi to handle calls back and forth between interpreted and natively compiled code. gcj is part of the GCC, the GNU Compiler Collection.

Supported Platforms

Libffi has been ported to many different platforms. For specific configuration details and testing status, please refer to the wiki page here

Adoption

Fiddle
A libffi wrapper in the Ruby Standard Library
Ruby-FFI
A Foreign Function Interface extension for Ruby.
RubyCocoa
Call Objective-C code from Ruby on Mac OSX.

See Also

Wikipedia page for Libffi
Calling C from JRuby
Foreign Function Interface
the Rubinius FFI

References

<references/>