CSC/ECE 517 Fall 2009/wiki2 1 SA

Metaprogramming

Introduction

Metaprogramming, is the creation of procedures and programs that automatically construct the definitions of other procedures and programs. Metaprogramming automates some of the tedious and error-prone parts of the programmer's job.It refers to a variety of ways a program has knowledge of itself or can manipulate itself.It's basically "writing code that writes code".

To understand metaprogramming, compiler is a very good example. It takes as input, code in some language, and creates code for another language.

The most liberal definition of metaprogramming is:

• Compile code generation or Runtime code generation (or both)
• Aspect-Oriented Thinking or Aspect Oriented Programming
• DRY Thinking

Metaprogramming in various languages

There are many languages that do metaprogramming. In languages like C#, reflection is a form of metaprogramming since the program can examine information about itself. For example, returning a list of all the properties of an object. In languages like ActionScript, we can evaluate functions at runtime to create new programs such as eval("x" + i).DoSomething() would affect an object called x1 when i is 1 and x2 when i is 2.

Another common form of metaprogramming is when the program can change itself in non-trivial fashions. LISP is well known for this. The program would change another part of the program based on its state. This allows a level of flexibility to make decisions at runtime that is very difficult in most popular languages today. It is also worth noting that back in the good old days of programming in straight assembly, programs that altered themselves at runtime were necessary and very commonplace.

It can be implemented by using any of these and in combination:

• Reflection
• DSLs (Domain Specific Languages)
• (.NET) or Annotations (Java)
• Generics (.NET/Java)
• Templates (C++)
• Method_missing (Ruby)
• Closures / first class functions / delegates
• AOP - Aspect Oriented Programming

Different languages that supports Metaprogramming

• Curl
• D
• Forth
• Groovy
• Haskell
• Lisp
• Lua
• Maude system
• MetaL
• MetaOCaml
• Nemerle
• Perl
• Python
• Ruby
• Smalltalk
• uniPaaS
• XL (concept programming)
• C++

Metaprogramming in Curl

Curl is an object oriented programming language mainly designed for interactive web content and widely used in enterprise B2B or B2C applications. It is reflection oriented language, so metaprogramming is done in Curl using reflection. Using evaluate() procedure an application can create Curl source code at run time and cause it to be compiled and executed in a chosen environment (package).

• Example:7

 evaluate (proc)
 public {evaluate
     exp:any,
     base-url:Url = unspecified-url,
     package:#OpenPackage = null
 }:any

In the above example, exp is an expression, which should be a StringInterface or CurlSource containing the source to be evaluated or an url specifying a file from which the source should be read. Package is the environment for evaluation and base-url is the base-url for the code to be evaluated.

Metaprogramming in D

D is an object oriented, multiparadigm programming language mostly influenced from C++. It has redesigned some aspects of C++ and also influenced features from other programming languages like Java and C#. In D, metaprogramming is supported by a combination of templates, tuples, string mixins and compile time function execution. Following are examples of template metaprogramming and metaprogramming using string mixins.

• Example: Template metaprogramming 6

Following example is a template crypt where the encryption adds a constant value 5 to the string. D has built in 'static if' statement used for compile-time conditional construct.

import std.string;
import std.stdio;
template makechar(int c)
{ 
    const char [] makechar=
    x"000102030405060708090a0b0c0d0e0f
    e0e1e2e3e4e5e6e7e8e9eaebecedeeef
    f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"[c..c+1];
}
template crypt(char[] n, int val)
{	
    static if (n.length == 1) const char[] crypt = makechar!(n[0] + val);
    else const char[] crypt = makechar!(n[0] + val) ~ crypt!(n[1..n.length], val);
}
int main()
{
    char[] str = crypt!("This is metaprogramming in D", 5);
    printf("%.*s\n", str);
    return 0;
}

• Example: String mixin metaprogramming 5

D code is generated using string operations at compile time using string mixins, combined with compile-time function execution. This can be used to parse domain-specific languages to D code, which will be compiled as part of the program.

 import FooToD; // hypothetical module which contains a function that parses Foo source code and returns equivalent D code
 void main()
 {
     mixin(fooToD(import("example.foo")));
 }

Metaprogramming in Forth

Forth is a structured, procedural and reflective programming language which supports metaprogramming using reflection feature. Forth is not a object oriented language which uses meta programming. It has self compilation and cross compilation facilities which generates code at run time. This technique is commonly called meta-compilation. The usual method is to redefine the handful of words that place compiled bits into memory 9.

Metaprogramming in Groovy

Groovy is an object oriented programming language for Java platform with dynamic language features. It is also used for scripting. It uses metaprogramming in the following way.

Below example demonstrates using classes from the Jakarta Commons Lang package for metaprogramming. All of the methods in org.apache.commons.lang.StringUtils coincidentally follow the Category pattern — static methods that accept a String as the first parameter. This means that you can use the StringUtils class right out of the box as a Category.

import org.apache.commons.lang.StringUtils
 class CommonsTest extends GroovyTestCase{
  void testStringUtils(){
    def word = "Introduction"
    word.metaClass.whisper = {->
      delegate.toLowerCase()
    }
     use(StringUtils, StringHelper){
      //from org.apache.commons.lang.StringUtils
      assertEquals "Intro...", word.abbreviate(8)
       //from the StringHelper Category
      assertEquals "INTRODUCTION", word.shout()
       //from the word.metaClass
      assertEquals "introduction", word.whisper()
    }
  } 
}
class StringHelper{
  static String shout(String self){
    return self.toUpperCase()
  }
}

Metaprogramming in Lisp

Lisp is a multiparadigm, functional, procedural and reflective programming language. Lisp provides DSL for metaprogramming. In lisp Metaprogramming is done using a construct called a "macro".

• Example : 10

Let us consider, the following s-expression for a to-do list in Lisp. Suppose we're writing a to-do manager application. We keep our to-do items serialized in a set of files and when the program starts up we want to read them and display them to the user. One way is to read the entire list and show the relevant data to the user. Instead of writing code to walk the s-expression that stores our data we could write a macro that allows us to treat data as code.

 (todo "Homeworks"
    (item (priority high) "Homework for subject 1.")
    (item (priority medium) "Homework for subject 2.")
    (item (priority medium) "Homework for subject 3."))

Corresponding macros for the to-do list items that will get called by lisp compiler and will transform the to-do list into code will be as follows. Now the to-do list will be treated as code and will be executed. Suppose all we want to do is print it to standard output for the user to read.

 (defmacro item (priority note)
    '(block
        (print stdout tab "Priority: "
            ~(head (tail priority)) endl)
        (print stdout tab "Note: " ~note endl endl)))

Metaprogramming in MetaOcaml

MetaOCaml is a multi-stage extension of the OCaml programming language. It is a research language and uses DSL for metaprogramming. The basic approach to implementing DSLs in MetaOCaml is the staged interpreter approach. First, an interpreter is implemented and tested for the DSL. Then, the three staging constructs are used to produce an implementation that performs the traversal of the DSL program in a stage earlier than the execution of the essence of the program. Implementations derived in this manner can be as simple as an interpretive implementation, and at the same time have the performance of a compiled implementation.

Metaprogramming in Python

Python supports metaprogramming through metaclasses, decorators and descriptors.

• Metaclasses considered as "the class of a class" enable the creation of new “types of classes” which can customize the class creation process and/or add class methods and attributes.
• Decorators look like Scala annotations but are applied only to function definitions and yield (possibly transformed) functions.
• Descriptors provide a very concise way to add custom properties to classes that invoke logic but syntactically manipulated as regular attributes. They can also carry extra information (such as titles, database column mappings, etc) that can be useful when accessed through reflection.

class _AutoWriteDescriptor(object):
    Descriptor for AutoWrite.
      Implements an attribute that allows an initial setting then
       calls the setter.
    
    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix
    def __get__(self, obj, objtype=None):
        self.check_dictionary(obj)
        return obj._auto_write_dict[self.name]
    def __set__(self, obj, new_value):
        self.check_dictionary(obj)
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
        else:
            obj._auto_write_dict[self.name] = new_value
    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")
    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__
                % obj.__class__)

Above example is a simple Descriptor. The only thing to understand about Descriptors is that they are attributes. So the methods above live on the attribute, not on the main object. When Python accesses an attribute it checks and, if it's a Descriptor, it invokes the appropriate method (get if the attribute is being read, set if its value is being changed, delete if the attribute is being deleted).

The advantage of doing things this way is twofold. First, we end up uniting methods and functions.Second,we have an efficient way to change how certain attributes behave. This is much better than changing the main object's __getattr__ and _setattr__ because the extra code is only invoked for the "special" attributes - there no speed penalty otherwise.

If the value is being read, the actual value is pulled from a dictionary that is stored on the main object.If the value is being set for the first time (ie when the object is populated from the database) the value is stored. Subsequent updates (ie when the object is modified via the user interface) call the setter method.

Metaprogramming in Ruby

We have a C program that needs to include a PNG image, but for some reason, the deployment platform can accept one file only, the executable file. Thus, the data that conforms the PNG file data has to be integrated within the program code itself. To achieve this, we can read the PNG file beforehand and generate the C source text for an array declaration, initialized with the corresponding data as literal values. This Ruby script does exactly that:

INPUT_FILE_NAME = 'ljlogo.png'
OUTPUT_FILE_NAME = 'ljlogo.h'
DATA_VARIABLE_NAME = 'ljlogo'
File.open(INPUT_FILE_NAME, 'r') do |input|
  File.open(OUTPUT_FILE_NAME, 'w') do |output|
    output.print "unsigned char #{DATA_VARIABLE_NAME}[] = {"
    data = input.read.unpack('C*')
    data.length.times do |i|
      if i % 8 == 0
        output.print "\n    "
      end
      output.print '0x%02X' % data[i]
      output.print ', ' if i < data.length - 1
    end
    output.puts "\n};"
  end
end

Metaprogramming in SmallTalk

Smalltalk is an dynamically typed, object-oriented, reflective programming language. As refection is one feature of smalltalk metaprogramming can be done using this feature. In SmallTalk run time code generation is done by manipulating strings.

• Example :

In this example, metaprogramming is done by creating anonymous classes and giving them behavior, then instantiate them run time.

      | newClass newInstance |
      newClass := Behavior new. "create anonymous behavior"
      newClass compile: 'theAnswer ^42'. "Add a method for instances"
      newInstance := newClass new. "create an instance"
      Transcript show: newInstance theAnswer; cr. "shows 42"

Metaprogramming in C++

These Metaprograms run before the load time of the code they manipulate. Precompilers: C/C++ precompiler. Open compilers: like writing transformations on AST (hygenic macros). Two level languages: C++ templates

template <int i>
struct D
{
    D(void *);
    operator int();
};
template <int p, int i>
struct is_prime
{
    enum { prim = (p%i) && is_prime<(i>2?p:0), i>::prim };
};
template <int i>
struct Prime_print
{
    Prime_print<i-1>    a;
    enum { prim = is_prime<i,i-1>::prim };
    void f() { D d = prim; }
};
struct is_prime<0,0> { enum { prim = 1 }; };
struct is_prime<0,1> { enum { prim = 1 }; };
struct Prime_print<2>
{
    enum { prim = 1 };
    void f() { D<2> d = prim; }
};
void foo()
{
    Prime_print<10> a;
}

// output:
// unruh.cpp 30: conversion from enum to D<2> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<3> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<5> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<7> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<11> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<13> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<17> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<19> requested in Prime_print

Index

• Multiparadigm language : Multiparadigm languages provides a framework in which programmers can work in a variety of styles, freely intermixing constructs from different paradigms. Examples are C, C++, Java, Ruby, Oz, Visual Basic, Perl, Python etc. The design goal of multiparadigm languages is to allow programmers to use the best tool for a job, admitting that a single paradigm can not solve all problems in most efficient or easiest way.

• Template metaprogramming : Template metaprogramming is a metaprogramming technique in which templates are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled.

• Reflection oriented language : In reflective programming languages computer programs can observe and modify its own structure and behavior.

• Stack based programming language : This type of languages uses stack based models for passing parameters.

• s-expression : It is called symbolic expression i.e. an way to represent semi-structured data in human-readable textual form mostly made of symbols and lists. It is used in Lisp family of languages.

References

1. http://www.ibm.com/developerworks/java/library/j-pg06239.html
2. http://stackoverflow.com/questions/514644/what-exactly-is-metaprogramming
3. http://aszt.inf.elte.hu/~gsd/halado_cpp/ch06s04.html
4. http://en.wikipedia.org/wiki/Multi-paradigm_programming_language#Multi-paradigm_programming_language - Multi-paradigm_programming_language
5. http://en.wikipedia.org/wiki/D_%28programming_language%29 - D Programming language
6. http://www.the-interweb.com/serendipity/index.php?/archives/66-Template-meta-programming-in-D.html - Template metaprogramming in D
7. http://developers.curl.com/userdocs/CurlDocs.htm#docs/en/api-ref/evaluate.html - Curl evaluation fumction
8. http://www.curl.com/pdf/content-language.pdf - Curl a content language for the Web
9. http://en.wikipedia.org/wiki/Forth_%28programming_language%29 - Meta compilation in Forth
10. http://www.defmacro.org/ramblings/lisp.html - Metaprogramming in Lisp
11. http://www.acooke.org/cute/PythonMeta0.html - Metaprogramming in Python
12. http://groups.google.com/group/comp.lang.smalltalk/browse_thread/thread/bb0519b6b4c3e850?pli=1 - Metaprogramming in SmallTalk