CSC/ECE 517 Fall 2009/wiki2 1 SA

Metaprogramming

Introduction

Metaprogramming, is the creation of procedures and programs that automatically construct the definitions of other procedures and programs. Metaprogramming automates some of the tedious and error-prone parts of the programmer's job. It refers to a variety of ways a program has knowledge of itself or can manipulate itself. It's basically "writing code that writes code". Metaprogramming codes manipulate other programs or themselves as their data, they do part of the work at compile time that would otherwise be done at runtime. Reflection is one of the main feature used for metaprogramming.

To understand metaprogramming, compiler is a very good example. It takes as input, code in some language, and creates code for another language.

The most liberal definition of metaprogramming is:

• Compile code generation or Runtime code generation (or both)
• Aspect-Oriented Thinking or Aspect Oriented Programming
• DRY Thinking
• Template metaprogramming
• Self modifying code
• or Self code generation

Metaprogramming in various languages

There are many languages that do metaprogramming. In languages like C#, reflection is a form of metaprogramming since the program can examine information about itself. For example, returning a list of all the properties of an object. In languages like ActionScript, we can evaluate functions at runtime to create new programs such as eval("x" + i).DoSomething() would affect an object called x1 when i is 1 and x2 when i is 2.

Another common form of metaprogramming is when the program can change itself in non-trivial fashions. LISP is well known for this. The program would change another part of the program based on its state. This allows a level of flexibility to make decisions at runtime that is very difficult in most popular languages today. It is also worth noting that back in the good old days of programming in straight assembly, programs that altered themselves at runtime were necessary and very commonplace.

It can be implemented by using any of these and in combination:

• Reflection
• DSLs (Domain Specific Languages)
• (.NET) or Annotations (Java)
• Generics (.NET/Java)
• Templates (C++, Haskell)
• Method_missing (Ruby)
• Closures / first class functions / delegates
• AOP - Aspect Oriented Programming

Different languages that supports Metaprogramming

• Curl
• D
• Groovy
• Haskell
• Lisp
• Maude system
• MetaL
• MetaOCaml
• Nemerle
• Perl
• Python
• Ruby
• Smalltalk
• C++

Metaprogramming in Curl

Curl is an object oriented programming language mainly designed for interactive web content and widely used in enterprise B2B or B2C applications. It is reflection oriented language, so metaprogramming is done in Curl using reflection. Using evaluate() procedure an application can create Curl source code at run time and cause it to be compiled and executed in a chosen environment (package).

• Example:7

 evaluate (proc)
 public {evaluate
     exp:any,
     base-url:Url = unspecified-url,
     package:#OpenPackage = null
 }:any

In the above example, exp is an expression, which should be a StringInterface or CurlSource containing the source to be evaluated or an url specifying a file from which the source should be read. Package is the environment for evaluation and base-url is the base-url for the code to be evaluated.

Metaprogramming in D

D is an object oriented, multiparadigm programming language mostly influenced from C++. It has redesigned some aspects of C++ and also influenced features from other programming languages like Java and C#. In D, metaprogramming is supported by a combination of templates, tuples, string mixins and compile time function execution. Following are examples of template metaprogramming and metaprogramming using string mixins.

• Example: Template metaprogramming 6

Following example is a template crypt where the encryption adds a constant value 5 to the string. D has built in 'static if' statement used for compile-time conditional construct.

import std.string;
import std.stdio;
template makechar(int c)
{ 
    const char [] makechar=
    x"000102030405060708090a0b0c0d0e0f
    e0e1e2e3e4e5e6e7e8e9eaebecedeeef
    f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"[c..c+1];
}
template crypt(char[] n, int val)
{	
    static if (n.length == 1) const char[] crypt = makechar!(n[0] + val);
    else const char[] crypt = makechar!(n[0] + val) ~ crypt!(n[1..n.length], val);
}
int main()
{
    char[] str = crypt!("This is metaprogramming in D", 5);
    printf("%.*s\n", str);
    return 0;
}

• Example: String mixin metaprogramming 5

D code is generated using string operations at compile time using string mixins, combined with compile-time function execution. This can be used to parse domain-specific languages to D code, which will be compiled as part of the program.

 import FooToD; // hypothetical module which contains a function that parses Foo source code and returns equivalent D code
 void main()
 {
     mixin(fooToD(import("example.foo")));
 }

Metaprogramming in Groovy

Groovy is an object oriented programming language for Java platform with dynamic language features. It is also used for scripting. It uses metaprogramming in the following way.

Below example demonstrates using classes from the Jakarta Commons Lang package for metaprogramming. All of the methods in org.apache.commons.lang.StringUtils coincidentally follow the Category pattern — static methods that accept a String as the first parameter. This means that we can use the StringUtils class right out of the box as a Category.1

import org.apache.commons.lang.StringUtils
 class CommonsTest extends GroovyTestCase{
  void testStringUtils(){
    def word = "Introduction"
    word.metaClass.whisper = {->
      delegate.toLowerCase()
    }
     use(StringUtils, StringHelper){
      //from org.apache.commons.lang.StringUtils
      assertEquals "Intro...", word.abbreviate(8)
       //from the StringHelper Category
      assertEquals "INTRODUCTION", word.shout()
       //from the word.metaClass
      assertEquals "introduction", word.whisper()
    }
  } 
}
class StringHelper{
  static String shout(String self){
    return self.toUpperCase()
  }
}

Metaprogramming in Haskell

Haskell is a statically typed, general purpose, standardized, purely functional programming language. It is not an object oriented programming language. It supports metaprogramming by compile time code generation using templates. Templates are type-safe compile-time meta-programming. Compile time manipulation of code gives Haskell a lot of power. Template Haskell is used by programmers for conditional compilation which is extremely useful for compiling a single program for different platforms, or with a different configuration, or with different debugging options. Following is an example of template Haskell.

• Example : 14

This example is to select an element from a tuple of arbitrary size. The '$' says 'evaluate at compile time'.

 > $(sel 2 3) ('x','y','z')
 'y'
 > $(sel 3 4) ('x','y','z','w')
 'z'
 sel :: Int -> Int -> ExpQ
 sel i n = [| \x -> $(caseE [| x |] [alt]) |]
   where alt :: MatchQ
         alt = match pat (normalB rhs) [] 
         pat :: Pat
         pat = tupP (map varP as) 
         rhs :: ExpQ
         rhs = varE(as !! (i -1)) -- !! is 0 based
         as :: [String]
         as = ["a" ++ show i | i <- [1..n] ]

Alternately:

 sel' i n = lamE [pat] rhs
   where pat = tupP (map varP as)
         rhs = varE (as !! (i - 1))
         as  = [ "a" ++ show j | j <- [1..n] ]

Metaprogramming in Lisp

Lisp is a multiparadigm, functional, procedural and reflective programming language. Lisp provides DSL for metaprogramming. In lisp Metaprogramming is done using a construct called a "macro".

• Example : 10

Let us consider, the following s-expression for a to-do list in Lisp. Suppose we're writing a to-do manager application. We keep our to-do items serialized in a set of files and when the program starts up we want to read them and display them to the user. One way is to read the entire list and show the relevant data to the user. Instead of writing code to walk the s-expression that stores our data we could write a macro that allows us to treat data as code.

 (todo "Homeworks"
    (item (priority high) "Homework for subject 1.")
    (item (priority medium) "Homework for subject 2.")
    (item (priority medium) "Homework for subject 3."))

Corresponding macros for the to-do list items that will get called by lisp compiler and will transform the to-do list into code will be as follows. Now the to-do list will be treated as code and will be executed. Suppose all we want to do is print it to standard output for the user to read.

 (defmacro item (priority note)
    '(block
        (print stdout tab "Priority: "
            ~(head (tail priority)) endl)
        (print stdout tab "Note: " ~note endl endl)))

Metaprogramming in Maude System

Maude system is an implementation of rewriting logic and an heavy emphasis on reflection based metaprogramming. The following example illustrates it using a set of rewrite rules using reflection. Here NAT represents the set of natural numbers. s(0) indicates successor of 0 i.e. 1. the rewriting rules r1 are created using the reflection of itself.

• Example: 17

  mod PERSON is
  including NAT . *** NAT is the set of natural numbers

  sort Person .
  sort State .

  op married : -> State [ctor] .
  op divorced : -> State [ctor] .
  op single : -> State [ctor] .
  op engaged : -> State [ctor] .
  op dead : -> State [ctor] . 
  op person : State Nat -> Person [ctor] .

  var N : Nat .
  var S : State .

  rl [birthday] :
  person (S, N) => person (S, N + s(0)) . 
  rl [get-engaged] :
  person (single, N) => person (engaged, N) .
  rl [get-married] :
  person (engaged, N) => person (married, N) .
  rl [get-divorced] :
  person (married, N) => person (divorced, N) .
  rl [las-vegas] :
  person (S, N) => person (married, N) .
  rl [die] :
  person (S, N) => person (dead, N) .
  endm

In the above example a person still is the same person after he is marriedhis marital status has been changed. This is shown in the rewrite rules. These rules do not have to be confluent and terminating so it does matter a great deal what rules are chosen to rewrite the term. The rules are applied at 'random' by the Maude system, indicating no surity that one rule is applied before another rule and so on.

Metaprogramming in MetaL

MetaL is shorthand for Meta-programming Language. MetaL programs source code is based on XML. MetaL compiler engine can be used to generate the same program from MetaL source code to potentially any target language. Currently supported target languages are PHP, Java and Perl.
The MetaL compiler engine traverses XML source code and executes actions associated with each command tag. Actions may be immediate, like creating a file, or generating some output, like the lines of code of a program in a given target language.Command actions are implemented by compiler modules. Such modules implement the different kinds of commands of the language, like execution flow, variable and expression manipulation, object oriented programming classes, etc.. Each module is responsible for generating the code in each of the supported target languages Code snippet is given below,check 16 for full code.

<class>
 <documentation>
  <idiom>en</idiom>
  <purpose>Composing and generating XML documents.</purpose>
  <usage>The class has several independent functions that should be used
   to compose the structure of an XML document.  When the structure is
   fully composed, the <functionlink>write</functionlink> should be used
   to output the document in the XML format.</usage>
   <examplecode>
   <comment>First create an object of the class.</comment>
   <set> <xmlwriterobject /> <createxmlwriterobject /> </set>
   <comment>Now, start defining the XML document from the root tag.</comment>
   <set> <hashvariable>noattributes</hashvariable> <hash /> </set>
   <void><addxmlwritertag>
    <arguments>
     <tag><string>myxmldocument</string></tag>
     <attributes><variable>noattributes</variable></attributes>
     <parent><emptystring /></parent>
     <path><variable>root</variable></path>
     <indent><true /></indent>
    </arguments>
   </addxmlwritertag></void>   
   <comment>...and a stylesheet for displaying the document in particular way in XML capable browsers.</comment>
   <set>
    <xmlwriterobjectvariable>stylesheettype</xmlwriterobjectvariable>
    <string>text/xsl</string>
   </set>
   <set>
    <xmlwriterobjectvariable>stylesheet</xmlwriterobjectvariable>
    <string>myxmldocument.xsl</string>
   </set>
   <comment>When you are done with the XML document definition, generate it.</comment>
   <if>
    <istrue>
     <writexml>
      <arguments>
       <output><variable>output</variable></output>
      </arguments>
     </writexml>
    </istrue>
    <then>
      <comment>If the document was generated successfully, you may not output it.</comment>
    <buffervalue><variable>output</variable></buffervalue>
    </then>
    <else>
     <comment>If there was an error, output it as well.</comment>
     <group><string>Error: </string><join /><xmlwriterobjectvariable>error</xmlwriterobjectvariable></group>
    </else>
   </ if>
   </examplecode>
 </documentation>
 <file><installpath />/library/xml/xml_writer_class<languageextension /></file>
 <basepath><openpath /></basepath>
 <tag>METAL_LIBRARY_XML_XML_WRITER_CLASS</tag>
 <version>@(#) $Id: sample.html,v 1.7 2006/01/12 19:51:37 mlemos Exp $</version>
 <copyright>Copyright © (C) Manuel Lemos 2001-2002</copyright>
 <title>XML writer</title>
 <author>Manuel Lemos</author>
 <authoraddress>mlemos@acm.org</authoraddress>
<variable>
  <name>outputencoding</name>
  <type>STRING</type>
  <value>utf-8</value>
  <documentation>
   <idiom>en</idiom>
   <purpose>Define the character set encoding of the output
    document.</purpose>
   <usage>Currently the class supports only the encodings utf-8
    or iso-8859-1.</usage>
  < /documentation>
 </variable>
< function>
do something
</function> 
</class>

Metaprogramming in MetaOcaml

MetaOCaml is a multi-stage extension of the OCaml programming language. It is a research language and uses DSL for metaprogramming. The basic approach to implementing DSLs in MetaOCaml is the staged interpreter approach. First, an interpreter is implemented and tested for the DSL. Then, the three staging constructs are used to produce an implementation that performs the traversal of the DSL program in a stage earlier than the execution of the essence of the program. Implementations derived in this manner can be as simple as an interpretive implementation, and at the same time have the performance of a compiled implementation.

Metaprogramming in Nemerle

Nemerle is statically typed, high-level, object-oriented programming language for the .NET platform. It has macro facility, aspect oriented programming features which give Nemerle power of metaprogramming.

• Example: 15

For selecting data from employee table based on firstname as parameter the general way to write code in Nemerle is as follows.

 string sql = "SELECT name, address FROM employee WHERE name = :a";
 NpgsqlCommand dbcmd = new NpgsqlCommand (sql, dbcon, dbtran);
 dbcmd.Parameters.Add("a", myparm);
 NpgsqlReader reader = dbcmd.ExecuteReader();
 while(reader.Read()) {
     string name = reader.GetString (0);
     string address = reader.GetString (1);
     System.Console.WriteLine ("Name: {0} {1}", name, address) 
 }
 reader.Close();
 dbcmd.Dispose();

In the above scenario, database connection is tested during run time not compile time and another drawback is the compiler has to remember the variable names during compilation and store them. To avoid this a macro can be used as follows.

 ExecuteReaderLoop (
     "SELECT name, address FROM employee WHERE name = $myparm",
     dbcon,
   {
   System.Console.WriteLine ("Name: {0} {1}", name, address) 
   });

Metaprogramming in Perl

Below code is for reducing the amount of boiler-plate code necessary for writing Perl subs.Here is an example of a module that uses "handy" new module.13

package SomeTestPackage2;
use strict;
use warnings;
use Data::Dumper;
use constant IVARS => qw[$none $href $aref $non_empty_aref];
BEGIN {
use base qw[SomeAttributes2];
__PACKAGE__->import( IVARS );
}
use constant custom_aref => q{    confess( '!!!package!!! !!!ivar!!! must be an array ref' ) if 'ARRAY' ne ref !!!ivar!!!; };
sub self_sub : Method( qw/ $none :none $href :href $aref custom_aref $non_empty_aref :non_empty_aref / ) { #
    return ( ref $self, ref $href, ref $aref, scalar @{ $non_empty_aref } );
}
1;

The previous code snip shows how the meta programming interface works for the most part.First,we specify a list of instance variables,that we would like to pull into subs. This pre-declaration is necessary for syntax reasons.Next, we can specify "custom" attributes as package subs if we desire. The difference between a custom attribute and a canned/common one is that canned attributes begin with a colon and custom do not.Now, in our sub attribute we specify a list of variables we would like pulled into our sub along with any attributes we would like called on them.

This is the test script running the prior code

#!/usr/bin/perl
use strict;
use warnings;
use Test::More qw[no_plan];
use SomeTestPackage2;
do { # Successfull call to self_sub
    my( $test1 ) = bless( {}, 'SomeTestPackage2' );
    my( @self_sub ) = $test1->self_sub( 'hi', { }, [ ], [ 1 ] );
    ok( $self_sub[0] eq 'SomeTestPackage2', 'ref $self' );
    ok( $self_sub[1] eq 'HASH',             'ref $href' );
    ok( $self_sub[2] eq 'ARRAY',            'ref $aref' );
    ok( $self_sub[3] > 0,                   'non empty array' );
};

Here is the output of running the previous script.

$ perl test2.pl
ok 1 - ref $self
ok 2 - ref $href
ok 3 - ref $aref
ok 4 - non empty array
1..4

Metaprogramming in Python

Python supports metaprogramming through metaclasses, decorators and descriptors.

• Metaclasses considered as "the class of a class" enable the creation of new “types of classes” which can customize the class creation process and/or add class methods and attributes.
• Decorators look like Scala annotations but are applied only to function definitions and yield (possibly transformed) functions.
• Descriptors provide a very concise way to add custom properties to classes that invoke logic but syntactically manipulated as regular attributes. They can also carry extra information (such as titles, database column mappings, etc) that can be useful when accessed through reflection.

class _AutoWriteDescriptor(object):
    Descriptor for AutoWrite.
      Implements an attribute that allows an initial setting then
       calls the setter.
    
    def __init__(self, name, ignore_identical=True, prefix='_set_'):
        self.name = name
        self.ignore_identical = ignore_identical
        self.prefix = prefix
    def __get__(self, obj, objtype=None):
        self.check_dictionary(obj)
        return obj._auto_write_dict[self.name]
    def __set__(self, obj, new_value):
        self.check_dictionary(obj)
        if self.name in obj._auto_write_dict:
            if (self.ignore_identical is False or
                new_value is not obj._auto_write_dict[self.name]):
                attr = getattr(obj.__class__,
                               "%s%s" % (self.prefix, self.name), None)
                obj._auto_write_dict[self.name] = attr(obj, new_value)
        else:
            obj._auto_write_dict[self.name] = new_value
    def __delete__(self, obj):
        raise AttributeError("cannot delete AutoWrite attribute")
    def check_dictionary(self, obj):
        if getattr(obj, "_auto_write_dict", None) is None:
            raise AttributeError(
                AutoWrite dictionary missing.
                   Class %s probably does not call AutoWrite.__init__
                % obj.__class__)

Above example is a simple Descriptor. The only thing to understand about Descriptors is that they are attributes. So the methods above live on the attribute, not on the main object. When Python accesses an attribute it checks and, if it's a Descriptor, it invokes the appropriate method (get if the attribute is being read, set if its value is being changed, delete if the attribute is being deleted)11.

The advantage of doing things this way is twofold. First, we end up uniting methods and functions.Second,we have an efficient way to change how certain attributes behave. This is much better than changing the main object's __getattr__ and _setattr__ because the extra code is only invoked for the "special" attributes - there no speed penalty otherwise.

If the value is being read, the actual value is pulled from a dictionary that is stored on the main object.If the value is being set for the first time (ie when the object is populated from the database) the value is stored. Subsequent updates (ie when the object is modified via the user interface) call the setter method.

Metaprogramming in Ruby

We have a C program that needs to include a PNG image, but for some reason, the deployment platform can accept one file only, the executable file. Thus, the data that conforms the PNG file data has to be integrated within the program code itself. To achieve this, we can read the PNG file beforehand and generate the C source text for an array declaration, initialized with the corresponding data as literal values. This Ruby script does exactly that:

INPUT_FILE_NAME = 'ljlogo.png'
OUTPUT_FILE_NAME = 'ljlogo.h'
DATA_VARIABLE_NAME = 'ljlogo'
File.open(INPUT_FILE_NAME, 'r') do |input|
  File.open(OUTPUT_FILE_NAME, 'w') do |output|
    output.print "unsigned char #{DATA_VARIABLE_NAME}[] = {"
    data = input.read.unpack('C*')
    data.length.times do |i|
      if i % 8 == 0
        output.print "\n    "
      end
      output.print '0x%02X' % data[i]
      output.print ', ' if i < data.length - 1
    end
    output.puts "\n};"
  end
end

Metaprogramming in SmallTalk

Smalltalk is an dynamically typed, object-oriented, reflective programming language. As refection is one feature of smalltalk metaprogramming can be done using this feature. In SmallTalk run time code generation is done by manipulating strings. 12

• Example :

In this example, metaprogramming is done by creating anonymous classes and giving them behavior, then instantiate them run time.

      | newClass newInstance |
      newClass := Behavior new. "create anonymous behavior"
      newClass compile: 'theAnswer ^42'. "Add a method for instances"
      newInstance := newClass new. "create an instance"
      Transcript show: newInstance theAnswer; cr. "shows 42"

Metaprogramming in C++

These Metaprograms run before the load time of the code they manipulate. Precompilers: C/C++ precompiler. Open compilers: like writing transformations on AST (hygenic macros). Two level languages: C++ templates3.

template <int i>
struct D
{
    D(void *);
    operator int();
};
template <int p, int i>
struct is_prime
{
    enum { prim = (p%i) && is_prime<(i>2?p:0), i>::prim };
};
template <int i>
struct Prime_print
{
    Prime_print<i-1>    a;
    enum { prim = is_prime<i,i-1>::prim };
    void f() { D d = prim; }
};
struct is_prime<0,0> { enum { prim = 1 }; };
struct is_prime<0,1> { enum { prim = 1 }; };
struct Prime_print<2>
{
    enum { prim = 1 };
    void f() { D<2> d = prim; }
};
void foo()
{
    Prime_print<10> a;
}

// output:
// unruh.cpp 30: conversion from enum to D<2> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<3> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<5> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<7> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<11> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<13> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<17> requested in Prime_print
// unruh.cpp 30: conversion from enum to D<19> requested in Prime_print

Index

• Multiparadigm language : Multiparadigm languages provides a framework in which programmers can work in a variety of styles, freely intermixing constructs from different paradigms. Examples are C, C++, Java, Ruby, Oz, Visual Basic, Perl, Python etc. The design goal of multiparadigm languages is to allow programmers to use the best tool for a job, admitting that a single paradigm can not solve all problems in most efficient or easiest way.

• Template metaprogramming : Template metaprogramming is a metaprogramming technique in which templates are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled.

• Reflection oriented language : In reflective programming languages computer programs can observe and modify its own structure and behavior.

• Stack based programming language : This type of languages uses stack based models for passing parameters.

• s-expression : It is called symbolic expression i.e. an way to represent semi-structured data in human-readable textual form mostly made of symbols and lists. It is used in Lisp family of languages.

• Type-safety : Type safety is a property of some programming languages which indicates the use of a type system to prevent certain erroneous or undesirable program behavior called type errors.

• Template metaprogramming : This is a technique in which templates are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled.

References

1. http://www.ibm.com/developerworks/java/library/j-pg06239.html
2. http://stackoverflow.com/questions/514644/what-exactly-is-metaprogramming
3. http://aszt.inf.elte.hu/~gsd/halado_cpp/ch06s04.html
4. http://en.wikipedia.org/wiki/Multi-paradigm_programming_language#Multi-paradigm_programming_language - Multi-paradigm_programming_language
5. http://en.wikipedia.org/wiki/D_%28programming_language%29 - D Programming language
6. http://www.the-interweb.com/serendipity/index.php?/archives/66-Template-meta-programming-in-D.html - Template metaprogramming in D
7. http://developers.curl.com/userdocs/CurlDocs.htm#docs/en/api-ref/evaluate.html - Curl evaluation fumction
8. http://www.curl.com/pdf/content-language.pdf - Curl a content language for the Web
9. http://en.wikipedia.org/wiki/Forth_%28programming_language%29 - Meta compilation in Forth
10. http://www.defmacro.org/ramblings/lisp.html - Metaprogramming in Lisp
11. http://www.acooke.org/cute/PythonMeta0.html - Metaprogramming in Python
12. http://groups.google.com/group/comp.lang.smalltalk/browse_thread/thread/bb0519b6b4c3e850?pli=1 - Metaprogramming in SmallTalk
13. http://www.perlmonks.org/?node_id=579458 - Perl Metaprogramming
14. http://www.haskell.org/haskellwiki/Template_Haskell - Metaprogramming in Haskell
15. http://en.wikipedia.org/wiki/Nemerle#New_language_constructs - Metaprogramming using macro in Nemerle
16. http://www.meta-language.net/sample.html#metal -MetaL
17. http://en.wikipedia.org/wiki/Maude_system - Reflection based metaprogramming in Maude system