Other than that not much else is new. I plan on working on getting the USB drivers working on the Atmega8u2 so it shows up as a keyboard and does something simple, like press ‘A’ every 2~3 seconds to verify that its working.

My next goal for the site will be to introduce a tutorial system into the website’s interactive terminal. The current plan is to follow along the outline of the rakudo book and build a set of interactive tutorials to teach new users about rakudo, and perl 6 in general. Maybe I can extend it to use other backend but for now its focused on the rakudo backend.

Any questions or comments are welcome. Please feel free to comment on the site and its usability or any flaws you may find.

My Google Summer of Code (2010) objectives are to integrate libffi into the NCI framework and to build a new Stack Frame builder that takes advantage of the llvm.

NCI Framework – The current NCI system has a few limitations which I am going to try to alleviate. I don’t know if I will be able to remove all of the limitations, but I will try to add all of the capabilities of the libffi library to the core of Parrot. This includes being able to define structures as data types for calling functions, adding a few new data types on systems that support them (this would be 64 bit integers on systems that have 64 bit integers, and etc.), as well as adding improvements for calling functions in foreign libraries. The current NCI system in Parrot is not capable of defining a structure for instance, or int64 types. I plan on implementing all of the supported data types in libffi as parts of the modified NCI system. I do plan on retaining the current functionality of the NCI system for people without libffi. I don’t think parrot currently bundles any third party libraries with parrot, but I do know python, for example, bundles libffi with its source code and builds its by default if you don’t make it use your system libffi.

Stack Frame Builder – There are a number of places for integrating the llvm into parrot. One obvious place to start is with the stack frame builder. Translating the stack frame into llvm-ir and running some of the llvm optimization passes over the resulting code could provide parrot with both a JIT system and some speed ups of the generated code. It could also be possible to dump the llvm-ir code to a file, so you end up with a sort of pbc to llvm-ir translation. The resulting llvm-ir can also be compiled into a native binary or dynamic library, which would also be useful because that could cut out some of the overhead for libraries.

The current plans for the stack frame builder are still being mapped out, but for now I am focusing on the NCI system, and I am hoping to have that knocked out relatively quickly so I can move on to the work with the llvm.

• A fairly well defined grammar for parsing nqp exists. It generates AST nodes for most of the grammar, but not all of the grammar.
  • In the future, it would be useful for AST optimizations, like constant folding
• Code generation supports integers and subs, but it does not yet support other data types like floats, strings, bools, or anything else like regular expressions.
  • Built in basic types should eventually include:
    • Packages
    • Subs
    • Classes
    • Roles
    • Arrays
    • Hashes
    • Ints
    • Nums
    • Strings
    • Bools
• Execution of code generated works.
  • Debugging is surprisingly easy with the gdb, but I will have to see how hard it is in the future. I know that if I start working with a JIT it becomes increasingly difficult to view whats being run on the processor.
• A limited number of binary ops, like add and subtract are working.
• The VM supports method calls, but the code does not currently generate code properly.

Lessons learned:

• LLVM is neat but advanced and complicated
• Grammars are fun, we should do more of those
  • Having a way of defining a grammar is difficult though currently, in C there is yacc/bison but that is a bit limited compared to other systems.
• Grammars are hard, parsing is not easy
  • Tokenizers and Lexers are difficult if you want to contextually call things based off of where they should be in your grammar
• Focusing on construction a real “Compiler” is probably a mistake. Start small, an ast that you can walk and execute each step of the way is better for learning than building a code generation system. Code generation is useful but hard, and can be deferred until later.

Things I would do different if I could start over:

• Focus on the grammar, and making an executable AST rather than translating the AST into code.
• If there was a VM out there I could link to for a runtime, I have not found an easy to use one. It would be nice to have because then I could of focused more on other parts of the language, but VM support is not that hard, just lots of typing.

My strategy for lexical scoping is on any block that adds to the scope, I call a special function that pushes a new Stash onto my current scope, and sets its OUTER to the former scope. Then, at the end of the second of code that causes the scope, I simply pop it by redirection the pointer back to what OUTER pointed to.

So, lets look at an example of code:

my $a := 5;
my $*b := 5;
sub g() {
  my $a := 4;
  my $*b := 4;
  f();
}
sub f() {
  say($a);
  say($*b);
}
g();
# prints:
# 5
# 4

In here this example you can clearly see how scoping works. I’ll make a chart for the of the values in each scope and say on which line numbers it creates the scope:

&f() has a scope too, but no variables are declared in it, so it wouldn’t be required to make a scope there in this case.

So, lexically scoped variables will look in the scope they were defined in as well as their parent scopes for a variable. Since the definition of &g() is in the same lexical scope as $a it, it can see the value of the nearest $a. When you look up a variable, any variable, it looks for the closest variable with the same name, lexical and dynamic scope just start looking in different places. Lexical scope is tied to the a specific place in the stack and looks up from there. Dynamic scope starts at the top of stack and traveses down the stack frame until it finds something with the same name, and then checkes the GLOBAL package for a variable with the same name, the raises an error that its not defined.

As you can see, there wouldn’t be to many complications implementing this, it just a hash with an extra pointer to its OUTER, which is exactly how I am implementing it. But I am glad I have this working now, so I can move on to code generation.

Here is a rough diagram of the relation between objects instances, their containers, and their lexical values:

For garbage collection for now I am simply going to use Boehm GC [1], it should do for now, till I can look into GC a bit more.

The meta model seems pretty solid, granted its from the perl 6 spec. The current nqp-rx doesn’t have its own meta model. It uses parrot’s meta model directly. So, this has lead me to an issue, since I can’t be consistant with nqp, since I don’t want to implement all of parrot. So, instead I am going to steal the MOP (Meta Object Protocol) from perl 6.

Dispatch is going to be interesting. I’d like to implement multi-dispatch, but maybe not just yet. I have been thinking about it, for now I will probably start with single dispatch, and build up to multi-dispatch later.

Here is a sequence diagram of how I think I could accomplish multi-dispatch, sorta. I still need to flush out some of my ideas for how to do this right.

Foot Notes:
1# http://www.hpl.hp.com/personal/Hans_Boehm/gc/

nq-nqp AST Nodes

I will be adding more as I need, but currently I feel I have enough to start with all the basic control flow. I need to figure out how packages will work, and how dispatching will work.

A current need for the system is a localization system for navigation inside the building we are working in. Utilizing OpenCV and web cameras mounted on the robots, I plan on creating a SLAM system that uses maps of the ceiling of our building for navigation purposes. With an upwards facing camera, and a map of the ceiling I should be able to use OpenCV to locate where the robots are on the map.

Right now, I am working on a proof of concept, if this works, I’ll update this and move on to maybe a more general library.

The lexer needs to be able to see the difference between these two operations, so, maybe its not the parsers fault? This would require flex (the lexer I am currently using) to be aware of the context it is in. It does support state based context, but the lexer has to be able to enter and exit the context. It is not callable from the parser generator. So, now I have a problem. Either I make the lexer do really complicated things, or I have a different kind of parser. Perl 5 has a bison based grammar but, its lexer is not at all sane. It has to do some tricky things to look around and find when a / is a regex vs a divide. A simple example of this might be 5 / 5 / 5; If your parser/lexer is not very smart, that might seem like an integer followed by a regular expression followed by an integer. Or is it an integer divided by an integer divided by an integer? Well, I know thats a divide not an regular expression. But its proving difficult, in my limited knowledge of bison and flex, to make bison and flex understand this.

So, talking to some of the folks in the #perl6 IRC channel, it seems nqp and the STD.pm for perl 6 both use a LL strategy for parsing content. Bison produces a LALR(1) style parser, which is a look-ahead left recursion parser. Left recursion produces a bottom-up parser. A LALR(k) or GLR parser would also give me some trouble, since I would have less knowledge of what I am parsing coming from the bottom-up. That doesn’t mean its impossible to make a bottom-up parser for NQP and Perl 6 but it seems more difficult.

I may not understand all the parser details as much as I should, but it seems to me that for now I am going to come up with a LL style parser strategy.