Building C++ modules on node.js 7.0

I did a bunch of Googling this morning looking for how to build a C++ node.js module, using the new gyp build system. There a plenty of blog posts on doing native module builds via node-waf but not with the gyp_addon python script.

I took the liberty of basing this example off Teemu Ikonen’s blog entry on threaded extensions.

Here is the C++ node.js module source. This is an extension which creates a thread, and then makes a callback every 5 seconds. It’s a good example because it also uses some of node’s internal libraries (namely libev):

GYP build system: gyp_addon
Go ahead and look at the original blog entry if you want to build with node-waf. Going forward, however, with new versions of node you will need to use the gyp build system. Luckily, there is a great project on github which will make this easier depending on your circumstances. This is TooTallNate’s node-gyp script, which itself uses node to make cross-platform building of your extension easier. I highly suggest going this route if

  • You are developing a standard extension for the node.js system
  • You are not doing cross compiling
  • and you want cross platform support (who wouldn’t)

On the other hand, if you are working off the tip and/or are doing cross-compiling, or some other case where you need a custom node.js source tree itself, then you may want to go this route… First you need a gyp build file. Here is one for the texample.cpp extension. More description follows.

Now, we are doing a few extra things here:

  • We are specifying a build for x86_64 arch. See target_arch line in variables. (For some reason node pays no attention to config.gypi in the root of the source folder. Not sure why… post something if you know.)
  • And to also support x86_64 we have a line ['cflags'] line under Linux section specifying -fPIC. You need this option when building shared libraries on x86_64 for node.
  • Notice node_root, a variable defined at the top. This is where the node.js source is in comparison to your source files
  • We also include some non-standard include directories, specifically stuff used inside node. deps/uv/include/uv-private is where the libev headers are. And this example uses those.

Building
For gyp_addon to work, you will need to run it out of the tools directory, unless you have node more permanently installed. Since texample.gyp references everything relative to where it (texample.gyp) is, then this will be fine. Just point gyp_addon to it:

ed@computer:~/node-0.7.0/tools$ ./gyp_addon ../../src/texample.gyp
ed@computer:~/node-0.7.0/tools$ make
  ACTION Regenerating Makefile
  CXX(target) out/Debug/obj.target/texample/../../src/texample.o
  SOLINK_MODULE(target) out/Debug/obj.target/../../src/texample.node
  SOLINK_MODULE(target) out/Debug/obj.target/../../src/texample.node: Finished
  COPY out/Debug/texample.node
ed@computer:~/node-0.7.0/tools$ ls out/Debug/
linker.lock  obj.target  src  texample.node

and your output should be similar.

At some later point we will show an example of cross-compiling using this method.

Quick test:

ed@computer:~/node-0.7.0$ ./node
> texample = require('./texample');
{ start: [Function] }
> texample.callback = function(i) {
... console.log("bang: " + i );
... }
[Function]
> texample.start();
true
> bang: 0
bang: 1
bang: 2
bang: 3

“Unable to open shared library”
If you get such an error, and you know for certain that node.js really is finding your library file (you could verify with strace), its likely that your object format’s are not compatible. This means you built the node.js for x86_64 and your object file is ia32. You can verify this by doing something similar to:

ed@computer:~/node-0.7.0/tools$ objdump -g out/Debug/texample.node
out/Debug/texample.node:     file format elf32-i386

This command also run on the node executable needs to output the same format, here elf32-i386. If you’re on 64-bit linux, it would be elf64-x86-64. Putting this here because I suspect folks running node on x64 will run into this issue.

Arduino Project (Bell Enclosure Part 1)

So I’ve been printing away many different projects and fine tuning my Reprap 3d Printer when my wife asks me when I am going to get her bell working again. A while back I made a bell that rings whenever someone visits her blog. We reorganized our home office and the ardruino, servo and bell that were literally screwed into the desk and wall had to get taken down.

Well this time I am going to make the whole set up a lot more portable and enclosed with the use of my 3d printer. In about 2 hours I designed and printed this enclosure:

Arduino Enclosure

Arduino Enclosure 2

The design is a little snug and I plan on modifying it and putting it on thingyverse.com in the near future. You may notice there isn’t a top yet. That is part of the plan. I now an going to design and print a top that will allow me to mount the bell and servo and attach it to my wall with only one screw. I figure this base can be used in other arduino projects that follow.

Merging source code with Meld

Merging source code from one repository tree to another can be a daunting task.  At WigWag we utilize the Contiki OS, “The Operating System for the Internet of Things,”  for all of our microcontroller devices.  Contiki is under constant development with daily changes and therefore every once in a while we like to merge in the latest work from the Contiki team, with our source repository.  With Meld, this merging effort is very simple.

Installation
If you are on Ubuntu, installing Meld is simple.

$ sudo apt-get install meld

A great list of similar tools for your OS can be found on this Stackoverflow article.

Setting up Meld for the merge

Meld source selection and target

  • Open Meld and chose File New
  • Select the “Directory Comparison” in the center tab
  • Point your source to “Mine” and the other source to “Original”… Press OK

Understanding the layout

meld window mine and target source

  • On the right is our source tree “Mine”
  • On the left is the new version of “contiki” just freshly checked out using GIT
  • All the files are displayed in a tree.   Colors represent the differences.
  • Green is a new file
  • Grey crossed out is a non-existing file
  • Red represents a file with a delta

Understanding the toolbar

meld toolbar

  • Up and down arrows “go to the next difference”
  • The left arrow pushes changes form “mine” to “contiki” –an operation we don’t need at this time
  • The right arrow pushes changes from “contiki” to “mine”
  • delete will delete a file
  • Hide will hide a file
  • Case will turn on and off case sensitivity

The Merging operation

  • Simply start at the top of the tree, and scroll down through the new files, missing files and deltas
  • Chose which files to merge based on your merging preferences.  (See our merging rules below)

Merging Deltas

Meld file delta red highlight

  • When encountering a delta you will see a red file on both sides
  • Double click the delta
The differences show up highlighted on the left and right
  • The comparison window will show you the differences between the left and right
  • You can push each difference into the other side by clicking the arrows
  • If you make a mistake, UNDO works just fine…. by the way this is a full editor too.  So you can also just edit the code in place
copy in text from on side to the other hold control
  • by holding down the control key, you can “copy text” from one side to the other, below or above the other source.
  • to actually complete the push using linux, you have to press ctrl-shift click

Our rules for merging

While we are still in a very early development stage, we follow a fairly loose merge policy.  clearly once we start shipping product, our merge rules will become much more stringent.

  • New Contiki file –> Push into Mine
  • Changed Contiki file, no WigWag edits in Mine –> Push into Mine
  •  changed Contiki file, with WigWag edits in Mine –> Push the delta lines that don’t effect our code, where it does effect our code, push in the contiki changes and comment them out

Quick demo video

  • to see the whole thing in effect, watch this short video

Using ICU4C Regex test program

If you need Unicode support, your goto library for the real deal is ICU. ICU also has very solid regex support. And while ICU has plenty of documentation, examples on the net sometimes fall short. The following is a simple command line program which will do both regex replacement, matching and capture groups. It also serves as a decent example of how to use the library. ICU is the basis for unicode and regex support in Android and iOS as well. So this little test program will help you there as well.

ICU likes to do everything in UTex, which I think is always UTF-16. So, to do UTF-8 or ASCII you need to conversion. The program converts the UTF-8 on the command line, runs it through the regex and returns.

Usage

Since you are running this on the command line, you need to escape ‘\’ some chars in bash, mainly backslashes and exclamation marks. It will printout the regex it actually used.

Source on gist. (Easier to copy)

$ ./regextest "Hello(.*)" "Hello Bob"
pattern:     -->[Hello(.*)]<--
test on:     -->[Hello Bob]<--
MATCH: string matches regex
Capture group 0: -->[Hello Bob]<--
Capture group 1: -->[ Bob]<--

$ ./regextest "(ftp|http|https):\\/\\/(\\w+:{0,1}\\w*@)?(\\S+)(:[0-9]+)?(\\/|\\/([\\w#\!:.?+=&%@\!\\-\\/]))?" http://www.google.com

pattern:     -->[(ftp|http|https):\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#\!:.?+=&%@\!\-\/]))?]<--
test on:     -->[http://www.google.com]<--
MATCH: string matches regex
Capture group 0: -->[http://www.google.com]<--
Capture group 1: -->[http]<--
Capture group 2: -->[]<--
Capture group 3: -->[www.google.com]<--
Capture group 4: -->[]<--
Capture group 5: -->[]<--
Capture group 6: -->[]<--

Source

// The following code is freeware:
// regextest.c
// Author: ed@wigwag.com
//
// Simple test program for regex using ICU's regex matching.

#include <stdio.h>
#include <getopt.h>
#include <string.h>
#ifdef __cplusplus
#include <cstdlib>
#endif
#include <unicode/regex.h>
#include <unicode/utext.h>
#include <unicode/errorcode.h>
#include <unicode/ucnv.h>
#include <unicode/utypes.h>
#include <unicode/uchar.h>

static const char OptionsInfo[] = "Usage: regextest [-r] \"PATTERN\" \"TEST STRING\"\n"
								  "-r : replace mode -> \"PATTERN\" \"TEST STRING\" \"replacement\"\n"
								  "You need to escape these items like this: \\\" and \\$ b/c of bash.\n"
								  "\n";

/* ICU library libs/programs...
 * http://www.linuxfromscratch.org/blfs/view/svn/general/icu.html
 * http://icu-project.org/download/4.4.html#ICU4C
 *
 */

int main(int argc, char *argv[]) {
	int exitcode = 1;
	bool replace_mode=false;
    int c;
    int digit_optind = 0;

    while (1)
      {
        c = getopt (argc, argv, "r");
        if (c == -1)
        	break;

        switch (c)
          {
          case 'r':
        	  replace_mode = true;
        	  break;
          }
      }

	if(argc-optind < 2) {
		printf("%s",OptionsInfo);
		exit(1);
	}
	if(replace_mode && argc-optind < 3) {
		printf("%s",OptionsInfo);
		exit(1);
	}

	printf("pattern:     -->[%s]<--\n", argv[optind]);
	printf("test on:     -->[%s]<--\n", argv[optind+1]);

	UErrorCode        status    = U_ZERO_ERROR;
	UText *regex1 = NULL;
	UText *matchthis = NULL;

	//	static const char *regex_validate_string = "(?:cp\\:([0-9]+)\\:){0,1}\"(.*)\"";

	regex1 = utext_openUTF8(regex1, argv[optind], -1, &status);
	//regex1 = utext_openUTF8(regex1, regex_validate_string, -1, &status);
	matchthis = utext_openUTF8(matchthis, argv[optind+1], -1, &status);

	RegexMatcher *matcher = new RegexMatcher(regex1, 0, status);
	if (U_FAILURE(status)) {
		// Handle any syntax errors in the regular expression here
		printf("Syntax error in regex?\n");
		icu::ErrorCode ec;
		ec.set(status);
		printf("Error was: %s\n",ec.errorName());
		utext_close(regex1);
		utext_close(matchthis);
		delete matcher;
		exit(exitcode);
	}

	if(!replace_mode) { // MATCH TEST

		//	UnicodeString    stringToTest = "Find the abc in this string";
		matcher->reset(matchthis);

		if (matcher->matches(status)) {
			// We found a match.
			printf("MATCH: string matches regex\n");
			//	   int startOfMatch = matcher->start(status);   // string index of start of match.

			if(matcher->groupCount() > 0) {
				UConverter *conv = ucnv_open("US-ASCII", &status);
				for(int x=0;x<=matcher->groupCount();x++) {
					//				UText *grp = NULL;
					UnicodeString US = matcher->group(x, status);
					if (U_FAILURE(status)) {
						icu::ErrorCode ec;
						ec.set(status);
						printf("Error was: %s\n",ec.errorName());
					} else {
						UChar *out = (UChar *) malloc(1000);
						char *outcs = (char *) malloc(1000);
						US.extract(out,1000,status);
						ucnv_fromUChars(conv,outcs,1000,out,u_strlen(out),&status);
						printf("Capture group %d: -->[%s]<--\n", x, outcs); // works as long as UTF8
						//					printf("Capture group %d: %s\n", x, grp); // works as long as UTF8
						if(out) free(out);
						if(outcs) free(outcs);
					}
				}
			}

			exitcode = 0;
		} else {
			printf("FAIL: no match\n");
			exitcode= 2;
		}

	} else { // REPLACER TEST
		UText *replacedtxt = NULL;
		UText *replacement = NULL;
		replacement = utext_openUTF8(replacement, argv[optind+2], -1, &status);
		matcher->reset(matchthis);
		printf("replacement: -->[%s]<--\n", argv[optind+2]);

		replacedtxt = matcher->replaceAll(replacement,replacedtxt,status);
		if (U_FAILURE(status)) {
			// Handle any syntax errors in the regular expression here
			printf("Syntax error in regex?\n");
			icu::ErrorCode ec;
			ec.set(status);
			printf("Error was: %s\n",ec.errorName());
			utext_close(regex1);
			utext_close(matchthis);
			utext_close(replacement);
			delete matcher;
			exit(exitcode);
		}

		if(replacedtxt) {
			// Replacement did something...
			printf("REPLACE: string replaced...\n");
			//	   int startOfMatch = matcher->start(status);   // string index of start of match.
			UChar buf[500];
			printf("new length: %ld\n", utext_nativeLength(replacedtxt));
			utext_extract(replacedtxt,0,500,buf,500,&status);

			if (U_FAILURE(status)) {
				// Handle any syntax errors in the regular expression here
				printf("Syntax error in extraction:\n");
				icu::ErrorCode ec;
				ec.set(status);
				printf("Error was: %s\n",ec.errorName());
				utext_close(regex1);
				utext_close(matchthis);
				utext_close(replacedtxt);
				utext_close(replacement);
				delete matcher;
				exit(exitcode);
			}

			// Yawn... and convert to ascii.  Can we make it any more complicated??
			UConverter *conv = ucnv_open("US-ASCII", &status);
			char *outcs = (char *) malloc(1000);
			ucnv_fromUChars(conv,outcs,1000,buf,u_strlen(buf),&status);

			// Also, use function u_strFromUTF8 / u_strToUTF8

			if (U_FAILURE(status)) {
				// Handle any syntax errors in the regular expression here
				printf("Syntax error in extraction:\n");
				icu::ErrorCode ec;
				ec.set(status);
				printf("Error was: %s\n",ec.errorName());
				utext_close(regex1);
				utext_close(matchthis);
				utext_close(replacedtxt);
				utext_close(replacement);
				free(outcs);
				delete matcher;
				exit(exitcode);
			}
			ucnv_close(conv);

			printf("Replaced result: -->[%s]<--\n", outcs); // works as long as UTF8
//			printf("Capture group %d: %s\n", x, grp); // works as long as UTF8
			if(outcs) free(outcs);
	utext_close(replacedtxt);
		} else {
			printf("Result: No replacement.\n");
		}

		utext_close(replacement);

	}
	utext_close(regex1);
	utext_close(matchthis);
	delete matcher;
	exit(exitcode);
}

Build

$ gcc regextest.cpp -licui18n -licuuc -licudata -o regextest