Cowboy Programming Game Development and General Hacking by the Old West

Today’s “programming” exercise: get the source for Ruby 1.9, and build a working version under Visual Studio, just like I did for version 1.8.

Step 1: Search Google, which gives me this starting point:

http://www.megasolutions.net/ruby/Help-building-Ruby-1_9-on-Windows-50037.aspx

> I am looking for some advice on building Ruby 1.9 on windows.

Here’s what I did on XP SP2 running under Parallels.

– Download and install MSFT Visual C++ 2005 Express Edition (gratis)
– Get the Windows Platform SDK
– Install Ruby 1.8.x using 1-click Windows Installer
– Grab Ruby 1.9 from Subversion

– Click through: Start Menu / Platform SDK / Open Build Environment
Window / Windows XP 32-bit … / Set Windows XP … Build Environment
– In the command window that opens, run “c:\Program Files\Microsoft
Visual Studio 8\Common7\Tools\vsvars32.bat”
– win32/configure.bat
– nmake

Result:

C:\dev\ruby\win32>ruby -v
ruby 1.9.0 (2007-04-12 patchlevel 0) [i386-mswin32_80]

Now I’ve already done the first three steps, so my first stumbling point is:

Grab Ruby 1.9 from Subversion

I don’t have Subversion installed, and I’ve never used it before. I’ve been using Perforce for years, and I use the free version of that for my home projects. So, I go to: http://subversion.tigris.org/project_packages.html, which is rather intimidating for the poor Windows user. All I want is the client, but there are a range of download options, none of which look trivial. One mentions a “friendly installer program”, so I click on that, and phew, a nicer list of install packages, I choose Subversion 1.4.5, download and run, accepting all defaults.

So, I feel like I should be able to go to the command line and type something in to get the Ruby 1.9 source, and with a few seconds of browsing, I find the correct instructions at http://www.ruby-lang.org/en/community/ruby-core/ :

svn co http://svn.ruby-lang.org/repos/ruby/trunk ruby

Imagine my surprise when this actually worked! Download seems a little slow, but it’s chugging away, downloading rather a lot of stuff (most of which I’m sure I do not need). It took about five minutes, and finally informed me it had checked out version 13655.

I’d run the above command line inside a directory \ruby-1.9, which then gives me the source tree in \ruby-1.9\ruby\, so I open a visual studio command prompt and do:
cd \ruby-1.9
md \build
cd build
\ruby-1.9\ruby\win32\configure

which gives me:
Creating Makefile
confargs.c
type `"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\nmake.exe"' to make ruby for mswin32.

So I type “nmake”, and sadly after a nice start (several files are compiled) I get a useful error:
bison -o y.tab.c /ruby-1.9/ruby/parse.y
'bison' is not recognized as an internal or external command,
operable program or batch file.
NMAKE : fatal error U1077: 'bison' : return code '0x1'
Stop.

Hmm, I don’t remember having to have bison installed for 1.8. Bison is the GNU parser generator, which takes a .y file (the grammar definition) and makes a .c file (the parser). It seems like a simple case of “missing tools” from the “why won’t it build” list. But I’m a little confusled that 1.8 actually has a file parse.c. If I try nmake parse.c in the 1.8 tree I get “‘byacc’ is not recognized ….”, which shows they have switched tools from byacc (Berkeley Yacc) to Bison (both the same kind of thing). But that’s not the problem, as I don’t have byacc either. Nope, the problem is that parse.c is not part of the subversion repository, which is very reasonable for a development source tree, as it’s dependent on parse.y. Unfortunately I now have to install Bison. I’m beginning to fear I might have to install Cygwin, eek!

So, I download the Bison 2.1 Windows install package and run it -which seems to do everything except set the path, so I add C:\Program Files\GnuWin32\bin to my PATH variable, restart my Visual Studio 2005 command prompt, an re-run nmake, and shucks if there is not another error:

bison -o y.tab.c /ruby-1.9/ruby/parse.y
sed -e "s!^ *extern char \*getenv();!/* & */!;s/^\(#.*\)y\.tab/\1parse/"
y.tab.c > parse.c
'sed' is not recognized as an internal or external command, operable program or batch file.
NMAKE : fatal error U1077: 'sed' : return code '0x1'


Great, another GNU tool that is missing. Luckily it seems like all these tools are available under GnuWin32, so I can simple download the Sed installer, and it goes in the same bin directory as above, and like magic my build continues. Until I get to the next error – actually many errors, in the link stage for miniruby.exe there are a load of unresolved externals like _rb_id2name, and _rb_intern
, which are labels defined in parse.y/parse.c.

So, I look at parse.c in \ruby-1.9\build, and it’s zero bytes! It seems like even though the build failed with the missing sed.exe, it still made a zero length file, which make nmake think it’s all up to date, and a zero length file is a valid C program, so it compiles. I delete parse.c, and re-run nmake, and it correctly rebuilds parse.c and continues building, and it works!

C:\ruby-1.9\build>ruby -v
ruby 1.9.0 (2007-10-07 patchlevel 0) [i386-mswin32_80]


Okay, now I’ve got it built and running from the command line, how about building it from Visual Studio, like I did with 1.8.6. So I’m going to follow my own instructions, and see what is different, specifically in how miniruby.exe is built.

First of all, there are a couple of extra libraries, shell32.lib and ws2_32.lib, and one that has been removed: winsock32.lib. Seems like those two replace this one. Ws2_32 is windows sockets version 2 (replacing winsock). That’s all simple enough, but we’ll get to that later.

The main library is msvcr80-ruby19-static.lib, which is built with:

lib -nologo -machine:x86 -out:msvcr80-ruby19-static.lib array.obj ascii.obj bignum.obj class.obj compar.obj dir.obj dln.obj encoding.obj enum.obj enumerator.obj error.obj euc_jp.obj eval.obj eval_load.obj proc.obj file.obj gc.obj hash.obj inits.obj io.obj marshal.obj math.obj numeric.obj object.obj pack.obj parse.obj process.obj prec.obj random.obj range.obj re.obj regcomp.obj regenc.obj regerror.obj regexec.obj regparse.obj regsyntax.obj ruby.obj signal.obj sjis.obj sprintf.obj st.obj string.obj struct.obj time.obj unicode.obj utf8.obj util.obj variable.obj version.obj blockinlining.obj compile.obj debug.obj iseq.obj vm.obj vm_dump.obj thread.obj cont.obj id.obj prelude.obj acosh.obj crypt.obj erf.obj strlcpy.obj strlcat.obj win32.obj dmyext.obj

Well, that looks like a whole load of new stuff (marked in bold), some stuff for character encoding, a new regular expression library split over several modules, and a new virtual machine and associated compile and debug stuff. So where is all this stuff? Well, it looks like it’s all in the root again. Let’s see if we can knock up a project roughly following my old instructions:

Create a win32 console project, no precompiled headers, add oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib, add all the .c files in the root (ruby\) except dmydnl.c, add acosh.c, crypt.c, erf.c and (now) strlcat.c and strlcpy.c from ruby\missing.

Now where’s config.h? In 1.8.6 a file config.h was created in the build directory, but does not seem to be there now. Instead there’s one in \ruby-1.9\build\.ext\include\i386-mswin32_80\ruby\config.h, so I’ll just copy that over.

(NEW) Add the include path: c:\ruby-1.9\ruby\include

Now we still can’t find config.h, as it’s included as “ruby/config.h”, meaning it’s looking in c:\ruby-1.9\ruby\include\ruby\config.h, but we don’t want to put anything there, as that’s our raw source. So I just put it in \ruby-1.9\mickruby\mickruby\ruby\config.h, and added \ruby-1.9\mickruby\mickruby to the include paths, and it’s compiling away.

Okay, just four compile errors now, seems like it can’t find “insns.inc”, which is a file generated in \build. It looks like an enum of the YARV vm op-codes, and says it’s generated by insns2vm.rb from template/insns.inc.tmpl. It seems a bit odd that a generated file ends up in the build directory after a configure. What if you add a new VM instruction? Anyway, copy it (and insns_info.inc) into \ruby-1.9\mickruby\mickruby and then we are missing optinsn.inc, node_name.inc, vm.inc, vmtc.inc, so i just copy all 8 .inc files from build to mickruby\mickruby, and now we just have one error:

Error 255 error C2099: initializer is not a constant c:\ruby-1.9\ruby\regenc.c 32
OnigEncoding OnigEncDefaultCharEncoding = ONIG_ENCODING_INIT_DEFAULT;

Which reminds me, I was supposed to change the character set to “not set”, and add some preprocessor defns. It seems like there is unicode support now, so I’m not so sure about the character set thing. But I’ll give it a go …. nope.

So what’s with this error? ONIG_ENCODING_INIT_DEFAULT should be ONIG_ENCODING_ASCII, which should be (&OnigEncodingASCII), which is a structure defined in ascii.c. Now I’m tempted to blame Visual C here, but it compile fine from the command line, so instead I’m blaming myself, and something that I did. Looks like something to do with character encoding. But then my computer start acting weird, and visual studio tells me it’s out of memory. So I reboot, and the error GOES AWAY. Huh?

No matter, I’m now faced with new errors, lots of unresolved external symbols, I think I’m missing win32\win32.c, yup, but then that’s missing dln.h, which is in \ruby, so add that to the include path (just for this file), so now I’m just missing all the parse.c stuff, which is now in build, but since I’m not going to change the language right now, I’m just going to add it from there. Slightly dodgy, if I’m ever going to change parse.y. Anyway, seems like it includes parse.y in some odd way, which needs #include “regenc.h” which is in the root, so added c:\ruby-1.9\ruby to the include paths, and now we have a new error:

Error 8714 error LNK2019: unresolved external symbol _Init_prelude referenced in function _rb_call_inits inits.obj

Init_prelude comes from \build\prelude.c, which I’ll add directly, and FINALLY….


C:\ruby-1.9\mickruby\debug>mickruby -v
ruby 1.9.0 (2007-10-07 patchlevel 0) [i386-mswin32_80]

What a palaver! And what a mess I made of it, shoehorning it in Visual Studio. Still, it’s up and running, with the various hacks documents above, so that’s a start if I want to make a more professional version of it later.

(This is Ruby 1.8.6, for Ruby 1.9, see here)

I decided to see what it would take to compile Ruby as a Visual Studio project. Now you could just include it as a makefile project, but I figure that doing this would give me a better understanding of exactly what is involved in a build of Ruby. Essentially I would want something that would compile miniruby.exe, so I could then use that as a base for a stripped down version I can incorporate it into a full project.

Miniruby.exe is linked from

• main.obj (built from main.c)
• dmydln.obj (built from dmydln.c)
• msvcr80-ruby18-static.lib

And the following windows system libraries

• oldnames.lib
• user32.lib
• advapi32.lib
• wsock32.lib

So obviously the big one here is msvcr80-ruby18-static.lib, which is built from:

array.obj bignum.obj class.obj compar.obj dir.obj dln.obj enum.obj error.obj eval.obj file.obj gc.obj hash.obj inits.obj io.obj marshal.obj math.obj numeric.obj object.obj pack.obj parse.obj process.obj prec.obj random.obj range.obj re.obj regex.obj ruby.obj signal.obj sprintf.obj st.obj string.obj struct.obj time.obj util.obj variable.obj version.obj acosh.obj crypt.obj erf.obj win32.obj dmyext.obj

These are mostly the platform independent files that live in the root source folder, there are three files at the end (acosh.obj crypt.obj erf.obj) that live in the \missing folder. Then there is win32.obj and dmyext.obj. Win32.obj simply comes from compiling /win32/win32.c, and dmyext.obj is from dmyext.c, which again is in the root (not sure why it’s not in with the rest of them).

So, in theory I should simply be able to compile all the constituent files of msvcr80-ruby18-static.lib, and then link in main.obj, dmydln.obj and the four windows libs, and I’ll be set.

Using Ruby 1.8.6 source, in c:\ruby-src\ruby-1.8.6, I started out by:

Create a Win32 console project, c:\ruby-src\mickruby. Do not use precompiled header. Add oldnames.lib;user32.lib;advapi32.lib;wsock32.lib to the additional dependancies in the project

Drag into the project (or just add) all the .c files from c:\ruby-src\ruby-1.8.6, except for lex.c, and dmydln.c (dmydln.c, just sets a flag and includes dln.c, I’m unsure why the normal build seems to include both)

Drag in (or just add) the extra files acosh.c, crypt.c and erf.c from c:\ruby-src\ruby-1.8.6\missing

Drag in (or just add) win32.c from c:\ruby-src\ruby-1.8.6\win32

Add the include paths c:\ruby-src\ruby-1.8.6 and c:\ruby-src\ruby-1.8.6\missing

You now need an appropriate config.h, which I temporarily copied from a “configured” \build directory, into the root source folder. This kind of assumes you were building Ruby from the command line, which involves running the batch file c:\ruby-src\ruby-1.8.6\win32\configure.bat from the build folder. Or, more explictly: md c:\ruby-src\build then cd c:\ruby-src\build then c:\ruby-src\ruby-1.8.6\win32\configure.bat (you’ll need to type the full path). This will create the file config.h in c:\ruby-src\build, and you can just copy it into c:\ruby-src\mickruby\

Change configuration\general\character set to “not set”

Add the preprocessor definitions:

• RUBY_EXPORT
• _CRT_SECURE_NO_DEPRECATE
• _CRT_NONSTDC_NO_DEPRECATE

In the project entry point file (mickruby.cpp), just comment out all the source code for now. We want to use main.c as the project entry point for now. (You could also obvious copy over the code from main.c into mickruby.cpp, and exclude main.c from the project.
At this point, I can build the project, with 0 errors, and 4735 warnings, and IT WORKS, in release mode I get a 659K executable (mickruby.exe) that runs ruby code. The normal build gives me a 593K executable for miniruby.exe – probably a bunch of extra windows crap? Or maybe some options are different.

If I turn off link-time code generation, and twiddle a few options, the executable goes down to 602,112. Setting the optimizations as closely as possible to -O2b2xty-, gives us 598,016. Close enough.

Adding all this to my procedural tree sample takes the release build from 104K to 452K – that’s with nothing being called. If I add a call to ruby_init() in the initialization code, then it jumps up to 712K, which sounds exactly as would be expected from the above. That’s a chunk of code, half a meg, but in the grand scheme of things games typically need 512MB to run (XBox 360 and PS3 both have 512MB, PCs have that as a minimum). So while an extra 600K might have been crippling to a PS2 game, next-gen (which is really current-gen now) platforms are less impacted.