Cowboy Programming

Game Development and General Hacking by the Old West

October 29th, 2007

Hotfixing Vista

I got a Blue Screen of Death a few minutes ago, and I’d had another one yesterday. The BSOD is possibly the most annoying thing that can posibly happen, short of total drive failure. Anyway, this time I wrote down the error:

DRIVER_POWER_STATE_FAILURE

With a little digging it seemed this was related to the fact that two days ago I accidentally put my computer to sleep, instead of powering down. This is a known issue, detailed here:

http://support.microsoft.com/kb/941858

Since the fix is apparently a bit tenuous, you have to contact Microsoft to get the hotfix. At first I attempted to do this via their on-line chat, but swiftly gave that up as it managed to crash IE7, and kept telling me it was about to charge me $54.

So I eventually discovered you can just submit an online request for a hotfix, here:

http://support.microsoft.com/kb/935195

So I did, and now I’m waiting. We shall see….

[UPDATE] About three hours later I got an email with a download link and a password. The password is only good for seven days. I installed it, and I’m keeping my fingers crossed. No crashes so far.

October 29th, 2007

MySQL Server Install Problems with Vista

Oh Vista, how do I dislike thee!

Grr – trying to install MySQl server 5.0 on Windows Vista was rather frustrating, as it simply stopped with “Unable to start service, error 1067″. Especially annoying as I flawlessly installed the thing on my XP laptop in two minutes last night.

You would think someone would have fixed this now, but no, the googlesphere advice seemed to be to re-run the install program as administrator. This failed to work. But I gathered from faint intimations that you had to remove the service for it to be properly re-installed. So, the steps that seemed to work for me were:

  1. Run regedit
  2. Delete the key MySQL from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
  3. Reboot (important, otherwise the service does not go away)
  4. Go to C:\Program Files\MySQL\MySQL Server 5.0\bin
  5. Right click on MySQLInstanceConfig.exe, and choose “Run as administrator”
  6. Set it up as you want, and cross your fingers when it comes to starting the service.

Worked for me!

October 24th, 2007

Optional code in WordPress header

I’ve been using the WordPress plugin SyntaxHighlighter to show code in posts. This worked fine, but after I was Redditted yesterday I was looking at optimizing my pages and I noticed that SH was serving up at least three files, even if it was not used on a page (which it was not, on the page that had been linked from Reddit).

So I did a little digging, and made my first plugin mod.
http://cowboyprogramming.com/2007/10/24/optional-code-in-wordpress-header/
View this Post
SyntaxHighlighter works by hooking in three actions:

add_action( 'wp_head', 'syntaxHeader' );
add_action( 'the_content', 'syntaxPost' );
add_action( 'wp_footer', 'syntaxFooter' );

The middle one does most of the work, handing a post to a syntax highlighter with the appropriate options. It’s called once per post on the page.

The other two functions respectively insert a CSS link into the page header, and a JavaScript link for each supported language into the page footer. The problem was they did this regardless of if they were actually going to be used, so every single page had these additional links, slowing down my finely tuned blog.

To fix this, I simply added this function:

// Return true if any post on this page contains the text '[source:'
function hasSyntaxPost()
{
$is_syntaxPost = false;
global $posts;
for($i=0;$i<=(count($posts)-1);$i++)
{
if( strpos( $posts[$i]->post_content, '[source:' ) !== false )
{
$is_syntaxPost = true;
}
}
return $is_syntaxPost;
}

and used it to wrap the header and footer code, like:

// Add CSS to header
function syntaxHeader() {
if (hasSyntaxPost())
{
$syntaxPath = syntaxPath();
echo "\n\t<link type=\"text/css\" rel=\"stylesheet\" href=\"{$syntaxPath}Styles/SyntaxHighlighter.css\" />\n";
}
}

That’s it. Since I know almost nothing about PHP or WordPress, I coded this with cut-and-paste coding, stealing a few lines of code from:

http://trevorcreech.com/geekery/wordpress/per-post-css/

Result: works perfectly – only pages that contain posts that need highlighting actually have the extra code added.

I then modified it to only include the languages that are used, so the user does not have to worry about setting them in the options page, and it gives an extra performance boost, as most posts will only include one language (like here I just have PHP, and on other pages I have C++).

You can see all the changed in the modified syntax.php here:

Note: When testing things like this, disable the cache until you get it working.

October 23rd, 2007

Reddit-proofing WordPress

[UPDATE] Since writing this, I had another huge spike in traffic (20,000 visitors in a day), which again caused a crash. As a result, I installed WP-Super Cache, which is much more robust that WP-Cache. I also adjusted the theme to use a static stylesheet (style.css) rather than a dynamic one (style.php) -. This totally eliminates all load on the PHP and MySQL backend, vastly improving the situation. The use of style.php vs. style.css varies by theme, it’s an option in the tiga theme used here, but for many themes is unnecessary.

…..

Some time this morning my blog started acting a bit wonky, giving me errors about not being able to connect to the database, and then finally becoming inaccessible. I could not even get into my server control panel, and had to request a power cycle.

After it came back up, I found out why it was having problems. I’d been linked from reddit.com, a user-generated news site, and I was getting a new connection every few seconds, already several thousand connection had been made, and the sheer number of requests was crashing the server. All of the request were for this page:

http://cowboyprogramming.com/2007/01/04/mature-optimization-2/

But why was this making the server crash? Well, the most immediate problem was that wp-cache was not enabled. This meant that every page view had to be re-built from the MySQL database using PHP. If the requests are coming in faster than the page can be built, then they’ll get backed up. Eventually thing crunch to a stop, seeing as the basic workings of WordPress also require database connections. And since cPanel also uses a database, it too fails to work. I could probably improve this somewhat by increasing the number of connections, but really they should be cached.

So I re-enabled the cache, and things seemed to go swimmingly. I then took a look at what was being downloaded as part of the page. I was rather surprised to discover that the SyntaxHighlighter plugin was serving up a bunch of JavaScript, even though it was not used on this particular page. That seems like a bit of a bug. I was able to trim this down quite a bit by disabling every language except for C++.  later I hacked it to only include the JavaScript if needed.

I then noticed that my header image and my background image were about 100K each, which is rather silly. I took them both into photoshop, and tweaked them down to about 15K. The background image was a GIF, but this was still compressable by a significant amount by just setting “lossy” to a small number like 4 or 5, and there was no perceptible degradation.

Finally, my Reddit effect was seemingly having a knock-on effect in South Korea, where I had linked to a 2.5MB pdf file that was hosted at Seoul National University. This was currently inaccessible. I’m not sure, but suspect a few people were trying to download it (I was at 5,000 visitors by then). I wanted to re-host it locally, but I could not even read it myself. So I just deleted the link, then 20 minutes later things had died down, so I was able to grab the file. I re-hosted it on my server, and put up a new link.

So my server has been up about six hours now, and since then it’s been continued to be hit fairly constantly from Reddit. In that time there has been 70,000 accesses (3.17 per second) , accounting for 724MB. It’s about 20 times the normal daily traffic for this site.

Here’s a graphical look at what happened:

As you can see, bandwidth is not really the problem. It peaks at 1Mb/sec, and my server can serve at 100Mb/sec. You could probably handle this load on a 768K DSL line.

The Server degrades severely at around 8:00AM, then around 8:20 is when I reboot it. I re-enable the cache a few minutes after this, but this does not have any effect on the bandwidth, seeing as the pages being served are identical, just less load on the PHP/MySQL back end.

Then at around 10:00AM I start optimizing, shrinking the background and header images, and trimming the syntax highlighter, which more than halves the bandwidth.

October 7th, 2007

Get and Build Ruby 1.9 for Windows

Today’s “programming” exercise: get the source for Ruby 1.9, and build a working version under Visual Studio, just like I did for version 1.8.

Step 1: Search Google, which gives me this starting point:

http://www.megasolutions.net/ruby/Help-building-Ruby-1_9-on-Windows-50037.aspx

> I am looking for some advice on building Ruby 1.9 on windows.

Here’s what I did on XP SP2 running under Parallels.

- Download and install MSFT Visual C++ 2005 Express Edition (gratis)
- Get the Windows Platform SDK
- Install Ruby 1.8.x using 1-click Windows Installer
- Grab Ruby 1.9 from Subversion

- Click through: Start Menu / Platform SDK / Open Build Environment
Window / Windows XP 32-bit … / Set Windows XP … Build Environment
- In the command window that opens, run “c:\Program Files\Microsoft
Visual Studio 8\Common7\Tools\vsvars32.bat”
- win32/configure.bat
- nmake

Result:

C:\dev\ruby\win32>ruby -v
ruby 1.9.0 (2007-04-12 patchlevel 0) [i386-mswin32_80]

Now I’ve already done the first three steps, so my first stumbling point is:

Grab Ruby 1.9 from Subversion

I don’t have Subversion installed, and I’ve never used it before. I’ve been using Perforce for years, and I use the free version of that for my home projects. So, I go to: http://subversion.tigris.org/project_packages.html, which is rather intimidating for the poor Windows user. All I want is the client, but there are a range of download options, none of which look trivial. One mentions a “friendly installer program”, so I click on that, and phew, a nicer list of install packages, I choose Subversion 1.4.5, download and run, accepting all defaults.

So, I feel like I should be able to go to the command line and type something in to get the Ruby 1.9 source, and with a few seconds of browsing, I find the correct instructions at http://www.ruby-lang.org/en/community/ruby-core/ :

svn co http://svn.ruby-lang.org/repos/ruby/trunk ruby

Imagine my surprise when this actually worked! Download seems a little slow, but it’s chugging away, downloading rather a lot of stuff (most of which I’m sure I do not need). It took about five minutes, and finally informed me it had checked out version 13655.

I’d run the above command line inside a directory \ruby-1.9, which then gives me the source tree in \ruby-1.9\ruby\, so I open a visual studio command prompt and do:
cd \ruby-1.9
md \build
cd build
\ruby-1.9\ruby\win32\configure

which gives me:
Creating Makefile
confargs.c
type `"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\nmake.exe"' to make ruby for mswin32.

So I type “nmake”, and sadly after a nice start (several files are compiled) I get a useful error:
bison -o y.tab.c /ruby-1.9/ruby/parse.y
'bison' is not recognized as an internal or external command,
operable program or batch file.
NMAKE : fatal error U1077: 'bison' : return code '0x1'
Stop.

Hmm, I don’t remember having to have bison installed for 1.8. Bison is the GNU parser generator, which takes a .y file (the grammar definition) and makes a .c file (the parser). It seems like a simple case of “missing tools” from the “why won’t it build” list. But I’m a little confusled that 1.8 actually has a file parse.c. If I try nmake parse.c in the 1.8 tree I get “‘byacc’ is not recognized ….”, which shows they have switched tools from byacc (Berkeley Yacc) to Bison (both the same kind of thing). But that’s not the problem, as I don’t have byacc either. Nope, the problem is that parse.c is not part of the subversion repository, which is very reasonable for a development source tree, as it’s dependent on parse.y. Unfortunately I now have to install Bison. I’m beginning to fear I might have to install Cygwin, eek!

So, I download the Bison 2.1 Windows install package and run it -which seems to do everything except set the path, so I add C:\Program Files\GnuWin32\bin to my PATH variable, restart my Visual Studio 2005 command prompt, an re-run nmake, and shucks if there is not another error:

bison -o y.tab.c /ruby-1.9/ruby/parse.y
sed -e "s!^ *extern char \*getenv();!/* & */!;s/^\(#.*\)y\.tab/\1parse/"
y.tab.c > parse.c
'sed' is not recognized as an internal or external command, operable program or batch file.
NMAKE : fatal error U1077: 'sed' : return code '0x1'

Great, another GNU tool that is missing. Luckily it seems like all these tools are available under GnuWin32, so I can simple download the Sed installer, and it goes in the same bin directory as above, and like magic my build continues. Until I get to the next error – actually many errors, in the link stage for miniruby.exe there are a load of unresolved externals like _rb_id2name, and _rb_intern
, which are labels defined in parse.y/parse.c.

So, I look at parse.c in \ruby-1.9\build, and it’s zero bytes! It seems like even though the build failed with the missing sed.exe, it still made a zero length file, which make nmake think it’s all up to date, and a zero length file is a valid C program, so it compiles. I delete parse.c, and re-run nmake, and it correctly rebuilds parse.c and continues building, and it works!

C:\ruby-1.9\build>ruby -v
ruby 1.9.0 (2007-10-07 patchlevel 0) [i386-mswin32_80]

Okay, now I’ve got it built and running from the command line, how about building it from Visual Studio, like I did with 1.8.6. So I’m going to follow my own instructions, and see what is different, specifically in how miniruby.exe is built.

First of all, there are a couple of extra libraries, shell32.lib and ws2_32.lib, and one that has been removed: winsock32.lib. Seems like those two replace this one. Ws2_32 is windows sockets version 2 (replacing winsock). That’s all simple enough, but we’ll get to that later.

The main library is msvcr80-ruby19-static.lib, which is built with:

lib -nologo -machine:x86 -out:msvcr80-ruby19-static.lib array.obj ascii.obj bignum.obj class.obj compar.obj dir.obj dln.obj encoding.obj enum.obj enumerator.obj error.obj euc_jp.obj eval.obj eval_load.obj proc.obj file.obj gc.obj hash.obj inits.obj io.obj marshal.obj math.obj numeric.obj object.obj pack.obj parse.obj process.obj prec.obj random.obj range.obj re.obj regcomp.obj regenc.obj regerror.obj regexec.obj regparse.obj regsyntax.obj ruby.obj signal.obj sjis.obj sprintf.obj st.obj string.obj struct.obj time.obj unicode.obj utf8.obj util.obj variable.obj version.obj blockinlining.obj compile.obj debug.obj iseq.obj vm.obj vm_dump.obj thread.obj cont.obj id.obj prelude.obj acosh.obj crypt.obj erf.obj strlcpy.obj strlcat.obj win32.obj dmyext.obj

Well, that looks like a whole load of new stuff (marked in bold), some stuff for character encoding, a new regular expression library split over several modules, and a new virtual machine and associated compile and debug stuff. So where is all this stuff? Well, it looks like it’s all in the root again. Let’s see if we can knock up a project roughly following my old instructions:

Create a win32 console project, no precompiled headers, add oldnames.lib user32.lib advapi32.lib shell32.lib ws2_32.lib, add all the .c files in the root (ruby\) except dmydnl.c, add acosh.c, crypt.c, erf.c and (now) strlcat.c and strlcpy.c from ruby\missing.

Now where’s config.h? In 1.8.6 a file config.h was created in the build directory, but does not seem to be there now. Instead there’s one in \ruby-1.9\build\.ext\include\i386-mswin32_80\ruby\config.h, so I’ll just copy that over.

(NEW) Add the include path: c:\ruby-1.9\ruby\include

Now we still can’t find config.h, as it’s included as “ruby/config.h”, meaning it’s looking in c:\ruby-1.9\ruby\include\ruby\config.h, but we don’t want to put anything there, as that’s our raw source. So I just put it in \ruby-1.9\mickruby\mickruby\ruby\config.h, and added \ruby-1.9\mickruby\mickruby to the include paths, and it’s compiling away.

Okay, just four compile errors now, seems like it can’t find “insns.inc”, which is a file generated in \build. It looks like an enum of the YARV vm op-codes, and says it’s generated by insns2vm.rb from template/insns.inc.tmpl. It seems a bit odd that a generated file ends up in the build directory after a configure. What if you add a new VM instruction? Anyway, copy it (and insns_info.inc) into \ruby-1.9\mickruby\mickruby and then we are missing optinsn.inc, node_name.inc, vm.inc, vmtc.inc, so i just copy all 8 .inc files from build to mickruby\mickruby, and now we just have one error:

Error 255 error C2099: initializer is not a constant c:\ruby-1.9\ruby\regenc.c 32
OnigEncoding OnigEncDefaultCharEncoding = ONIG_ENCODING_INIT_DEFAULT;

Which reminds me, I was supposed to change the character set to “not set”, and add some preprocessor defns. It seems like there is unicode support now, so I’m not so sure about the character set thing. But I’ll give it a go …. nope.

So what’s with this error? ONIG_ENCODING_INIT_DEFAULT should be ONIG_ENCODING_ASCII, which should be (&OnigEncodingASCII), which is a structure defined in ascii.c. Now I’m tempted to blame Visual C here, but it compile fine from the command line, so instead I’m blaming myself, and something that I did. Looks like something to do with character encoding. But then my computer start acting weird, and visual studio tells me it’s out of memory. So I reboot, and the error GOES AWAY. Huh?

No matter, I’m now faced with new errors, lots of unresolved external symbols, I think I’m missing win32\win32.c, yup, but then that’s missing dln.h, which is in \ruby, so add that to the include path (just for this file), so now I’m just missing all the parse.c stuff, which is now in build, but since I’m not going to change the language right now, I’m just going to add it from there. Slightly dodgy, if I’m ever going to change parse.y. Anyway, seems like it includes parse.y in some odd way, which needs #include “regenc.h” which is in the root, so added c:\ruby-1.9\ruby to the include paths, and now we have a new error:

Error 8714 error LNK2019: unresolved external symbol _Init_prelude referenced in function _rb_call_inits inits.obj

Init_prelude comes from \build\prelude.c, which I’ll add directly, and FINALLY….


C:\ruby-1.9\mickruby\debug>mickruby -v
ruby 1.9.0 (2007-10-07 patchlevel 0) [i386-mswin32_80]

What a palaver! And what a mess I made of it, shoehorning it in Visual Studio. Still, it’s up and running, with the various hacks documents above, so that’s a start if I want to make a more professional version of it later.

|