Cowboy Programming

Game Development and General Hacking by the Old West

December 26th, 2010

Words With Friends word list changes

Words With Friends is a popular “Scrabble” style word game on the iOS. To judge if a word is playable or not it uses a modified version of the Enable1 word list. The WWF list has some deletions and some additions:

This is the command I used to find the lists of changed words between the versions. It only works for word lists that are reasonably close, so would probably not work for comparing SOWPODS to TWL/OWL.

diff enable1lf.txt enable1-wwf3.12.txt | sort | grep "^>.*" | cut -c 3- | tr '\n' ',' | sed s/.$// | pbcopy

Delete words are mostly profanity and racial slurs, close your eyes now if you don’t want to see.

Read the rest of this entry »

January 27th, 2008

Blogroll and Categories not working in WordPress 2.3.2

I had a problem with the “Links” widget not showing up in the sidebar in WordPress, and categories not showing up in admin. I originally thought it was a problem with upgrading to 2.3.2, but eventually tracked it down to what seems like a problem with MySQL (I’m using 4.1.22).

The solution for me was to delete the file /tmp/#sql_5de0_0.MYD (or a file with a similar looking name, and probably zero size)

Note though that vanishing categories and links seems to be a symptom of several problems – including no deactivating (or upgrading) plugins while/after upgrading to 2.3.X. Check this threads for additional details:
http://wordpress.org/support/topic/135564?replies=95

I think my problem was something to do with character sets, given that I have a few blogs (some I just host for other people), and the blogs that worked were in utf8 and the blogs that did not were in latin1, so another solution might be to convert your blog to utf8. This is a much more complex procedure than deleting a spurious temporary file.

To track this down, I first enabled debugging in wp-config.php, by adding the line:

define(‘WP_DEBUG’, true);

Then when you view your blog (turn off caching first), instead of just blankness where the links should be, you will now see an error, and also the query that cause that error. In my case it looked like this:

SELECT t. * , tt. *
FROM wp_terms AS t
INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id
WHERE tt.taxonomy
IN (
'link_category'
)
AND tt.count >0
ORDER BY t.term_id ASC
LIMIT 0 , 30
MySQL said:
#1 - Can't create/write to file '/tmp/#sql_5de0_0.MYD' (Errcode: 17)

I verified that it’s not a problem with WordPress by trying the same query in phpMyAdmin. With my working blogs the query succeeded, and with the non-working blogs it failed, with the same error.

So, I looked in /tmp/ and the file #sql_5de0_0.MYD was there, and of zero length. So, I just deleted it, and everything immediately started working perfectly again.

I suspect that the file was simply left over from when my server crashed under a particularly heavy load a few weeks ago.

This may be related to this problem:

 http://bugs.mysql.com/bug.php?id=11824

This error can also happen on windows installations due to an anti-virus program such as McAfee. You should add the folder where you find the file to MacAfee’s exclusion list.

October 24th, 2007

Optional code in WordPress header

I’ve been using the WordPress plugin SyntaxHighlighter to show code in posts. This worked fine, but after I was Redditted yesterday I was looking at optimizing my pages and I noticed that SH was serving up at least three files, even if it was not used on a page (which it was not, on the page that had been linked from Reddit).

So I did a little digging, and made my first plugin mod.
http://cowboyprogramming.com/2007/10/24/optional-code-in-wordpress-header/
View this Post
SyntaxHighlighter works by hooking in three actions:

add_action( 'wp_head', 'syntaxHeader' );
add_action( 'the_content', 'syntaxPost' );
add_action( 'wp_footer', 'syntaxFooter' );

The middle one does most of the work, handing a post to a syntax highlighter with the appropriate options. It’s called once per post on the page.

The other two functions respectively insert a CSS link into the page header, and a JavaScript link for each supported language into the page footer. The problem was they did this regardless of if they were actually going to be used, so every single page had these additional links, slowing down my finely tuned blog.

To fix this, I simply added this function:

// Return true if any post on this page contains the text '[source:'
function hasSyntaxPost()
{
$is_syntaxPost = false;
global $posts;
for($i=0;$i<=(count($posts)-1);$i++)
{
if( strpos( $posts[$i]->post_content, '[source:' ) !== false )
{
$is_syntaxPost = true;
}
}
return $is_syntaxPost;
}

and used it to wrap the header and footer code, like:

// Add CSS to header
function syntaxHeader() {
if (hasSyntaxPost())
{
$syntaxPath = syntaxPath();
echo "\n\t<link type=\"text/css\" rel=\"stylesheet\" href=\"{$syntaxPath}Styles/SyntaxHighlighter.css\" />\n";
}
}

That’s it. Since I know almost nothing about PHP or WordPress, I coded this with cut-and-paste coding, stealing a few lines of code from:

http://trevorcreech.com/geekery/wordpress/per-post-css/

Result: works perfectly – only pages that contain posts that need highlighting actually have the extra code added.

I then modified it to only include the languages that are used, so the user does not have to worry about setting them in the options page, and it gives an extra performance boost, as most posts will only include one language (like here I just have PHP, and on other pages I have C++).

You can see all the changed in the modified syntax.php here:

Note: When testing things like this, disable the cache until you get it working.

October 23rd, 2007

Reddit-proofing WordPress

[UPDATE] Since writing this, I had another huge spike in traffic (20,000 visitors in a day), which again caused a crash. As a result, I installed WP-Super Cache, which is much more robust that WP-Cache. I also adjusted the theme to use a static stylesheet (style.css) rather than a dynamic one (style.php) -. This totally eliminates all load on the PHP and MySQL backend, vastly improving the situation. The use of style.php vs. style.css varies by theme, it’s an option in the tiga theme used here, but for many themes is unnecessary.

…..

Some time this morning my blog started acting a bit wonky, giving me errors about not being able to connect to the database, and then finally becoming inaccessible. I could not even get into my server control panel, and had to request a power cycle.

After it came back up, I found out why it was having problems. I’d been linked from reddit.com, a user-generated news site, and I was getting a new connection every few seconds, already several thousand connection had been made, and the sheer number of requests was crashing the server. All of the request were for this page:

http://cowboyprogramming.com/2007/01/04/mature-optimization-2/

But why was this making the server crash? Well, the most immediate problem was that wp-cache was not enabled. This meant that every page view had to be re-built from the MySQL database using PHP. If the requests are coming in faster than the page can be built, then they’ll get backed up. Eventually thing crunch to a stop, seeing as the basic workings of WordPress also require database connections. And since cPanel also uses a database, it too fails to work. I could probably improve this somewhat by increasing the number of connections, but really they should be cached.

So I re-enabled the cache, and things seemed to go swimmingly. I then took a look at what was being downloaded as part of the page. I was rather surprised to discover that the SyntaxHighlighter plugin was serving up a bunch of JavaScript, even though it was not used on this particular page. That seems like a bit of a bug. I was able to trim this down quite a bit by disabling every language except for C++.  later I hacked it to only include the JavaScript if needed.

I then noticed that my header image and my background image were about 100K each, which is rather silly. I took them both into photoshop, and tweaked them down to about 15K. The background image was a GIF, but this was still compressable by a significant amount by just setting “lossy” to a small number like 4 or 5, and there was no perceptible degradation.

Finally, my Reddit effect was seemingly having a knock-on effect in South Korea, where I had linked to a 2.5MB pdf file that was hosted at Seoul National University. This was currently inaccessible. I’m not sure, but suspect a few people were trying to download it (I was at 5,000 visitors by then). I wanted to re-host it locally, but I could not even read it myself. So I just deleted the link, then 20 minutes later things had died down, so I was able to grab the file. I re-hosted it on my server, and put up a new link.

So my server has been up about six hours now, and since then it’s been continued to be hit fairly constantly from Reddit. In that time there has been 70,000 accesses (3.17 per second) , accounting for 724MB. It’s about 20 times the normal daily traffic for this site.

Here’s a graphical look at what happened:

As you can see, bandwidth is not really the problem. It peaks at 1Mb/sec, and my server can serve at 100Mb/sec. You could probably handle this load on a 768K DSL line.

The Server degrades severely at around 8:00AM, then around 8:20 is when I reboot it. I re-enable the cache a few minutes after this, but this does not have any effect on the bandwidth, seeing as the pages being served are identical, just less load on the PHP/MySQL back end.

Then at around 10:00AM I start optimizing, shrinking the background and header images, and trimming the syntax highlighter, which more than halves the bandwidth.

|