making bbPress (and WordPress) work better!

bbPress

An Easy Way to Speed Up MySQL FullText Natural Search on Large Tables

As your WordPress or bbPress install grows over the years, if you have a large site you will notice the search slows down quite a bit on large tables. Another problem is the quality of search results becomes poor as it has to muck through more and more posts.

It’s difficult to jump to a solution like sphinx right away, and things like the google api are not aware enough about the context of your content, so some people will code their own searches within MySQL in the meanwhile.

For example, a minimal search is done something like the following, taking advantage of MySQL’s natural search mode which takes into account the number of times the words appears and their proximity to return a score.

SELECT post_id,  MATCH (post_text) AGAINST ('some words') AS score
FROM posts WHERE MATCH (post_text) AGAINST ('some words')
ORDER BY score DESC

In my situation, using a much more complex version of that query that takes into account all posts in a topic was taking up to 12 seconds which was completely unacceptable.

So it occurred to me that MySQL also has a binary search mode that is far faster. It’s not what I wanted for the overall results because it’s relevancy can be poor, but it instead could just be used to eliminate posts that would have to be scanned. Anytime you can eliminate rows in mysql you will get a speed boost.

So simply do this instead:

SELECT post_id,  MATCH (post_text) AGAINST ('some words') AS score
FROM posts WHERE MATCH (post_text) AGAINST ('some words' IN BOOLEAN MODE)
ORDER BY score DESC

Basically that IN BOOLEAN MODE will eliminate rows very quickly that do not have either of those words in them, but the SELECT still calculates a valuable natural search score for the remaining results.

When using boolean mode you have to take some extra steps to sanitize the query because some characters have special meaning so keep that in mind but it’s trivial to do.

Eventually I took it a step further and preprocessed the search terms even further. (more…)


bbPress still deleted from Wikipedia

5/26 update: a noble group of folks have taken up the effort to restore and rebuild the page on wikipedia – I’ll leave this post for posterity…

bbPress is approaching 10,000 active installs and millions of users by my (heavily researched) estimation and yet it’s still deleted from Wikipedia. However BuddyPress which definitely has less than a thousand active installs, features it’s own page. Go figure.

Wikipedia has pages on ultra obscure fonts that aren’t even in use anymore from ancient Windows 3.1 applications, but a dedicated bbPress page is excessive to the editors?


First ever TalkPress powered website?

Unless I missed an announcement somewhere,
I think I’ve found the first ever TalkPress powered website:

http://ask-the-expert.health.com/

Looks really nice. I believe it quietly went live in the middle of March.

TalkPress is essentially Automattic hosted bbPress forums,
just like what WordPress.com is to WordPress blogs.
(it uses WordPress.com global logins/cookies)

Matt first mentioned TalkPress a year ago.

update: Yup, I did indeed miss an announcement, here it is:
http://watershedstudio.com/blog/2009/03/healthcom-on-talkpress-vip/


Some Tech Tips to find a better WordPress or bbPress Host

I’m going to reprint some advice here that I shared on bbPress.org as there is a great deal of confusion and misguided ideas over how to find a good host for WordPress or bbPress.

A quick google shows there are dozens of “Top 10” lists of hosts. Some are dubious with commission links and others that as well intentioned as they are, focus on non-technical aspects that don’t matter much in the end.

There is a MASSIVE amount of competition in the hosting world and virtually everyone offers things like “uptime guarantees” and seductive low prices to try to grab customers. In the end it doesn’t matter. You can go with a provider that’s been highly recommended and have the worst experience, or try one out that others have complained about and have a good experience.

(more…)


Firefox plugin to detect when website sessions in use?

update: presenting my very first firefox plugin!
http://bbshowcase.org/firefox/website-sessions-detector.xpi
Detects if any of several session types are in use on a website. It has some room for improvement but the fact that it works at all is good start…

—–
I dislike the use of sessions to track users between web pages because it slows things down on active websites and if cookies are disabled, they can create a mess when they get appended to the URL. They are also useless across multiple servers if they aren’t stored in a common memory pool (memcached, etc).

Here are some common session names:
ASP uses “ASPSESSIONID”
PHP uses “PHPSESSID”
.NET uses “ASP.NET_SessionId”
JSP uses “JSESSIONID”
ColdFusion uses “CFID”
(let me know if you are aware of others)

I try to avoid using sessions in my bbPress plugins but there are two (Human Test & OpenID) where I’ve been too tired (aka too lazy) to come up with a complex way around them that would involve mysql tables, etc. However I try to make sure that sessions are not activated when not needed (in most cases, only during registration. But there is a popular third-party plugin that I’ve taken over ownership for (bb-topic-views) that uses sessions all the time when reading topics to prevent re-counting on multiple pages of the same topic.

(more…)


simple WordPress and bbPress security plugin to block long requests

I opened a WordPress security ticket today with something that’s been bothering me for awhile – Apache will take long URL queries up to 8k (8192) characters in length but it’s completely unnecessary and allows XSS exploits to get into WordPress and bbPress. Why not block them entirely with this scrap of a plugin (save as “_block_long_queries.php” without the quotes but with the leading underscore so it loads as early as possible and doesn’t need activation – also can’t be easily deactivated by hackers)
(more…)


bbPress turns 2 today! Happy Birthday bbPress!

bbPress turns 2 years old today!!!

It’s been a big year with new features being added at a rapid pace thanks to hard work by Sam and Mike. It’s an easy guess that we should see a stable 1.0 beta by the end of 2008.

But don’t let the modest sounding “1.0” fool you, it’s practically a “2.0” inside with all the changes since 0.9 including BackPress integration (merges core functions from WordPress) and an XML-RPC api like WordPress as well.
(more…)


BBXF cross-forum import/export program

BBXF is a fascinating idea. What if you could easily import or export your forums between any programs like vBulletin/phpBB/SMF/bbPress? For blogs is this fairly straightforward and already exists. But forums are much more complex than blogs. bbPress doesn’t have any of the typically native forum features like polls,user tracking, local avatars, signatures or attachments – it’s all done via plugins. So the question is, what happens to all that data between programs? Importing into bbPress you could just shove it all into meta and hope to manipulate it into place later (good luck with that). Exporting you’d essentially lose all that data.
(more…)


Wikipedia Overlords delete bbPress page

5/26 update: a noble group of folks have taken up the effort to restore and rebuild the page on wikipedia – I’ll leave this post for posterity…

I’m not quite sure when this happened (update: oh now I see, just last week) but I know it was there a few months ago as I wanted to see again what it said:
http://en.wikipedia.org/wiki/BbPress
It had some nice history and progress about bbPress. Virtually every other forum software out there is on Wikipedia so this makes me very curious what kind of politics are going on or if someone had something against Automattic.

Apparently bbPress fails their elite “Notability Guidelines”
http://en.wikipedia.org/w/index.php?title=Special:Log/delete&page=BbPress
So at least 5000 installs with millions of people using those installs doesn’t count eh? I guess the 2k of text was taking up too much precious space on their servers, sheesh.

From what I can tell, it was a speedy deletion, meaning it was not even reviewed or given notice for deletion – an administrator just arbitrarily did what they felt like doing.

At least Deletionpedia has saved it and here’s the page cache from MSN Yahoo and Wayback Machine
(google doesn’t cache Wikipedia deletions apparently)


bbPress presentation at WordCamp UK 2008?

For my UK mates, WordCamp UK 2008 is coming up next Saturday and Sunday 19-20 July 2008 at The Studio in Birmingham.

(42 out of 100 tickets have been sold as of yesterday, so it’s bound to sell out)

I’d love to hear any recordings (video or just audio) of the more technical sessions and of course Sam’s presentation on bbPress

ps. check out Sam’s impressive WP crests!


bbPress 0.9 final, officially released!

bbPress 0.9 forum software has just been made ready for the public and I highly recommend it.

Many subtle bug fixes and many new features including easier WordPress integration:

http://bbpress.org/blog/2008/04/bbpress-09-brubeck/


http://bbpress.org/download/

If you are planning to integrate with WordPress, you’ll need the new WordPress 2.5 to match the new secure cookie method (or the bbPress plugin to force the old cookie method).


new bbPress plugin: Unread Posts

Indicates previously read topics with new unread posts across most types of pages within bbPress. Features “mark all topics read”.

download: http://bbpress.org/plugins/topic/78

This is not just “yet another” Show Unread Posts plugin.
Builds on concepts by fel64 and henrybb with feature and performance improvements.
No additional plugins or tables required. Produces no overhead for non-members and as little as possible otherwise.

Users can be given a link to “mark all topics read” and you can adjust the number of topics to track to limit bloat. Topics with new posts can be customized via css styles, ie. bold, underline, colors, or even an icon.

To see a demonstration, create an account at http://bbShowcase.org
read some posts and wait for some replies.
(more…)


new bbPress plugin: Human Test for Registration

I’ve written a plugin to slow down bot registrations
which was becoming a problem on a couple of my bbPress websites.

http://bbpress.org/plugins/topic/77

demo: http://bbshowcase.org/forums/register.php

Simply adds a new field at the bottom of your registration page with two random numbers to add together. Uses a few tricks like javascript and entity encoding to slow down the smarter bots.

(no core modifications required, 100% plugin based)

Image based writing coming soon, I have to research which captcha methods haven’t been broken yet.


Happy Birthday bbPress!

It’s still before midnight somewhere in the world, so I can still wish bbPress a happy 1st birthday.

bbPress is the forum software spinoff from the same person (Matt) that started WordPress (though others have completely taken over at this point).

While Matt had been working on it since Christmas 2004, version 0.7.2 (nickname “bix”) was finally officially released for public consumption on Saturday October 14th 2006 making it a year old today.

It’s come a long way since then. If you’d like to see what kind of tricks bbPress can do now, take a peek at my bbPress Showcase, which now has a permanent home at bbShowcase.org

most of the known bbPress themes:
http://bbshowcase.org/forums/view/available-themes

most of the known bbPress plugins:
http://bbshowcase.org/forums/view/available-plugins

top 100 forums using bbPress (informal, based on activity)
http://bbshowcase.org/forums/view/top100

1000 known forums using bbPress (must be registered to see the list)
http://bbshowcase.org/forums/view/top1000


Is it bad when you start daydreaming about code?

I had this daydream the other day where I reversed the way WordPress/bbPress works and instead of running forwards by loading the entire engine first, then ALL the plugins and then finally the template, it worked BACKWARDS – where it first loaded the template and analyzed what was being requested, then loaded only the parts of the core it needed and then the plugins that were flagged as affecting the required output.

Way more complex, sure. But way less code execution and less plugin loading – and far, far, far more easy to cache parts of the pages while allowing other parts to be dynamic, without having to load nearly 500k of code among nearly 100 files.


The WordPress/bbPress Performance Limit

I was looking at some of the “most requested features” for WordPress and I noticed that “Make It Faster” was pretty high on the list.

I’ve learned a lot about WordPress code over the past few years but even more so after working with it’s forum spinoff bbPress. A few things about WordPress/bbPress performance became obvious when I starting noticing that another plugin developer’s forum running SMF was running considerably faster than bbPress and WordPress on the same poorly performing host. I took SMF apart to understand why and here’s what I found:

WordPress cannot (in theory) be made much faster, unless entire pages are made static with a caching system and it’s core is not even loaded for the visitor. It’s simply a limit of WordPress’s architecture and would require an entire rewrite to change. If you have mysql caching turned on and an opcode cache (apc, eaccelerator, xcache, etc) that’s the most you can (easily) do before solutions like wp-cache.

The reasons why include:

1. dozens upon dozens of files that have to be included before any page rendering can begin. Other blog/cms software tries to keep this number down but in WordPress it’s almost runaway. Even an opcode cache must check the timestamp on the source files so the filesystem is kept constantly busy – if it’s not in the filecache, there is a performance hit. If it’s being rendered on a different server from where the files are stored (like on Dreamhost and others) there is even more of a performance hit.

2. Each plugin is a separate file that is not a direct mod on the source but an additional include that is required to load and more code to execute. All plugins are loaded regardless of what kind of page is being rendered. See above why this is a problem. Your WordPress setup can easily be requiring near 100 files to load and 500k of code to execute before a single character is output.

3. All output goes through a language translation hook. Even if you are using the default language this will always be slower than direct output.

4. Output is done in bits and pieces with typically no buffering done (ala ob_start) to accelerate and smooth this process. Any programmer should know from experience that outputting a few characters at a time, 1000 times is far slower than storing them all up in a string and dumping the few thousand all together at once at the end.

All these weaknesses can also be found in bbPress (the forum software that is based on WordPress architecture). Maybe oneday a great mind will come up with a magic, backwards compatible speedup solution without rewriting the core but I doubt it will be any time soon.

Despite these weaknesses, on a single dedicated or VPS server with proper resources, proper opcode, mysql and file caching, WordPress (and bbPress) can give just about any other blog/cms software a run for it’s money. The problems begin when these performance solutions are simply not available on a shared hosting server.

ps. I’ll also note that I am not abandoning WordPress or bbPress anytime soon. The benefits of their plugin architecture far outweigh the server performance requirements. Another method for speedup that I did not mention is that replacing apache with lighttpd or litespeed which can also be a big benefit but of course that is much more work than simply turning on the mysql cache and installing an opcode cache.


bbPress plugin: Ignore Member

One of my forums is not even a few months old and I already have two members that just can’t get along. But they both are decent contributors to the site so I don’t want to ban them.

So I decided it was time for bbPress to have a proper Ignore filter like the better forums have, that way they don’t even see each others posts. Here it is!

download from the bbPress plugin browser (recommended)
or https://ckon.files.wordpress.com/2007/08/ignore-member.txt
(rename .txt to .php)

It’s beta but tested working in both 0.8.2.1 and the alpha trunk.
Members will get an “Ignore” link on other’s posts (once you edit post.php) and any ignored members will show up in their edit profile so they can remove them if they change their mind.
(more…)


bbPress Benchmark, performance diagnostics

I’ve made another bbPress plugin, this time to benchmark and diagnose performance problems in page load times – bbPress Benchmark.

It is based on Jerome Lavigne’s Query Diagnostics for WordPress but I have taken it much further in what it measures (plugin & template load times, etc.) Here’s the story of why I created it: (more…)


bbPress Polls plugin

I am pleased to announce a new bbPress Polls plugin (in beta testing).

Your members can now add polls to their topics, or optionally you can allow any member to add a poll to any topic (even not their own), within a specified time period (hours) since the topic was started.

bb-Polls allows single answer selection or multiple answer selection polls.

No template or core edits needed of any kind. There are no additional db tables added, it uses bbpress topicmeta only. Only one entry is created per poll, regardless of size or options (this allow very easy cleanup if desired for some reason).

his is a beta test, however should be functional except for:
1. missing admin menu (coming soon – edit plugin directly)
2. missing administrative editing of existing polls.
3. missing some more deluxe options (poll ending time, etc.)

If you can live with these limits and would like to help test and give me some useful, constructive feedback, please download:

http://bbpress.org/plugins/topic/62
or
https://ckon.files.wordpress.com/2007/08/bb-polls.txt?v026
(rename .txt to .php, install, activate, start a new test topic)


(more…)


new plugin: bbPress Signatures

Adds signatures with ZERO template modifications required.

updated to v0.13 – now with admin menu!
http://bbpress.org/plugins/topic/63
or
https://ckon.files.wordpress.com/2007/08/bb-signatures.txt?v013
(rename .txt to .php)

If you have plugins for smilies and/or allow images in post text, it will obey them, this is probably a good thing and desired.

You can set the minimum user level needed to show signatures and also the ability not to repeat a user’s signature more than once per page.

Instructions: install, activate, tinker with settings in admin menu

If you would like the optional toggle on new/edit posts to disable signatures you must edit the edit-form.php & post-form.php templates and place at or near the bottom:
<? bb_signatures_checkbox(); ?>

(you can wrap it in a DIV and float it to the left, right style anyway you’d like
sorry but there’s no way to do this directly through a plugin)

Here’s a fancier version that I use:
<div style=”float:left” ><? if (function_exists(‘bb_signatures_checkbox’)) {bb_signatures_checkbox();} ?></div>

Version History:
0.05 :slashes & autop fixed, replaced input with textarea, max_lines now supported in post-processing, max_length checked in realtime (as well as post processing)
0.06 :internal testing/bugfix
0.07 :per-post signature toggle
0.08 :toggle for allow html and allow images should now work
0.10 :basic functioning admin menu
0.11 :more intelligent admin menu


bbPress plugin: My Views

Update: I’ve done a major update to this plugin and it now works in the bbPress trunk (1.0 alpha) or 0.8.2.1 Please download it directly from the bbPress plugins page:
http://bbpress.org/plugins/topic/67

My Views is a powerful addition to the default “views” in bbPress. It will let you customize output and adds several new views. My Views consists of a main plugin and several optional “module” plugins that can be added if desired for additional new views.

Appended bbPress Views by My Views:
* Latest Discussions
* Topics I’ve Started
* Topics I’ve Participated In

Extended bbPress Views by My Views if optional “bb Topic Views” plugin by Mike Wittmann is installed http://bbpress.org/plugins/topic/53
* Topics with the most views
* Topics with the least views

Extended bbPress Views by My Views if optional “Plugin Browser for bbPress” plugin by Sam Bauers is installed http://bbpress.org/plugins/topic/57
* Installed bbPress Plugins
* Available bbPress Plugins

and many little other tweaks to views including proper titles and headers as well as a dropdown views list box.


bbPress plugin: BB Tweaks

This plugin fixes some annoying default behaviours on bbPress 0.8.2.1

1. Get rid of the ?replies=# ugliness in topic links
2. add nofollow to post links
3. add target=_blank to post links
4. auto-close tags – bbPress’s auto-close tags is broken by default
5. get rid of [closed] on subjects – useful if you have a narrow template
6. adds extra tags to the allowed markup such as <hr>

You can edit it even further as you so desire.
bb-tweaks.txt (rename .txt to .php)


bbPress plugin: Admin Can Post Anything

This simple plugin makes bbPress behave more like WordPress in that admin (key masters/administrators) can post any content regardless of tag restrictions for the rest of members. This includes images, javascript, objects like youtube flash video player, etc.

now at version v0.05
http://bbpress.org/plugins/topic/55
or
https://ckon.files.wordpress.com/2007/08/admin-can-post-anything.txt?v005
(rename .txt to .php)


bbPress plugin: Recent Posts (admin menu)

This is so simple I bet I am duplicating what someone else has already done. I am not sure why this isn’t in bbPress by default and it doesn’t even begin to address what I would consider basic moderation tools but it’s a start.

Install and check under “content” submenu. Why the posts menu =deleted posts in bbpress’s default I have no clue.

http://bbpress.org/plugins/topic/54