making bbPress (and WordPress) work better!

To Save the Future of WordPress, Steal This Idea…

A Little History…

Nearly 13 years ago, Michel Valdrighi quietly added some code to his
program which unknowingly was to become one of WordPress’s greatest abilities:

That code commit back in October 2002 contans a single note: “now with b2_filter support !” – perhaps the exclamation point exposing a little thrill that this was something new and exciting, but I have to wonder if he could actually imagine what it really meant for the future.

In fact with the addition of those functions “add_filter/apply_filters” one can begin to easily recognize some of the oldest source code as WordPress itself, compared to earlier versions.

For the first time, coders could affect the code output in a standardized way,
without directly modifying the core of the program. It gave b2 tremendous potential.

b2/cafeblog version 0.6 would shortly afterwords be taken over by Matt Mullenweg, simply renamed and re-released as WordPress 0.7 with few other modifications. But even more filters and later a related ability called an “action” were widely adopted.

All the work by by Michel Valdrighi (and later the wholesale adaptation of Justin Vincent’s EZSQL) which breathed life into WordPress are virtually unknown to the millions of users today. Their names are only mentioned once without any weight, buried in the source code.

The Present Day…

When I say WordPress needs “saving” people might understandably think I am crazy. Matt regularly boasts how many millions of people are using it, both standalone and on his hosted service – one could never imagine it fading away at this point.

But here’s the ugly truth – WordPress is the Titanic and its going to rip apart based on it’s own sheer weight. The iceberg is dead ahead and its name is massive bloat. Each year it grows bigger and more troublesome and more dangerous with every new release.

WordPress’s dirty little secret is that it has only one way to run – virtually ALL it’s code, megabytes of it, must be loaded, every single time, to create any single page. It never gets smaller and always grows with every release as mandatory “features” are folded in.

Numerous caching plugins have been invented to desperately offset this problem but on busy sites where there are constant cache misses and constant clearing of caches with new posts and new comments there is no way to avoid the growing problem.

Fortunately servers grow more and more powerful and less expensive to operate each year which compensate for some of this performance problem – and PHP keeps getting faster – PHP7 will double the performance of all previous versions. But even today, get several editors mucking about in the admin area and you will certainly see the strain of resources across the entire system.

And just try to find a WordPress installation that runs without plugins today. The average website uses at least a dozen as the desire for uniqueness, customization and functionality grows. Plugins are the very life of WordPress, there is no way it would be so popular without the thousands of free extensions and mods that are available to anyone who wants all their abilities.

However all these plugins also load virtually all their code, every single time, on every single page. Hundreds of files, megabytes of code, regardless of the output. Iceberg dead ahead!

The Future…

While working on a bbPress replacement of my own, I wanted to have the best of both worlds, custom plugins which could richly enhance the core program, but not suffer from the massive bloat that WordPress has experienced over the years. It seemed to be a dichotomy.

Why should plugin authors have huge amounts of code load on every single page for every single user like WordPress does? Previously on my bbPress plugins I created a way to have a “stub”, a little piece of code that would detect when to load the admin functions. But even that was wasteful, it was just another file that still had to be loaded and parsed every time.

So now I give you my simple evolved solution that is inspired by Michel Valdrighi filters from nearly 13 years ago. It’s 100% backwards compatible and is incredibly easy for plugin authors (as well as core developers) to apply to their work, and forever save WordPress from the threat of it’s own sheer bulk.

“Pre-threaded” Plugins

What if a plugin could load “on demand”. Even parts of it, only if and when it was finally needed, instead of every page load?

What if this was as simple as taking advantage of those very helpful filters and actions scattered throughout WordPress?

What if it was as simple as plugin authors declaring where and when their plugins should load, with just one line in their plugin?

Starting to get the idea? Here’s how simple this is and the mystery is why no-one else has done it for nine years.

Every plugin has a header, it’s mandatory, at least the Plugin Name:

Plugin Name: Your Plugin
Author: Your Name
Version: 1.2.3

Now imagine one tiny addition:

Plugin Name: Your Plugin
Author: Your Name
Version: 1.2.3
Load on Action: admin_init, 9

What does this mean? Well during the installation/activation of the plugin,
WordPress could now can see and parse that special header “Load on Action”
It adds this to it’s database, just like it currently stores all the plugin names and locations, when to load the plugin!

Before any plugins are loaded, WordPress loads it’s database.
It can easily and extremely quickly parse this kind of threading declaration
then simply add the actions (filters) from each plugin to load the file,
only if and when it hits that filter/action later in the page loading process.

The “9” is an optional priority – it simply means it would load before other plugins or actions that might have the unlisted default priority of 10. Only use it if you need it.

Why is this all so beautifully simple, yet important?

Well you no longer need any kind of master code or “stub” to load any part of your plugin. Large parts can simply be left standalone and WordPress will know when and where to load them.

Admin functions can be completely on their own. Registration functions can be on their own. Feed functions can be completely on their own. If they are megabytes in size it doesn’t matter, they do not affect all the other page loads.

Now think bigger – even CORE functionality can work with this.
Instead of guessing what it needs to include for unknown later processes,
load the functions only if and when it reaches that point.
Pre-thread the core and let it determine what it needs later.

Have different situations where one action might not happen but another? Simply list more action lines. The pre-threading would track if the plugin was loaded and do it only once, like “require_once” (but faster as PHP has a performance penalty for “require_once”).

What about all the huge legacy of existing plugins? Well they will still work exactly as they are and load at the regular plugin loading time because they will not have the extra header. Meanwhile newer plugins with threading can load before, after or not at all anywhere in the page loading process.

I hope to see this functionality in WordPress 4.4
But I have bug reports from half a decade ago that were only recently fixed in 4.x
So I guess I will wait and hope for sometime by the end of the decade…

15 responses

  1. I think the real solution is making WordPress more object-oriented, and using class autoloading. But that would be a very long-term project, so something like this could be useful while we work on making WordPress more OO.

    A correction though: WordPress doesn’t store the plugin headers in the database. It does cache them, but only in the context of the current request. The cache is non-persistent. This means that WordPress would have to read the first so-many bytes of each file on each request to check for the header. The reason that file headers aren’t stored persistently is that the files can change without notice. WordPress doesn’t know when a new version of a plugin may have been uploaded using FTP, for example.

    August 20, 2015 at 4:30 pm

    • Classes and object-oriented actually make code slower, measurably.

      They are only used to make coding easier.

      Last time I looked at WordPress plugin loading (back in 2.x) it only re-examines the state of the plugins when you go to the actual plugin page in admin, the rest of the time it is “cached” in wp_options.

      The file size and timestamp could be stored in that cache and before the plugin is included, it could compare the size/time to see if the file changed and needs to update the cache.

      August 20, 2015 at 4:36 pm

      • J.D. Grimes

        My point about OOP is that autoloading for classes is built-in.

        I don’t think that WordPress stores any of the plugin file headers in the database anymore. It only uses them on the plugins screen, so that’s the only time that it parses them.

        You’re right that the file time and size could be checked, but that still requires touching the file in some way. It might still be much more performant though.

        August 20, 2015 at 4:52 pm

      • Modern linux/windows is very good about caching file stat so the overhead is trivial to check size and timestamp (vs reading the file). The only problem is when the files are on NFS storage but one would be insane to try to run WordPress from a non-locally attached drive on a live server. Thanks for all your feedback.

        August 20, 2015 at 5:00 pm

      • J.D. Grimes

        OK, that’s good to know. What about checking if new files were added? (Edge cases, edge cases…)

        It would be interesting to see a benchmark on this idea. Of course, WordPress would need to be refactored heavily to take full advantage of it, I think.

        August 20, 2015 at 5:13 pm

      • You can’t add plugins without activating them right? So that would scan and store the new load-on-action.

        WordPress itself would need a little refactoring to take advantage but for plugin developers, this is a no brainer.

        Virtually all plugins load all their admin code ALL the time, even for regular users just browsing the site. Some devs have caught on and conditionally load the admin code from the main plugin as needed, but not enough. This makes it completely painless for them.

        August 20, 2015 at 5:24 pm

      • J.D. Grimes

        But what about when a plugin is updated via FTP? I’m assuming here that any file in a plugin could include this header and be loaded by WordPress as needed. So I’m thinking we might want to handle the case where a plugin is updated using FTP and new files are added that include this header. WordPress would need to scan for new files to detect that those files needed to be loaded.

        I only load the admin code when is_admin() in my plugins. But I know that many plugins don’t. I’m not sure why plugin devs would pick this header up over that though, but I guess maybe it is a bit easier since you don’t actually have to write any code.🙂 But for devs that are paying attention, yes, it would make this easier and at the same time allow for more granularity.

        August 20, 2015 at 5:43 pm

      • J.D. Grimes

        I think the solution to this is to put all of the directives in a single file. It could be JSON or whatever. It would list each of the files to be loaded and on what action(s) to load them. Then you would only need to check that one file to see if it had changed.

        August 21, 2015 at 8:45 am

  2. Dang, that’s a really interesting concept. For thanks for sharing your idea.

    August 21, 2015 at 3:02 am

  3. Greg Ichneumon Brown

    You could run pre-fork which keeps most of the code loaded in a php process and then forks new processes off of that process to avoid reloading the same code.

    For certain use cases (the REST API for instance) it can shave a fair bit of time off.

    Moore’s law and caching are pretty powerful and effective mechanisms. Also there is likely to be more and more going into client side code and REST APIs which will significantly change the performance game.

    August 21, 2015 at 5:39 pm

  4. You could get a pretty huge boost simply by splitting out visitor-loaded code from admin-loaded code or, as WP used to do, only loading most of the code needed to display on that page itself. Hooking it to actions (which I’m proud of as an extension to the filter idea) is a pretty elegant way to think about it, but could probably be simplified more.

    However Greg does have a good point on the “for free” boosts we’re getting, and overall I think an architecture that’s closer to a single-page javascript app with server-side rendering, where you’re just moving data around from page to page, is going to be much more performant and bandwidth efficient in the long run.

    August 28, 2015 at 5:10 pm

    • Honored to have you comment here!

      re: Javascript solutions, I feel this is a mistake, or at least premature, as the web migrates more and more to mobile, yet mobile devices have a fraction of the processing power of desktop and servers.

      Remember, Javascript is single-threaded, it all has to run single-file – you get a pile of JS on a page with three ad frames on top of that on a typical blog page and you are going to grind a low-end phone or tablet to a halt.

      August 28, 2015 at 5:28 pm

      • You definitely have to be cognizant of front-end performance and processing power, but we’re starting to see more and more examples of it done really well. It might never make sense for blogs, where people only see one post and bounce (server-side rendering helps), but for forums or admin sections where you have large amounts of logged-in activity it can be a super-fast approach.

        Ads are always the worst code on the internet, and once you include them you can’t really be accountable for performance any more.🙂

        August 28, 2015 at 5:45 pm

  5. Reblogged this on Teague Talk.

    September 3, 2015 at 10:54 am

  6. Not to sound like a modern day Valley Girl, but “Oh My God!” You totally defined in words what I have poorly attempted to convey for years. Perfect explanation of the problem and solution. Thank you.

    September 3, 2015 at 10:59 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s