Scaling WordPress to handle massive amounts of content and traffic is a challenge that many organizations face, especially as their site grows in size and popularity. Every site is unique, but there are common obstacles encountered when building a WordPress site that can efficiently serve millions of users and manage vast amounts of content. Over the years, working with large-scale WordPress installations, I’ve learned key lessons on building scalable, robust WordPress themes and plugins.
In my WordCamp talk, From Bottlenecks to Breakthroughs: How to Master Scalability (video, slides), I shared strategies my team at Alley developed to build WordPress sites that have no limits on content or traffic.
Below is a summary of the most important technical points from that presentation.
Key Architecture Requirements
Persistent Object Cache
A persistent object cache, such as Memcached or Redis, stores data across page views, dramatically reducing database load and improving site performance. While many WordPress hosts offer this feature, themes and plugins must be optimized to make full use of it. For instance, data from external services or large database queries can be saved in the object cache, reducing repeated database calls and improving load times across the site.
WordPress has a built in Transients API for storing data to be re-used across requests. However, without a persistent object cache, these transients are stored in the database, which adds more load to the database. With a persistent object cache installed, transients are instead stored in the object cache—removing the strain on the database.
We achieved a 98% object cache hit rate by ensuring we’re caching data that can be reused across page views. We developed the Remote Backstop plugin to automate storing data from remote requests. This allows the site to handle temporary issues with external resource availability gracefully—the data will automatically be returned from the cache if the external server is stressed or down—thereby alleviating the load on the external service and ensuring that the user’s experience is uninterrupted.
Offloading Complex Queries to Elasticsearch
WordPress’s normalized MySQL database design is optimized for content insertion and updates. However, it can struggle with large, complex queries as the database grows, especially when it comes to searching through taxonomies or meta fields. Elasticsearch, which uses an inverted index, is far more efficient for these kinds of queries. By offloading complex searches—such as taxonomy-based queries—to Elasticsearch, we can dramatically reduce the time it takes to retrieve and display content. With our custom plugins, SearchPress and ES WP Query, complex queries are seamlessly offloaded to Elasticsearch.
Regional Page Cache
By utilizing WordPress VIP’s geographically distributed page caching, we ensure that visitors across the world receive content as quickly as possible. When a page is requested, it’s automatically cached and stored in the nearest server to the visitor. This means subsequent visits from the same region load pre-built HTML without triggering expensive PHP processes. For sites with high traffic from logged-out users, like those driven by viral content or breaking news, this reduces server load significantly while delivering content faster to users.
No Frontend Database Writes
One of the quickest ways to make your site crumble under heavy traffic is by allowing database writes during user visits. This is a common issue we’ve encountered with plugins, and nearly every critical production incident we’ve faced can be traced back to overwhelming the database with write operations. WordPress is not designed with a separate read-only frontend. Many standard WordPress API actions—such as updating rewrite rules or scheduling cron events—result in database writes. To prevent this, we’ve developed architecture best practices and flag frontend database writes with a doing_it_wrong notice.
You may wonder how we handle typical features like user registration, paywalled content, and commenting without database writes. These actions are offloaded to external services. For example, when someone registers for site membership, they don’t become a WordPress user. This means their page view is handled just like any other user’s, minimizing database impact. We do maintain separate cache buckets for paid content, but from WordPress’s perspective, there’s no difference in handling between regular and premium users.
Similarly, comments are managed by an external system. This way, even if an article receives thousands of comments, WordPress doesn’t experience any additional load. Offloading these tasks ensures the site can scale smoothly without impacting WordPress’s performance as traffic and user activity grow.
Conclusion
Building a scalable WordPress site is no small task, but with the right architecture and optimizations in place—such as a persistent object cache, Elasticsearch integration, and a solid caching strategy—it’s possible to ensure your site can handle massive traffic and serve millions of users seamlessly. At Alley, we specialize in building high-traffic sites that are flexible, reliable, and scalable for the future.
If you’d like to learn more about how we can help your organization achieve this level of scalability, feel free to reach out or explore Alley’s work.