High performance

When a page.tpl.php file is generated all of the variables are populated with content. This includes all the regions, the css and javascript, and everything. Some of these are never used in some themes (e.g., $mission) and some are often rebuilt later like $styles and $scripts.

I propose we lazy load the content in these variables when the variables are called the first time which will typically be in the template files. Since lazy loading is a feature of objects it will mean these variables will be attached to objects.

So, in drupal 6 you theme would have:
<?php
print $styles;

I have noticed (I think) a strange thing. My cache_get times are excessive compared to rest of queries consistently. E.g on front page for logged in user I am seeing these numbers through devl module

Last night I (finally) updated my (ultra-simple) personal website to 6.0, and while I was at it I decided to try out some caching ideas that had been brewing in the back of my mind for a while.

Though all caching strategies have their pitfalls and drawbacks, the idea I wanted to try -- combining Boost and Memcache -- seems like a real live-wire. However, if it can be made to work, I think it might offer some significant advantages.

I am struggling with the speed on my Views generated blocks. These blocks are essential for the functionality of the front page.

Logged out, with cache enabled both for page and block devel returns horrid page load times:
Executed 144 queries in 59187.65 milliseconds. Queries taking longer than 5 ms and queries executed more than once, are highlighted. Page execution time was 59675.94 ms.

Logged in as admin (uid=1) shows equally horrid page load times:

Hello Drupal world!

I released beta1 of the 7.x branch of Cache Router over the holiday. The new release passes all Core Cache Tests for APC, File, Database, eAccelerator, and Memcache. XCache is currently broken, and I'm working to get that engine working again as well.

Sorry, I'm sure this is addressed somewhere, but I can't find it. Can someone point me to a discussion that can help me decide whether I should be using Drupal 6 "Page Compression" or Apache2's mod_deflate?

Thanks.

--
Daniel

I've run into what seems to be a problem with memcache. Memcache has a gig of memory and generally sits at around 85% full with over 50% cache misses. The site has been hit with some traffic spikes of around 40k+ visitors per hour (mostly anonymous visits) and this has been enough to take it down (with the additional traffic to another Drupal site running on the same servers). There are two fairly powerful web servers and I think that if memcache were working properly, things would be fine.

Hi

I'm thinking about to port a site running on self build scripts to Drupal Version 6.
Here're some facts:

Registered Users: ~117.000
Averrage Online: ~6.500
Hits (monthly): ~2.200.000

Most used Modules:
- Forum (14.000+ threads)
- Private Messaging
- Image Gallery
- Buddylist

At the moment it all runs at a Clustered Hosting Plattform.
The Performance is very badly.

My goal is to write import scripts to Drupal and move to a Server like this:

Hardware:

• Intel Core2Duo 2x2,4Giga
• 2Gb of DDR Ram
• 1TB Mirrored SATA
• 2x1Gb Intel LAN

Software:

Hello,
I am planning on doing some benchmarking to assess two models of CCK used in Drupal 5.

We’re looking to expand our team by adding a couple more advanced Drupal experts. At this time we are only looking those individuals that have extensive Drupal experience developing custom modules and working with complex integration projects.

I am working on a new Drupal site and it appears to be pretty well tuned for traffic. There are currently 450,000 nodes that are updated on a weekly basis. I am running into some issues with drupal_execute. The script I have created queries a mysql database for all the information for nodes and loops through using drupal_execute to either create a node if one doesn't exist or update if it finds a match based on a specific CCK ID. I can get through about 40,000 nodes before my server comes to a crawl and starts swapping.

Hi Everyone,

I was thinking if some advanced server technologies can help to speed up the Drupal Site ?

Something like Zeus (zeus.com) or Litespeed (litespeedtech.com).

Will they be compatible with Drupal.

And has anyone already tried any of these or some other solutions.

I am not very good at programming and linux, so my question may not seem very intelligent to Geeks.

Best Regards
Dhaliwal

I will be hosting an auction site which will get quite a bit of traffic (expected 2000 viewers) on the day of every auction. Most of the time it will be quite a low traffic. What hosting would I need to accommodate this? What steps should I take to optimize performance. I've looked at media temple, mosso, slicehost and a few others.
Should I go for a cluster, vps or dedicated server.

Thanks.

I support a D5 site that has been averaging around 18 secs load time for the front page (it has lots of external httpd calls, javascript, css files, etc). The architecture consists of a redundant front-end load balancer along with a pair of Squid caching servers, a RHEL5 GFS 4-node cluster, Memcached cluster (1GB per server, 4 servers total). D5 site is running the Memcache API as well.

LoadRunner Setup: 20 Minute Run-Time, 100 Virtual Users (Starts at 2, increases by 2 each 10 seconds)

So I have a drupal 5.x site set up with 1600 nodes, 10 users, a custom theme, and lots of modules.

I just finished setting up a support part for the site where the client is going to be adding an invoice for all the products he sells. He has roughly 5000 buyers and each buyer buys around 2-5 products minimal from him a year.

So lets say I'll be adding roughly 25,000 extra nodes a year to the site for the support section.
In 5-10 years that could reach 250,000 nodes. Is this a lot? Would this slow down my main section on the website?

hi all,

posted this on the dev list, adding it here in case some in this group don't watch the dev list.

i've been watching the inspiring work to integrate testing patches against HEAD at testing.drupal.org, and wondering if it would be useful/possible/desirable to have benchmarking integrated into d.o. as well?

like the testing stuff, but maybe opt-in?

i'd be willing to do some dev/sysadmin work, and i'd hope there would be enough dev resources, but we'd need some hardware dedicated to it.

thoughts? suggestions? flames?

cheers
justin

I am the back-end engineer/architect working on consolidating a number of production web sites using Drupal 5.11 with the Drupal Memcache API as a base along with the multi-site configuration.

Here's the architecture so far...
- 2-node load balancer front-end (2 Cisco ASA 5510s)
- 4-node web/app servers: RHEL 5 + GFS + Drupal Memcache API + Memcached (Quad Xeon, 8GB Ram each)
- 1-node MySQL 5.x database server with two other MySQL servers in hot stand-by mode (Quad Xeon, 16GB Ram each)
- EMC SAN Storage (1.5TB allocated)

This is a wiki to plan a module to allow admins to add indexes to their site schema - this can be done easily in Drupal 6 and beyond using http://api.drupal.org/api/function/hook_schema_alter/6 - this discussion came from http://drupal.org/node/231453, where it was realized that CCK does not really have the usage context to be able to do this effectively, and also there are tables beyond CCK getting used in new ways by views and other modules that also need alternate indexes.

I am setting up 2 load balanced servers under 1 ip to run my Drupal sites (multisite installation).
The sites will be using a separate dedicated server for the MySQL databases, on the same rack and 100Mbps network for faster access.
All 3 servers (2 app servers load balanced under one ip and the dedicated mysql server) are on the same physical rack.

Several questions come to mind, in particular...

APPROACH 1: rsync /home directories accross both servers.

Hi!

Is it possible to gzip CSS and JS files in Drupal5?

Thanks

forgive my poor English, but i really want to have my problem fixed and join you though i am not a programer.

i use drupal6.6 to convert my origin website , which is a great flux one in my school about 5000IP/day .
i fixed a lot of problems through GOOGLE , tpl.php,form.api,CCK+VIEWS,etc

but i really come up a problem : too slow to do some update after login!

every day , my website will have 50 students and teachers have to update their information like news and so on . they tell me that the new test website(drupal) is great but the performance is too bad.

i was pondering how to speed up some complex pages for authenticated users and I had an idea. the gist is to almost exactly emulate block caching but applied to the return value of the hook_menu's 'page callback' rather than hook_block('view').

To make it easier to find information on these matters, I've started the new handbook page Drupal caching, speed and performance in the Tutorials section of drupal.org. It's just a guide to information and tutorials on optimizing Drupal's performance, speed, and scalability, with an initial listing of fourteen resources. If anything is missing, you may edit it to add other useful Drupal performance resources.

We conducted some benchmarks for Drupal 5.12, Drupal 6.6 and Drupal 7.x (Oct 24 checkout).

We used the same data set for all 3 instances:

Users 4,950
Nodes 5,000
Comments 20,000
Vocabularies 10
Terms 1,000

And same test parameters: 10 concurrent users, 2 minutes stress test on 30 individual nodes, and the home page.

As seen on Planet Drupal, the recent article Improving Drupal's Performance with the Boost Module for the UN's Millennium Campaign (October 23rd, 2008) describes how a Drupal site successfully managed a very high traffic situation. They achieved this with "one enormously helpful Drupal module, called Boost", and "some fine tuning".

Hi

I am developing a bigger site (D5) with a free tagging vocabulary. Currently the vocabulary has 16.000 entrys. i have searched with google and on drupal.org, but I have not found some performance improvement like cache or pacthes.

What are you doing?

I think I am not the first one with bigger vocabulary. Some links or code snippets would be great.

thx
bennos

I would like to know if you guys using Views for high traffic web sites or not?

There are a lot of mess in our current websites.. Some of the pages and blocks were built using views and some of them using about 6000 queries to display nodes listing on the page! And some using custom modules and custom queries. I personally prefer to write custom code because it easier to maintain and it's easier to find where is that query if u need to modify it.

I have been testing turning css aggregation on/off, i am using drupal 6.5 with a modified version of aboutpeople theme.

In either case, it looks fine in firefox.

But in ie, with css caching enabled, can not find the aggregated css file.

I think it would be nice to know exactly how css aggregation works, and maybe instead of a different file name each time, it creates one file that it updates, or deletes depending on css aggregation being on.

I've been pointed in the direction of the High Performance group to discuss my requirements (http://drupal.org/node/317535). I’m looking at building a gallery site that runs the following predicted metrics each month:

• 320k visits
• 200k unique visits
• 5m page views

• 14k uploads
• 16Gb total uploads

Anecdotally I'm being told that running this on Drupal is a risk, but all the documentation suggests that as long as we have good (dedicated) hardware, good 'normal' website tuning practices and a caching facility, then we should be OK.

You are invited to attend Drupal Camp Chicago on October Friday 24 & Saturday 25 2008.

http://drupalcampchicago.org

Who/Why:

For over 2 years Chicago Drupal has truly highlighted the benefits of an open source initiative. We have built resourceful bridges between many local Chicago establishments including non-profits, entrepreneurs, Drupal development companies, independent consultants, hobbyist, and others in the mid-west. Our goal for this Drupal Camp is to bring everyone together for open communication, education, fun, and industry networking.

What:

This position is in our office in San Francisco or Los Angeles

WBR is the top label in the US with some of the most respected acts in the business, including R.E.M, Neil Young, Muse, White Stripes and Green Day. We are currently celebrating our 50th anniversary. Besides that, we are the only label with a Technology Department entirely dedicated to doing great stuff online. We run the department as a startup, with all the benefits, toys, electronics, servers, Macs, conferences and geeking out. And free soda. And video games.

Patches that impact Drupal's performance. Have a need for speed and some mad benchmarking skills? Help out here!
General big improvements

• Cache node_load
• static caching of user objects in user_load(), respecting conditions
• multiple load users
• multiple load files
• Adaptive path caching
• Improve performance of locale cache

Remove LOWER() from queries

• Remove LOWER() queries in user module
• Remove lower from taxonomy (use ILIKE on postgresql)

Remove db_rewrite_sql

• drupal_lookup_path() optimization: maintain a whitelist of Drupal paths

Other issues

• make modules only require .info files
• Descriptions node_types too long causing queries
• make variable prefetching optional
• Unify and rewrite module_rebuild_cache() and system_theme_data()
• Caching (per user) for authenticated users
• temporary tables in taxonomy module queries - eliminate queries that write temp tables to disk with MYSQL
• Support disabling anonymous sessions - makes it easier to use Drupal with HTTP accelerators like Varnish
• Allow JS/CSS aggregation with private files - let users of private files get these performance improvements too
• Separate Revisioning from Node module - improves performance for sites that don't rely on node revisioning

Mindmap

• Catch: have block_cache work with the who's online block, and not disable itself when node_access modules are enabled.
• RMuilwijk: Enable multiple user_load as like patch Load multiple nodes at once

Hello, i am currently trying to optimize a website with 82000 nodes and 749414 comments. Unfortunately, it's badly developed, many modules, etc. but I can't do anything with it right now.

As far as I've been searching, frontpage took 800 (!) SQL queries to display. I used:

• Block cache
• Drupal lookup path optimization http://drupal.org/node/106559
• Removed database locking http://drupal.org/node/55516
• Index for pager query on frontpage http://drupal.org/node/295283
• Advcache with taxonomy/node/comment cache
• using eaccelerator

What's the wierdest things you've used drupal for?
Something other than a website serving up nodes of content to the masses... Funky creative uses..

I've been tweaking a my.cnf file for the last couple weeks, using the tuning primer sh script as a guide. You know the routine, I make my change, restart mysqld, then run the script in a day or two to check improvements.

I'm understanding all of it except for Temp Tables. I always get something like this.

I am releasing a site next week that has (in the past) gotten up to 1200 simultaneous users. My worry is that the performance of the site will not be good enough to handle that kind of traffic. I used YSlow, but there are probably better testing plugins somewhere. Here is the case:

Server:
4x 3.0GHz Cores
4GB RAM
Dedicated/Co-Located
Linux (Cent OS)
Apache
MySQL
PHP 5
Drupal 5

YSlow Analysis Score: 72 (C)

A 1. Make fewer HTTP requests
F 2. Use a CDN
A 3. Add an Expires header
F 4. Gzip components
A 5. Put CSS at the top
C 6. Put JS at the bottom

A partial investigation of running a Drupal (6) site containing millions of nodes

The (three) problems that I've run into so far are all in search.module -- the "more" core modules seem to do okay.

PHP Accelerators do not have any additional benefit! Are we doing something wrong? Any special configuration we need for enabling them? What kind of performance benefits can we expect?

We are running RHEL5.1, Apache2, MySql5, PHP5 and have run into an issue. Our site runs find with normal usage, but if you hold the refresh (F5) key, then the site will crash. We have gotten it to the point that it will not crash the server, but it will take a while before the page responds (for any user). I know that this does not happen on other sites, so we are most likely missing a configuration setting somewhere. Here is what we have:

Apache

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 2
StartServers 8
MinSpareServers 5
MaxSpareServers 20