High performance

Not sure if the title is good enough to describe this problem, but here goes...

I've had some small input into the KML module (http://drupal.org/project/kml). This generates KML (Google Earth) files which show placemarks for content with locations. While rather cool this can be very slow when you have many nodes with location information.

I'm having serous issue with private file transfer, I have about 50 thousand images added to the system and all transfer via drupal private file system. I have about 2lks users and when about 200 online, average server load about 30-40, its heavy load to server, I'm running my mysql server on localhost, what should I do to increase performance?. its taken about 20s to load a single page, about 20 mobule installed with views , cck etc.
I recently installed boost module but still the server load is heavy. What should I do?

The APC documentation says about the apc.stat option:

On a production server where you rarely change the code, turning stats off can produce a significant performance boost.

Did anyone ever benchmark this and cares to share the results?

Have another idea for the boost module, use the -s flag instead of -f in RewriteCond.
http://httpd.apache.org/docs/trunk/mod/mod_rewrite.html#rewritecond

Le Figaro (the oldest and the biggest daily newspaper in France) invite media organisations using Drupal to discuss “Drupal and Media”.

We will stream live our first meeting about the use of Drupal in Media
Stay tuned on Ustream http://www.ustream.tv/channel/drupalmedia Thursday the 11th from 2PM to 5PM Paris's time (8AM to 11AM East Time)
You will be able to tchat and have your input.

Simon Elliott http://groups.drupal.org/user/27802 will translate as we go on the tchat so you can follow our debate

We have News site with lots of hits per day. So we are decided to optimize the code for anonymous user. We logged the query list of all pages, then we confirmed that all pages are retrieved from performance cache except the node page. The same bunch of queries repeated twice in the node page. In our node page we have some custom blocks, node comments and captcha.
Then I take the following steps to find the cause.
Removed the block from the node page --- No improvement
Enabled the block to the node page and disabled the comment module ---- No improvement

Hi,

I've been testing some code generation techniques for Drupal; I created a module that copies the code of your drupal installation to a temporary directory. First it splits all functions and class into seperate files. Then it modifies the function calls so that functions gets loaded when needed.

It will decrease memory usage a lot! Especially when a great number of modules are installed.

Hi Everyone,

It has been many years since I have posted on the Durpal forums, and as many years since I have used Drupal. For the past 8 years we have been developing our e-learning and corporate communication system on a PHP framework established in 2001 and recent developments have brought me back to Drupal.

I just wanted to share some technical information about a very different caching scheme, in hopes that someone would be able to make use of the information for Drupal.

Hi-

I am new to Drupal, but I have always avoided use of .htaccess files and mod_rewrite because I was under the impression that they could negatively impact performance.

Here are some excerpts from Wrox's 'Professional Apache' in the chapter on improving Apache's performance:

We're getting close to relaunching our site (information.dk - see http://drupal.org/information.dk) on Drupal 6. We're still using panels. This time it the SUPER panels3 for D6. However we have one concern and that is speed.

On d5 we used panels_preempt, but right now there seems to be no other cache for Panels 3 than Drupals page cache or panels3's simple cache. Have any of you got an alternative?

On a D5 site with high (registered) traffic, there is also a requirement to count node views by anonymous visitors.

The existing setup is using Boost, and node view count is done using an ajax call on the pages, into a hook_menu that increments the counter.

hi!

Do someone has an idea on cache db vs cache file? i mean which one is faster.

Hi!

First, let me apologize if my english hurts. I'm not a native.

Thanks for the existence of that group. Though i come here for the first time, I'm sure it helped people many times before.

I'd like to expose my concerns and worries too about drupal. I'm in charge of a website which has about:
- 11.000.000 pages views a month
- 300.000 unique visitors by month
- 100.000 registered users

The settings are:
- each user can post a content

Please refer: http://drupal.org/node/230235

Thanks for the patch, it gives a good performance benefit.
if (!isset($count)) {
//$count = db_result(db_query('SELECT COUNT(pid) FROM {url_alias}'));
$count = (int)db_result(db_query('SELECT pid FROM {url_alias} LIMIT 1'));
}
Here the $count variable is used to indicate that a url alias has been created. Can't we use $count =1 or session variable like $_SESSION['count'] instead of running the query "(int)db_result(db_query('SELECT pid FROM {url_alias} LIMIT 1'));" ?.

Any help for a relative noob?

I've set up a drupal server that uses nginx + php-fpm and boost. Boost and its rewrite rules are working correctly for anonymous users, as evidenced by the embedded boost comments in the pages I'm seeing.

I'm trying to evaluate if our VPS LAMP server can accommodate a website for a university department. Last years stats show this heavy day:

Page views: 23,000
Unique Visitors 2,500
Average Page views/visitor: 7

This is a "read only" site. Users just browse, they don't create content or submit any forms of any kind.

I am trying to use the memcache.

I am logging the queries after and before enable memcache.

I am using following in _db_query function in database.mysql.inc I am using following code to log the queries.

if($_SERVER['REMOTE_ADDR'] == '192.168.99.180') {
$file = '/var/www/html/example/files/query.txt';
$handle = fopen($file,'a+');
fwrite($handle,$query."\r\n");
fclose($handle);
}

Before enable the memcache file size is 1.09 kb after enable the memcache the log file size is 11.09 kb.

Hello High Performance-Group,

we are about to move our client-sites from an only-static-CMS to Drupal and worry how our high registered traffic will do and what would be best setup for Drupal, caching, database- and servercluster-setup. Anybody has ideas and/or experience?

Some figures:

Traffic:
PageImpression-peak per hour: 150,000 (42 per sec.)
PageImpression-peak per hour in closed area/registered user: 75,000 (21 per sec.)
Unique visit-peak per hour: 75,000 (21 per sec.)
Active session-peak (open parallel sessions, timeout 15 min.): 1,500

Content:

How/where to check the number of queries running in a page other than devel module.

Regards
R.Navaneethakrishnan.

I just have tested memcached on Windows/Apache to investigate, if a performance can be expected.
I was very surprised, but it seems that memcached does not perform better than the Drupal built-in DB-Caching mechanism.
I am very curious, if anybody else had different results.

Detailed information follows.

My environment
- Windows XP
- Apache/2.0.61 (Win32)
- PHP Version 5.2.6
- MySQL 5.0.77
- Memcache 1.2.6
All installed on single box.

I have set up two environments with copies of the same DB.

Hi,

As we all know a Drupal site becomes less efficient the more modules are added to it. I found myself working on a site with 115 modules or more and found in one instance that Ubercart was loading css on every page load, and not only was this contributing to the overheads, it was breaking the site in some forms in IE. While e-commerce forms a small part of the site functions, most of the site does not need it at all, so I looked at the idea of splitting the site into subdomains, each with about 40 modules or less.

There are many ways to improve the performance & scalability of Drupal.

Static html file generation - Anonymous Users Only

http://drupal.org/project/boost
http://drupal.org/project/cachestatic
http://drupal.org/project/html_export
http://drupal.org/project/savetoftp

Caching of pages in DB

http://drupal.org/project/advcache
http://drupal.org/project/authcache
Drupal's Core Cache

DB Cache

http://drupal.org/project/fastpath_fscache
http://drupal.org/project/memcache
http://drupal.org/project/cacherouter

To Do

My site is high performance site To improve the performance I have installed the BOOST module in development version.

The node is cleared once the node gets updated/deleted.

My doubt is:

1.The home page nodes not get cleared once the node get updated [I am using panel for home page].

2.Boost cached files are not deleted after x times [ Set in performance ].

Can you suggest me...

Regards,
R.Navaneethakrishnan.

The Drupal project now tests every core patch for Drupal 7. To date we have tested almost 7000 unique patches, 40000 times. We are getting ready to roll out the 3rd version of the patch and core testing system, http://testing.drupal.org. In order to ensure the testing system is running correctly we are looking for a volunteer who can tune each of the volunteered servers(4-20). You don't have to be an expert in Drupal performance, you just have to be willing to learn and have some basic competency in Linux System Administration.

At MySQLcon I ran into the guys from Codership. They did a 25 minute presentation on how to do multi-master synchronous replication across many instances of MySQL.

In layman's terms.

If you are running 5 Master MySQL databases behind your giant Drupal site, and one of those servers crashes, you won't lose a transaction, and you won't corrupt your logs and destroy your database.

I was thinking about trying to monitor several Drupal metrics using Cacti and went looking for someone who's done it before. Doesn't seem so, but I would have thought monitoring statistics on your site(s) is an essential part of any high performance setup so bring it up here.

Anyhow - found something similar only and added a comment over on a thread about monitoring Drupal:

http://groups.drupal.org/node/20271#comment-74657

Anyone here interested in such an effort and how it might be built?

Hi all,
I'm running a zen Based VPS at blacknight with some, relatively speaking, low traffic sites.

I upgraded one of those sites to Drupal 6 this morning, and now the server seems to have gone haywire.

I'd appreciate any advice as to how to recover the situation.

One of the sites is www.athenryac.com, and judging from the browser, a connection is being made to the server,
and it is is either extremely slow, or just times out.
Even ssh access is very slow.

The server is running Ubuntu 8.4.01
Php 5
with APC installed.
Mysql 5.

Phpinfo, and my.cnf pasted below.

I'm using Amazon EC2 small instance with EBS, which according to Amazon is:

Small Instance (default)*

1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage (150 GB plus 10 GB root partition)
32-bit platform
I/O Performance: Moderate

Software wise:
Web server Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_perl/2.0.3 Perl/v5.8.8

MySQL database 5.0.51a

I've installed APC and cache router, but wasn't really able to tell if it's working:
http://drupal.org/node/434950
http://drupal.org/node/436236

In my never-ending quest to run larger and more complex drupal stacks for more and more users, I'm starting to hit the wall in what I'm able to accomplish with good old Siege, which has been my command-line tool of choice for benchmarking and performance testing for the past couple years. In particular, it breaks down too often in high-load simulations, and doesn't allow for any complex multi-threaded testing, making it very difficult to model near-reality user scenarios like "10 logged in users + 100 anons".

Lately, I've been getting into Jmeter, which has a lot more bells and whistles -- including a GUI! -- and which I think can offer a lot to Drupal developers. However, their basic web test plan barely scratches the surface of what's possible. With the right configuration, jmeter can effectively simulate complete user-behavior patterns like logging in, posting comments, etc.

I'm just getting started, but am curious if people "out there" are already way ahead of me, or if not if folks are interested in seeing the results of my testing work?

I suspect that the tables in the Nodequeue module can be optimized, but I'm not sure how to start. Nodequeue allows people to create arbitrarily ordered lists of nodes, and consists of queues and subqueues. A subqueue is a distinct list of nodes that inherits properties from its parent queue. Nodequeue's Smartqueue API allows modules to define new types of subqueues, such as a subqueue per-user or for each taxonomy term of one or more vocabularies. Below is the desc output of nodequeue's tables.

We've got an issue with one of our servers that hosts multiple drupal sites. It was running just fine, and now there's a problem. We're looking to see if there are any other issues causing the server slow down. We're also reassessing our server requirements and so what kind of machine would we need - how many sites can you run serving what n number of requests? I know there's a bunch of variables, how many modules etc... but i'll try and be as specific as possible.

When I saw this posting, http://groups.drupal.org/node/20494, I thought déja vu but as it turned out, no such luck, though the end result is the same, or worse.

Situation:

development server: XAMPP 1.6.8 on vista premium, PHP Version 5.2.6, acquia drupal 6.10, eAccelerator 0.9.5.3 (couldn't get binary for APC for this PHP version).

Staging server: Ubuntu 8.10, PHP Version 5.2.6-2ubuntu4.1, APC 3.0.18

Well, I read your article about fcgid.

Let's talk about architecture of apache + fcgid and nginx + php-fpm.

fcgid and php-fpm look similar in the way they spawning processes.
Both use preforked processes of php and respawn about 1000 times of running script.

Apache and nginx both use fastcgi to call php from itself.

Hence, the difference is between apache and nginx.

As i know, apache run by default 4 preforked processes and pass request to one of its workers.
fork is the only stable model of apache in linux. http://httpd.apache.org/docs/2.2/mod/event.html - experimental

If your server has high load you may consider following approach:
1. Use logs of server (i.e. Apache) + awstats updated by cron.
2. run awstats to parse log on another server or nightly when no load.
3. Apache work slower with logging. I prefer to log only php,html - content, no jpg,css,js etc.
4. Why Apache? nginx works much faster. nginx + php-fpm + e-accelerator 2-3 times faster and don't use so much memory like apache.

eaccelerator.shm_size="16" - you may consider to add some memory here
eaccelerator.compress="1" - there is no need if you use gzip/deflate in apache.

What is the effect of Autologout vs having active sessions for all users on high configuration site?

The session for an anonymous user is based on the PHP settings: in php.ini, session.gc_maxlifetime = 1440

Eaccelerator is causing a Segmentation fault on our server. When the segmentation fault occurs the site is accessible, only the node pages are not. There is no error logged in the eaccelerator log. Only error logged is a segmentation error in apache:

child pid <22103> exit signal Segmentation fault (11)

On restart the site starts functioning and error is fixed.

The server conf details:
Hardware Component:
2 Seagate \ 146GB:SAS:15K \ ST3146855SS
4 Generic \ 2048 MB \ DDR2 667 ECC
1 Intel \ 2.13 GHz 1066FSB - Kentsfield \ Xeon 3210 (Quad Core)

Hi folks. I'm looking for some feedback and advise on a odd performance issue.

We have a Drupal 6 multisite that we've developed that currently houses 7 sites. The sites themselves are, as Drupal sites go, not particularly exciting or complex. It even has fewer modules than most sites we build. (All of the sites use the same modules and configuration.)

On our development server, performance is fine.

On the client's test server, performance is fine.

I am currently trying to convince the bigwigs at my company that Drupal is the way to go to provide us with a CMS solution for our clients' sites. We set up new sites for clients all the time and manage and host them. Right now we are running on IIS utilizing a CMS written in classic ASP that cost the company some umpteen gaziliions of dollars to make.

Hi,
I have a question about scalability issue of buddylist. Suppose I have a site with 1 million or more users and admin(or any user) have near about 200000 buddies or more. My question is can buddylist module handle the access of buddylist of admin specially when buddylist calls the function buddylist_get_buddies and execute a query :

$sql = 'SELECT b.buddy, u.name, u.mail, u.uid FROM {buddylist} b
INNER JOIN {users} u ON b.buddy = u.uid
WHERE b.uid = %d';

Hello Everyone,

I have a website with 100k Visitors/Day , I am a newbie in Php and Servers, when a downtime occurs managed hosting guy was resetting the server and I was optimizing the databases and I was optimizing the DB everyday and I was fine.

But this week site became to crash twice a day. And Reboot/optimize only works for 5-6 hrs and it crashes again. My hosting guy said it is caused by cache_page table in mySQL and it gives ~300MiB overhead in 2hrs after mysql optimization. And he said its a wrong method to keep cache in database it causes this errors.

I setup a MySQL 5.0 Cluster on a spare server with multiple Xen VMs and discovered that Drupal 6 (and most likely D7 as wel) as it exists today cannot function properly under the MySQL NDBCLUSTER engine. It seems that under the NDBCLUSTER engine (also known as NDB), D6 has a table called menu_router that fails to be converted.

I posted a bug report about it under the D6 platform as it is rather detailed, but I have an idea this will affect D7 as well.
http://drupal.org/node/391130

Authcache offers page caching for logged-in authenticated users. This allows Drupal to load pages in 1-2 milliseconds and take the load off the the database & processor. Please help test this module (the site I was creating this for got axed, but I continued development on this module). Feedback & patches are welcome. This module can be used for anonymous users (it's faster than Drupal core since the database won't be hit), supports the statistics module, and can cache blocks of user-customized content.

Many thanks to Steve Rude for his work on the Cache Router module, which Authcache uses as its caching system.

This is how it works:

I have created a low memory version of system.module for Drupal 6.10 which can be found here. I found it gave a modest performance improvement with low load performance testing on my development machine. Where it shines is under high load relative to tight memory conditions.

With ab -c 100 -n 1000 http://example.com/user on a real site, applying this patch gave somewhere between 18 and 40% reduction in average time per request.

Unlike a lot of the conversations here, my question is not about a single high-traffic site, but dozens of small to medium traffic sites. I run a small web design company which hosts approx. 50 low-traffic sites on a single dedicated server. We are working to increase this to 100+ in the next year. Some are Drupal 5, but most are Drupal 6. Most sites are also on multi-site configurations, with around 6 different Drupal instances running the 50 sites.

I'm new to the High Performance group and I'm interested to find out if people here have run a site with the following stats:

• 4,000 simultaneous visitors, up to 500 who are logged in at once
• approaching 2 million comments
• 200,000 forum posts
• 30,000 registered users

In my never-ending quest for greater Drupal Glory, I've been spending the past year boning up on the various ways to improve site performance and address issues of scalability. Today, doing some noodling with Amazon EC2 instances (and remaining unconvinced about their raw performance as potential master database servers) I had a thought:

What would it take to extend the static node cache in node_load() beyond the individual drupal bootstrap?

Like just about everyone else, I've been loving how much memcached helps speed site performance. It simply rocks, and everyone looking to reduce server load and speed page responses should be looking into it. One of the better things about it is that it can store and return data objects natively, meaning not only are you letting PHP pull something out of a lightning-fast memory cloud, you also don't incur the CPU overhead of having to unserialize() a string into an object or array.

This let me to my thought. If you want a massively scalable interactive drupal site, you need ready access to tons of nodes. Inevitably you will hit the wall with logged-in requests for these from your database. But what if we were able to take the performance boost we get from node_load()'s static cache, and make it work persistently across an entire site, rather than just for one pageload?

While investigating a large descrepancy between query time ( 243 queries taking roughly 37 milliseconds) and total page execution time ~390 millisecond, I looked at various filters I have installed. I noticed video filter module added a significant overhead and by disabling it was abale to reduce the page execution time to ~212 milliseconds. I also opened an issue for the module here http://drupal.org/node/375415 and you can see more details about my test.

I already use APC.

Are there other filters or modules that you aware could add significant overhead to page execution time?

Hi

First, excuse me if theres already some solution for.
I've surfed drupal.org and also the groups but didn't find something...

What I've to do is building a "stack" like Acquia but with advanced functionality.

For this I'll need to grab some already existing Modules and change some of their functions to work "cleaner" with other or mine, too. Because of this it would make updating Core and also Modules a horrobale job I'm looking for a standardize solution to let Modules communicate with each other without bigger modifying.