LAMP forms the base of most web applications. As the load on an server increases, the bottlenecks in the underlying infrastructure become more apparent in the form of slow response to user requests.
To overcome this slow response the primary choice of most people is to add more hardware resources ( incase of AWS increasing the instance type). This will definitely increases performance but will cost you more money. The webserver and database eat most of the resources. Most commonly used web server is apache and database is MySQL. So if we can optimize these two we can improve the performance.
Apache optimization techniques can often provide significant acceleration boosts even when other acceleration techniques are in use, such as a CDN. mod_pagespeed is a module from Google for Apache HTTP Servers that can improve the page load times of your website. you can read more on this from here. If you want to deploy a PHP app on AWS Cloud, Its better to using some kind of caching mechanism. Its already discussed in our blog .
Once we came into a situation where we have to use a micro instance for a web server with less than 500 hits a day
When the site started running live, and we feel like disappointed. when accessing website, it would sometimes pause for several seconds before serving the requested page. It took hours to figure out what was going on. finally we run the command top and quickly discovered that when the site was accessing by certain amount of users the CPU would spike, but the spike was not the typical user or system CPU. For testing what’s happening in server we used the apache benchmark tool ‘ab’ and run the following command on localhost.
#ab -n 100 -c 10 http://mywebserver.com/
This will show how fast our web server can handle 100 requests, with a maximum of 10 requests running concurrently. In the meantime we were monitoring the output of top command on web server.
For further investigation we started with sar – Linux command to Collect, report, or save system activity information
#sar 1
According to amazon documentation “Micro instances (t1.micro) provide a small amount of consistent CPU resources and allow you to increase CPU capacity in short bursts when additional cycles are available”.
If you use 100% CPU for more than a few minutes, Amazon will “steal” CPU time from the instance, meaning that they throttle your instance. This last as long as five minutes, and then you get a few seconds of 100% again, then the restrictions are back. This will effect your website, making it slow, and even timing-out requests. basically means the physical hardware is busy and the hypervisor can’t give the VM the amount of CPU cycles it wants.
Real tuning required on prefork. This is where we can tell apache to only generate so many processes. The defaults values are high, and which cant be handled by micro instance. Suppose you get 10 concurrent requests for a php page and require around 64MB of RAM when requested (you have to make sure that php memory_limit is above that value). That’s around 640MB of RAM on micro instance of 613MB RAM. This is the case with 10 connections – apache is configured to allow 256 clients by default, We need to scale these down , normally with 10-12 MaxClients. As per out case, this is still a huge number because 10-12 concurrent connections would use all our memory. If you want to be really cautious, make sure that your max memory usage is less than 613MB. Something like 64M php memory limit and 8 max clients keeps you under your limit with space to spare – this helps ensure that our MySQL process when your server is under load.
Maxclients an important tuning parameter regarding the performance of the Apache web server. We can calculate the value of this for a t1.micro instance
Theoretically,
MaxClients =(Total Memory – Operating System Memory – MySQL memory) / Size Per Apache process.
t1.micro have a server with 613MB of Total memory. Suppose We are using RDS instead of mysql server.
Stop apache and run
#ps aux | awk ‘{sum1 +=$4}; END {print sum1}’.
we will get the amount of memory thats used by processes other than apache.
Suppose we get a value around 30.
from top command we can check the average memory that each apache resources use.
suppose its 60mb.
Max clients = (613 – 30 ) 60 = 9.71 ~ 10 approx …
Micro instances are awesome, especially when cost becomes a major concern, however that they are not right for all applications. A simple website with only a few hundreds hits a day will do just fine since it will only need CPU in short bursts.
For Servers that serves dynamic content, better approach is to employ a reverse-proxy. This would be done this apache’s mod_proxy or Squid. The main advantages of this configurations are content caching, load balancing etc. Easy method is to use mod_proxy and the ProxyPass directive to pass content to another server. mod_proxy supports a degree of caching that can offer a significant performance boost. But another advantage is that since the proxy server and the web server are likely to have a very fast interconnect, the web server can quickly serve up large content, freeing up a apache process, why the proxy slowly feeds out the content to clients
If you are using ubuntu, you can enable module by
#a2enmod proxy
#a2enmod proxy_http
and in apache2.conf
ProxyPass / http://192.168.1.46/
ProxyPassReverse / http://192.168.1.46/
The ProxyPassreverse directive captures the responses from the web server and masks the URL as it would be directly responded by the Apache hiding the identity/location of the web server. This is a good security practice, since the attacker won’t be able to know the ip of our web server.
Caching with Apache2 is another important consideration. We can configure apache to set the Expires HTTP header, max-age directive of the Cache-Control HTTP header of static files ,such as images, CSS and JS files, to a date in the future so that these files will be cached by your visitors browsers. This saves bandwidth and makes web site appear faster if a user visits your site for a second time, static files will be fetched from the browser cache
#a2enmod expires
edit /etc/apache2/sites-available/default
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/gif “access plus 4 weeks”
ExpiresByType image/jpg “access plus 4 weeks”
</IfModule>
This would tell browsers to cache .jpg, .gif files for four week.
If your server requires a large amount of read / write operations, you might consider provisioned IOPS ebs volumes on your server. This is really effective if you use database server on ec2 instances. we can use iostat on the command line to take a look at your read/sec and write/sec. You can also use CloudWatch metrics to determine read and write operations.
Once we move to the security side of apache, our major concern is DDos attacks. If a server is under a DDoS attack, it is quite difficult to detect the attack before the damage is done. Attack packets usually have spoofed source IP addresses. Hence, it is more difficult to trace them back to their real source. The limit on the number of simultaneous requests that will be served by Apache is decided by the MaxClients directive, and is set to safe limit, by default. Any connection attempts over this limit will normally be queued up.
If you want to protect your apache against DOS, DDOS attacks use mod_evasive module. This module is designed specifically as a remedy for Apache DoS attacks. This module will allow you to specify a maximum number of requests executed by the same IP address. If the limit is reached, the IP address is blacklisted for the time period you specify.
There are many sites which is built in PHP. PHP provides the power to simply ‘pull’ content from an external source. it could just as easily be an MySQL database or an XML file etc.
The downside to this is processing time, each request for one page can trigger multiple database queries, processing of the output, and formatting it for display. This can be quite slow on complex sites (or slower servers). Dynamic sites probably have very little changing content, this page will almost never be updated after the day it is written. Each time someone requests it the scripts goes and fetches the content, applies various functions and filters to it, then outputs it to you
This is where caching can help us out, instead of regenerating the page every time, the scripts running this site generate it the first time they’re asked to, then store a copy of what they send back to your browser. The next time a visitor requests the same page, the script will know it’d already generated one recently, and simply send that to the browser without all the hassle of re-running database queries or searches.
Different Caching mechanism are discussed below.
APC stands for Alternative PHP Cache, and is a free and open opcode cache for PHP. It provides a robust framework for caching and optimizing PHP performance. APC also provides a user cache for storing application data. APC for caches that do not change often and will not grow too big to avoid fragmentation. The default setting of APC will allow you to store 32 MiB for the opcode cache and the user cache combined
Installing apc on ubuntu
#apt-get install php-apc
edit apc.ini ; default location on new php5 is –> /etc/php5/conf.d/20-apc.ini
extension = apc.so; uncomment this line apc.shm_segments=1; ( by default its enabled .. give 0 to disable)
you can customize your values here. for getting the default values install php5-cli and from command line run
#php -i | grep apc
For monitoring apc cache hits and miss, apc providing a php script. which is located at /usr/share/doc/php-apc/apc.php. Copy this file to your document root and you will be able to monitor your apc status.
for performance benchmarking we created two php files
test1.php
<?php $start = microtime(true); for ($i = 0; $i < 500000; $i++) { include('test_include.php'); } $end = microtime(true); echo "Start: " . $start . "<br />"; echo "End: " . $end . "<br />"; echo "Diff: ". ($end-$start) . "<br />"; ?>
test2.php
<?php $t = "migrate2cloud"; ?>
Without apc…
Start: 1360937874.8965
End: 1360937883.1506
Diff: 8.2541239261627
With apc..
Start: 1360937935.5746
End: 1360937936.7291
Diff: 1.1545231342316
without apc it took 8 seconds to complete the request .. with apc.. 1.15 seconds..
Memcached system uses a client–server architecture. The servers maintain a key–value associative array; the clients populate this array and query it. Keys are up to 250 bytes long and values can be at most 1 megabyte in size. Clients use client-side libraries to contact the servers which, by default, expose their service at port 11211. Amazon provides a Service called Amazon elasticache for memcache through which we can configure memcache clusters for caching purposes.
installation and configuration
apt-get install memcached
apt-get install php5-memcached
enable memcache module in /etc/php5/apache2/conf.d/20-memcached.ini or in php.ini
edit php.ini
session.save_handler = memcached
extension=memcache.so
extension=memcached.so
Restart apache and memcache..
php script used for memcache testing..
<?php //memcached simple test $memcache = new Memcache; $memcache->connect('localhost', 11211) or die ("Could not connect"); $key = md5('42data'); //something unique for ($k=0; $k<5; $k++) { $data = $memcache->get($key); if ($data == NULL) { $data = array(); //generate an array of random shit echo "expensive query"; for ($i=0; $i<100; $i++) { for ($j=0; $j<10; $j++) { $data[$i][$j] = 42; //who cares } } $memcache->set($key,$data,0,3600); } else { echo "cached"; } }
You can monitor memcache using phpmemcacheadmin
http://code.google.com/p/phpmemcacheadmin/
Varnish has a concept of “backend” or “origin” servers. A backend server is the server providing the content Varnish will accelerate. Our first task is to tell Varnish where it can find its content. open the varnish default configuration file. Iif you installed from a package it is probably /etc/varnish/default.vcl.
Somewhere in the top there will be a section that looks a bit like this.:
backend default { .host = "127.0.0.1"; .port = "80"; }
Change the port number to your apache ( or whatever the webserver you are using) port number.
this piece of configuration defines a backend in Varnish called default. When Varnish needs to get content from this backend it will connect to port 80 on localhost (127.0.0.1).
# varnishd -f /etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:80
The -f options specifies what configuration varnishd should use.
The -s options chooses the storage type Varnish should use for storing its content
-T 127.0.0.1:2000 — Varnish has a built-in text-based administration interface
-a 0.0.0.0:80 — specify that I want Varnish to listen on port 80
For logging varnish — In terminal window you started varnish type varnishlog
When someone accessing your page you will get log like
#varnishlog 11 SessionOpen c 127.0.0.1 58912 0.0.0.0:80 11 ReqStart c 127.0.0.1 58912 595005213 11 RxRequest c GET 11 RxURL c / 11 RxProtocol c HTTP/1.1 11 RxHeader c Host: localhost:80 11 RxHeader c Connection: keep-alive
Caching should not be used for some things like search results, forums etc… where the content has to be upto the times and changes depending on user’s input. It’s also advisable to avoid using this method for things like a Flash news page, in general dont use it on any page that you wouldn’t want the end users browser or proxy to cache.
Mongo query language
The Ubuntu package management tool (i.e. dpkg and apt) ensure package consistency and authenticity by requiring that distributors sign packages with GPG keys. Issue the following command to import the 10gen public GPG Key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
Create a /etc/apt/sources.list.d/10gen.list file and include the following line for the 10gen repository.
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
Now issue the following command to reload your repository:
sudo apt-get update
Issue the following command to install the latest stable version of MongoDB:
sudo apt-get install mongodb-10gen
When this command completes, you have successfully installed MongoDB! Continue for configuration and start-up suggestions.
These packages configure MongoDB using the /etc/mongodb.conf file in conjunction with the control script. You will find the control script is at /etc/init.d/mongodb.
This MongoDB instance will store its data files in the /var/lib/mongodb and its log files in /var/log/mongodb, and run using the mongodb user account.
Note
If you change the user that runs the MongoDB process, you will need to modify the access control rights to the /var/lib/mongodb and /var/log/mongodb directories.
You can start the mongod process by issuing the following command:
sudo service mongodb start
You can verify that mongod has started successfully by checking the contents of the log file at /var/log/mongodb/mongodb.log.
As needed, you may stop the mongod process by issuing the following command:
sudo service mongodb stop
You may restart the mongod process by issuing the following command:
sudo service mongodb restart
As of the current release, there are no control scripts for mongos. mongos is only used in sharding deployments and typically do not run on the same systems where mongod runs. You can use the mongodb script referenced above to derive your own mongos control script.
Using MongoDB
Among the tools included with the MongoDB package, is the mongo shell. You can connect to your MongoDB instance by issuing the following command at the system prompt:
mongo
> show dbs (); —> To show your databases
> use <databasename> —-> To switch database
> db.createCollection(“collectionname”) —> To create collection
> db.collectionname.find(); —> To see the contents in the collection
> db.addUser(“theadmin”, “anadminpassword”) —> To create user and password
To monitor database system we can use Mongotop
Mongotop tracks and reports the current read and write activity of a MongoDB instance.
Mongotop provides per-collection visibility into use.
Use mongotop to verify that activity and use match expectations.
Mongotop returns time values specified in milliseconds (ms.)
Mongotop only reports active namespaces or databases, depending on the –locks option.
If you don’t see a database or collection, it has received no recent activity.
By default mongotop connects to the MongoDB instance running on the localhost port 27017. However,mongotop can optionally connect to remote mongod instances
Next, we can use Mongostat
Mongostat captures and returns counters of database operations. Mongostat reports operations on a per-type (e.g. insert, query, update, delete, etc.) basis. This format makes it easy to understand the distribution of load on the server. Use Mongostat to understand the distribution of operation types and to inform capacity planning.
The Mongostat utility provides a quick overview of the status of a currently running mongod or Mongos instance. Mongostat is functionally similar to the UNIX/Linux file system utility vmstat, but provides data regarding mongod and Mongos instances.
Use db.serverStatus()
It provides an overview of the database process’s state.
Then REST interface
MongoDB provides a REST interface that exposes a diagnostic and monitoring information in a simple web page. Enable this by setting rest to true, and access this page via the local host interface using the port numbered 1000 more than that the database port. In default configurations the REST interface is accessible on 28017. For example, to access the REST interface on a locally running mongod instance: http://localhost:28017
These are a few basic tips on making your application better/faster/stronger without knowing anything about indexes or sharding.
Connecting
Connecting to the database is a (relatively) expensive operation. Try to minimize the number of times you connect and disconnect: use persistent connections or connection pooling (depending on your language).
there are some side effects with the PHP connection code.
$connection = new Mongo ( );
$connection->connect( );
In this code it appears the user wants to create a new connection. However, under the hood the following is happening:
The constructor connects to the database.
connect( ) sees that you’re already connected, assumes you want to reset the connection.
Disconnects from the database.
Connects again.
The result is that you have doubled your execution time.
ObjectIds
ObjectIds seem to be uncomfortable, so they convert their ObjectIds into strings. The problem is, an ObjectId takes up 12 bytes but its string representation takes up 29 bytes (almost two and a half times bigger).
Numbers vs. Strings
MongoDB is type-sensitive and it’s important to use the correct type: numbers for numeric values and strings for strings.
If you have large numbers and you save them as strings (“1234567890″ instead of 1234567890), MongoDB may slow down as it strcmps the entire length of the number instead of doing a quicker numeric comparison. Also, “12″ is going to be sorted as less than “9″, because MongoDB will use string, not numeric, comparison on the values. This can lead to some errors.
Driver-specific
Find out if you’re driver is particularly weaknesses (or strengths). For instance, the Perl driver is one of the fastest drivers, but it is not good at decoding Date types (Perl’s DateTime objects take a long time to create).
MongoDB adopts a documented-oriented format, so it is more similar to RDBMS than a key-value or column oriented format.
MongoDB operates on a memory base and places high performance above data scalability.Mongo DB uses BSON for data storage
Mongo uses memory mapped files, which means that a lot of the memory reported by tools such as top may not actually represent RAM usage. Check mem[“resident”], which tells you how much RAM Mongo is actually using.
“mem” : {
“resident” : 2,
“virtual” : 2396,
“supported” : true,
“mapped” : 0
},
Backup
There are basically two approaches to backing up a Mongo database:
Mongodump and Mongorestore are the classic approach. Dumps the contents of the database to files. The backup is stored in the same format as Mongo uses internally, so is very efficient. But it’s not a point-in-time snapshot.
To get a point-in-time snapshot, shut the database down, copy the disk files (e.g. with cp) and then start mongod up again. Alternatively, rather than shutting mongod down before making your point-in-time snapshot, you could just stop it from accepting writes:
> db._adminCommand({fsync: 1, lock: 1})
{
“info” : “now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock”,
“ok” : 1
}
To unlock the database again, you need to switch to the admin database and then unlock it
> use admin
switched to db admin
> db.$cmd.sys.unlock.findOne()
{ “ok” : 1, “info” : “unlock requested” }
Replication
Start your master and slave up like this:
$ mongod –master –oplogSize 500
$ mongod –slave –source localhost:27017 –port 3000 –dbpath /data/slave
When seeding a new slave server from master use the –fastsync option.
You can see what’s going on with these two commands:
> db.printReplicationInfo() # tells you how long your oplog will last
> db.printSlaveReplicationInfo() # tells you how far behind the slave is
If the slave isn’t keeping up,Check the mongo log for any recent errors. Try connecting with the mongo
console. Try running queries from the console to see if everything is working. Run the status commands
above to try and find out which database is taking up resources.
Timeout
Connection timeout in milliseconds. Defaults to 20000
Connection::query_timeout.
How many milliseconds to wait for a response from the server. Set to 30000 (30 seconds) by default. -1 waits forever (or until TCP times out, which is usually a long time).
Default pool
The default pool has a maximum of 10 connections per mongodb host. This value is controlled by the variable “connectionsPerHost” within the class
MongoDB Server Connections
The MongoDB server has a property called “maxConns” that is the max number of simultaneous connections. The
default number for maxConns is 80% of the available file descriptors for connections. One way to check the number of connections is by opening the mongo shell and executing:
>db.serverStatus() and in the previous mail I have send the screen shot of this.
The standard format of the MongoDB connection URI used to connect to a MongoDB database server.
mongodb://[username:password@]host1[:port1][,host2[:port2],…[,hostN[:portN]]][/[database][?options]]
Finding the Min and Max values in MongoDB
In MongoDB, the min() and max() functions work as limitors – essentially the same as “gte” (>=) and “lt” (<).
To find the highest (maximum) value in MongoDB, you can use this command;
db.thiscollection.find().sort({“thisfieldname”:-1}).limit(1)
This essentially sorts the data by the fieldname in decending and takes the first value.
The lowest (minimum) value can be determined in a similar way.
db.thiscollection.find().sort({“thisfieldname”:1}).limit(1)
Memory Mapped Storage Engine :-
This is the current storage engine for MongoDB, and it uses memory-mapped files for all disk I/O. Using this strategy, the operating system’s virtual memory manager is in charge of caching. This has several implications:
There is no redundancy between file system cache and database cache: they are one and the same.
MongoDB can use all free memory on the server for cache space automatically without any configuration of a cache size.
Virtual memory size and resident size will appear to be very large for the mongod process.
This is benign: virtual memory space will be just larger thanthe size of the datafiles open and mapped; resident size will vary depending on the amount of memory not used by other processes on the machine.
This command shows the memory usage information :- db.serverStatus().mem
For example :-
> db.serverStatus().mem
{
“bits” : 64,
“resident” : 31,
“virtual” : 146,
“supported” : true,
“mapped” : 0,
“mappedWithJournal” : 0
}
We can verify there is no memory leak in the mongod process by comparing the mem.virtual and mem.mapped values (these values are in megabytes). If you are running with journaling disabled, the difference should be relatively small compared to total RAM on the machine. If you are running with journaling enabled, compare mem.virtual to 2*mem.mapped. Also watch the delta over time; if it is increasing consistently, that could indicate a leak.
Also we can use to check what percent of memory is being used for memory mapped files by the free command:
Here 2652mb of memory is being used to memory map files
root@manager-desktop:~# free -tm
total used free shared buffers cached
Mem: 3962 3602 359 0 411 2652
-/+ buffers/cache: 538 3423
Swap: 1491 52 1439
Total: 5454 3655 1799
Garbage collection handling :-
When we remove an object from MongoDB collection, the space it occupied is not automatically garbage collected and new records are only appended to the end of data files, making them grow bigger and bigger.MongoDB maintains lists of deleted blocks within the datafiles when objects or collections are deleted. This space is reused by MongoDB but never freed to the operating system.
To shrink the amount of physical space used by the datafiles themselves, by reclaiming deleted blocks, we must rebuild the database by using the command “db.repairDatabase( )” . repairDatabase copies all the database records to new files.
We will need enough free disk space to hold both the old and new database files while the repair is running, the repairDatabase will take a long time to complete.Also rather than compacting an entire database,
you can compact just a single collection by using “db.runCommand({compact:’collectionmname;})”
This does not shrink any datafiles,however; it only defragments deleted space so that larger objects might reuse it.
The compact command will never delete or shrink database files, and in general requires extra space to do its work.
Thus, it is not a good option when you are running critically low on disk space.
Memached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load and session management. Lets focus on session management first and build up a caching daemon to store PHP sessions in a load balanced environment. In this post I will explain how you can easily install it and make it available in LAMP. Read more…
