Archive for the ‘Linux’ category

Optimizing your web server: Part 4b – XDebug Profiler

November 23rd, 2009

Here we are, part 4b. By using XDebug, a pecl extension, our php code can be analyzed pinpointing the slower functions.

Installing XDebug is easy:

pecl install xdebug

Now, you’ll have to enable it in the php.ini file:

extension=xdebug.so
xdebug.profiler_enable=1
xdebug.profiler_output_name=profiler-%s-%t.cache
xdebug.profiler_output_dir=/tmp/xdebug

Don’t forget to create the directory and give apache enough permissions to write in it.

So after restarting apache you’ll see new files showing up under ‘/tmp/xdebug/’.
These are, as the XDebug documentation says ‘cachegrind compatible files’.

As I’m using putty on windows to connect to my webserver and the webserver does not run a GUI, I’ll be using wincachegrind to display the information.

To make it easy for myself, I use winscp and make it so that the cachegrind files are opened with WinCacheGrind.
Setting this up goes beyond the scope of this post and greatly depends on your setup/way of working.

Anyway, up till now we managed to install XDebug, enabling the extension, generate the cachegrind files and now opening them with WinCacheGrind.

Upon opening such a file in wingrind, you’ll see something similar to this:

main(click to enlarge)

As this blog post was made while I was profiling one of my own sites, I’ll use a real world example.
In the right hand side, the {main} takes 89ms to process.

Let us expand this to see what function is taking the most time.
functions

When looking at the functons that take the most time. I noticed, as you can see in the screenshot above, the Component->startup and the beforeFilter both take 13ms.
The component is the cakephp cookie class, it seems the startup of this class takes 12ms. To be exact, it’s the decrypting of the cookie that takes 11ms.
The same is true for the beforeFilter, in that method I read the cookie, taking 12ms, because it decrypts again.

cookie

After seeing this, I changed the code so that on the first hit on the site the information in the cookie is is copied into the session, as this was super fast(less then 1ms). Saving me 2 times 12ms.

That comes down to 27% faster!

To put things even more in context. This was a base class, all my pages inherited from that class, so it saved me those 24ms on EACH request.

Imagine doing that to all pages, or one of those heavy visited pages on your own site.
Worth the 10 minutes it takes to install XDebug with WinCacheGrind, no?

Optimizing your web server: Part 1 – Gzip
Optimizing your web server: Part 2 – Keep Alives
Optimizing your web server: Part 3 – Opcode Caching
Optimizing your web server: Part 4a – PHP
Optimizing your web server: Part 4b – XDebug Profiler

No comments »

Posted in General, Linux, PHP

Tags:

Create a Kickstart netinstall with CentOS

November 22nd, 2009

After looking on the internet after a good kickstart guide, there was not one that helped me set up everything from start to end.

I’m not one that likes to read allot. So after 10minutes together with Google the road to ‘the perfect kickstart’ started.

This is what the target was:
- Mount a small iso, about 10mb big
- Automatically start the installation
- Use all the settings that are predefined (disk/language/…)
- Install the packages, the target was a basic install + some extras
- Do some post install things

The test environment was two virtual machines, one kickstart server, the place where the kickstart config, rpm’s and post install scripts are located and a second virtual machine that will do the kickstart.

To start the installation an iso file is needed that will start using the kickstart file located on the server.
I used the CentOS 5.3 i386 net install ISO as a basis for this.
So download it from somewhere, for example:

cd /tmp
wget http://ftp.belnet.be/packages/centos/5.3/isos/i386/CentOS-5.3-i386-netinstall.iso

Now mount it by doing something similar to:

mkdir -p /cd/
mount -o loop -t iso9660 /tmp/CentOS-5.3-i386-netinstall.iso /cd/

Now copy the contents to a new location where the new iso file will be created:

mkdir –p /tmp/centos5.3_kickstart
cp –R /cd/* /tmp/centos5.3_kickstart

Now the isolinux.cfg file needs to be modified to start the installation with our kickstart config, this is the one I use:

default linux
prompt 0
timeout 1
 
display boot.msg
F1 boot.msg
F2 options.msg
F3 general.msg
F4 param.msg
F5 rescue.msg
 
label linux
  menu label Crazy's Manga CentOS 5.3 Kickstart(32bit)
  menu default
  kernel vmlinuz
  append initrd=initrd.img ks=http://kickstart.crazytje.be/CentOS-5.3_i386-Kickstart.cfg text

Be setting the prompt to ‘0′ and timeout to ‘1′ the install is started right away.
The last line makes it so that the installation is started with the kickstart file located on the network:

append initrd=initrd.img ks=http://kickstart.crazytje.be/CentOS-5.3_i386-Kickstart.cfg text

Make sure this config is accessible for the machine you are kickstarting, so replace the part after ‘ks=’ to the location of your own kickstart config.

Once edited, the new ISO can be created, this is done with the following command:

mkisofs -o CentOS-5.3_i386-Kickstart.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -v -T .

Now that we have the ISO, the next step is the kickstart server and kickstart config.

The ISO is only needed to start the installation and get the kickstart file. Everything after that does not require any intervention of the ISO. This makes it easy to manage a large number of kickstarts. It’s not needed to notify everyone to use a new kickstart ISO, just change the config on the kickstart server and anyone that kickstarts will get the new set of instructions. Anyone with vi can do it :)

So let’s start setting up our kickstart server, first we need the rpm’s.
Because we’re going for a minimal install it’s not needed to copy or download all the rpm’s. The first CentOS 5.3 CD will be enough.
Other possibilities are using the dvd or rsync to get the latest version of a repository.

In this case, I downloaded this one:

cd /tmp
wget http://ftp.belnet.be/packages/centos/5.3/isos/i386/CentOS-5.3-i386-bin-1of6.iso

Now mount the ISO:

umount /cd
cd /tmp
mount -o loop -t iso9660 CentOS-5.3-i386-bin-1of6.iso /cd/

Create a world accessible directory, for example a web server and copy over the images and rpm directories:

mkdir –p /var/www/html/kickstart/centos/5.3/i386/
cp –R /cd/CentOS /var/www/html/kickstart/5.3/i386/
cp –R /cd/images /var/www/html/kickstart/5.3/i386/
cp /cd/repodata/comps.xml /var/www/html/kickstart/5.3/i386/

Now that we have the rpm’s and images, lets create the kickstart config file, I’ll give you mine as example:

## i386 kickstart config  installation
key d6dbc277d6578dc2
# Install OS instead of upgrade
install
# Use CDROM installation media
cdrom
# Network information
network --device eth0 --bootproto dhcp
# Repository Location
url --url=http://kickstart.crazytje.be/centos/5.3/i386/
#Root password
rootpw --iscrypted $1$rB3lJzDT$PmMFJJiLgVPbnvJF6UZ1x/
# Firewall configuration
firewall --disabled
# SELinux configuration
selinux --disabled
# System authorization information
authconfig --enableshadow --enablemd5
### Language Specification
lang en_US
langsupport en_US
# System keyboard
keyboard us
# System timezone
timezone --utc America/Chicago
# System bootloader configuration
bootloader --location=mbr --driveorder=sda
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part /boot --fstype ext3 --size=200
part pv.1 --size=2000 --grow
volgroup VolGroup00 --pesize=32768 pv.1
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=384 --grow --maxsize=768
logvol / --fstype ext3 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
# Run the Setup Agent on first boot
firstboot --disable
# Installation logging level
logging info
 
%packages
## groups
@base
@core
-anacron
-autofs
-yum-updatesd
-wireless-tools
-irda-utils
-nfs-utils
-NetworkManager
 
## editors
vim-enhanced
vim-common
 
## Webserver
httpd
httpd-devel
 
# PHP
php
php-devel
php-gd
php-mysql
php-pear
php-xml
php-xmlrpc
php-pecl-Fileinfo
 
#pecl and various
libxml2-devel
libxml2
perl-libwww-perl
 
#mysql
mysql
mysql-devel
mysql-server
 
#perl
perl-HTML-Parser
perl-DBI
perl-Net-DNS
perl-Digest-SHA1
 
# Mail
fetchmail
 
## Tools
#compilers
gcc
gcc-c++
 
#web
curl
curl-devel
wget
openssl
 
#compression
bzip2
unzip
zip
 
#other
ntp
 
#development
#subversion
phpmyadmin
 
%post
chvt 3
cd /tmp
wget http://kickstart.crazytje.be/centos/5.3/i386/postinstall.sh
bash postinstall.sh > /root/postinstall.log
chvt 1

The kickstart configuration file has a couple of sections.
The first are the settings used by the installer, that includes network configuration, where it can find the repository, firewall configuration, language settings and partition information.
Make sure to change the “url –url=http://kickstart.crazytje.be/centos/5.3/i386/” to your own repository location.
The root password in this config is ‘today12′. It’s passed encrypted with “rootpw –iscrypt”.

The second part is a list of packages the installer will install.
I include two groups, core and base, minus some small packages. Below that you’ll see allot of tools specific for a web server. Because this is was intended to be a test server for building websites.

For each package you add to the list, make sure to include it in the rpm’s directory ‘/var/www/html/kickstart/centos/5.3/CentOS/’.
The first cd downloaded in the beginning does not contain most of these.

The last part, is the post install. Most of the time this will be a script that is downloaded onto the machine. In the case of a web server, copy over a tar with php and apache settings for example. When you do not want a post install script, just delete everything below the %post.

Once your kickstart config is done and all your packages have been added to the CentOS(rpm) directory, we’ll run a test to check all of the dependencies:

mkdir –p /tmp/testdb
rpm --test --dbpath /tmp/testdb/ -Uvh /var/www/html/kickstart/centos/5.3/i386/CentOS/*.rpm

Most likely more rpm’s will have to be added. Keep adding them until there are no more dependencies missing.

Once the test succeeded, we can create a repository:

cd /var/www/html/kickstart/centos/5.3/i386/
createrepo -g comps.xml /var/www/html/kickstart/centos/5.3/i386/

That was it!, at least to start off.
When deploying for a larger group(this was only for myself), it might come in handy to add your own repository to yum(/etc/yum.repos.d) and copy over all the rpm’s from another up to date repository instead of a cd. That way the rpm’s are the latest version.

After this, mount the ISO, sit back, and enjoy the ride :)

No comments »

Posted in Bash, Linux

Tags:

Single Instance PHP Script

November 12th, 2009

When using php scripts on the command line or in the cron, there are often times that you do not want to allow the same script to run more then once at a the same time.

So a single instance application in php.

Personally I use php scripts like this allot. My cron is filled with all kind of tasks that run from the cron.
From backups to just database cleanup scripts and many other.

But most of those scripts will cause problems when running it multiple times.

This small php class has helped me in allot of cases to prevent just that:

class Process {
    private $strPIDFile;
 
    function __construct($pProcessIdFile) {
        $this->strPIDFile = $pProcessIdFile;
        if(file_exists($this->strPIDFile)) {
            if(!is_writable($this->strPIDFile)) {
                throw new Exception('File Not Writable', 101);
            }
 
            $pid = trim(file_get_contents($this->strPIDFile));
            if(posix_kill($pid, 0)) {
                if($this->is_alive($pid)) {
                    //process is alive
                    throw new Exception('Process Already Running', 100);
                } else {
                    //cleanup
                    unlink($this->strPIDFile);
                }
            }
        } else {
            if(!is_writable(dirname($this->strPIDFile))) {
                throw new Exception('Directory Is Not Writeable', 102);
            }
        }
 
        $id = getmypid();
        file_put_contents($this->strPIDFile, $id);
 
    }
 
    public function __destruct() {
        if(file_exists($this->strPIDFile)) {
            unlink($this->strPIDFile);
        }
    }
 
    private function is_alive($pId){
        exec('ps '.$pId, $ProcessState);
        return(count($ProcessState) >= 2);
    }
}

An example on how to use it:

try {
    $Process = new Process('/tmp/myphpscript.pid');
} catch(Exception $ex) {
    switch($ex->getCode()) {
        case 100:
            echo 'Script already running...';
            return;
        case 101:
            echo 'File Not Writable';
            return;
        case 102:
            echo 'Folder Not Writable';
            return;
    }
}

Many people will say that working with exceptions this way isn’t good practice, it’s easy to modify to your own needs if needed.

Now lets break it in small pieces, well, not that many, the class contains a constructor and deconstructor, nothing more.

First the constructor, it takes the full path to the pid file as a parameter.
This pid file is something used by most applications running on your linux machine.
A couple of checks happen to see if the file and directory are writable, once verified that they are, the fun starts.

If the pid file exists, your php script is already running, that is ‘most of the time’ the case.
The file is left behind if your script crashes, so some extra work is required to make sure the process is still alive or not.
If the process can’t be found, then your php script crashed the last time it ran.

Second, the deconstructor. This does the cleanup when your script finishes. All this does is remove the pid file, meaning it will allow the script to be started again by another process.

That’s it!, nothing more to it.
Copy pasting the class and the small try/catch will make your php script a single instance

No comments »

Posted in Linux, PHP

Tags:

Sqlite and postgres in bash

November 10th, 2009

Recently I needed to query a postgres database and put some of the info in a sqlite database.

At first I wondered how am I going to do that?, but it’s much much easier then you’d think it is.

A small example of a bash script doing just that:

#!/bin/bash
export PGPASSWORD=mypassword
psql="psql -h localhost -U myuser -d mydatabase"
 
sqlite3 sqlitedatabase.db  "CREATE TABLE mytable(Id TEXT, myfield1 TEXT, myfield2 TEXT);"
 
$psql --quiet --no-align --field-separator ' ' -t --c "select id, field1, field2 from mypostgrestable;" | while  read -a Record ;do
  Id=${Record[0]}
  first=${Record[1]}
  second=${Record[2]}
 
  sqlite3 sqlitedatabase.db  "INSERT INTO mytable(Id, myfield1, myfield2) values ('${Id}','${first}','${second}');"
done
 
sqlite3 sqlitedatabase.db  "select * from mytable";

Not much right?
Lets go over it:

First we set our password.
This is done by using the “export PGPASSWORD=mypassword”.
It can also be done in two lines, frist setting the PGPASSWORD and then exporting it.

After this, we create our table.
Note that this creates the database if it does not exist.

Our next, and most important step is to query the postgres database and loop over the result.
This is done by piping(|) the result of the query to the “while read -a Record ;do”. Everything between this and the “done” will be repeated for each record returned by the query.

To read more about using pipes, there is an interesting article you can read here

After we finish the loop, as a check, do a “select *” from the table to see if the inserts worked.

This reminds us again how powerful using bash can be.

No comments »

Posted in Bash, Linux

Tags: