As mentioned in a previous post, I use LVM volumes directly to store the virtual disks for my Virtualbox VMs. This post will guide you through how to access the contents of the virtual disk directly (so that you don’t need to boot the VM). The disk I’m working with is called ‘debian’.

# lvscan | grep debian
  ACTIVE            '/dev/vbox/debian' [5.00 GiB] inherit

We need to create device maps from this LVM device’s partition tables.

# kpartx -av /dev/vbox/debian
add map vbox-debian1 (253:8): 0 9912042 linear /dev/vbox/debian 63
add map vbox-debian2 (253:9): 0 562275 linear /dev/vbox/debian 9912105
add map vbox-debian5 : 0 562212 linear 253:9 9912168

Now we can mount the image and grab any files we may need.

# mkdir foo
# mount /dev/mapper/vbox-debian1 foo/
# ls foo/
bin   cdrom  etc  home        lib    lost+found  mnt  proc  sbin     srv  tmp  var
boot  dev    foo  initrd.img  lib64  media       opt  root  selinux  sys  usr  vmlinuz

Once we are done accessing our files, we can go ahead and unmount the partition and delete the partition mappings.

# umount foo/
# kpartx -d /dev/vbox/debian

I recently spent quite a bit of time installing and configuring request tracker to run on Ubuntu 10.04 using nginx as the web server. The documentation on doing this was scarce (or incorrect), so I thought this would be a good place to centralize all the information needed to replicate my setup.

The first thing to do is install all the required packages. I assume you already know how to do this. For what it’s worth, I’m using a PPA repository for the nginx package since the one available in Ubuntu is extremely out of date. Here are the contents of /etc/apt/sources.list.d/nginx.list.

deb http://ppa.launchpad.net/nginx/stable/ubuntu lucid main
deb-src http://ppa.launchpad.net/nginx/stable/ubuntu lucid main

I have apt configured such that it doesn’t install Recommended or Suggested packages. Therefore, I had to manually install the libcgi-fast-perl package because this will be necessary in order for nginx to run the RT code.

I then had to install the mysql-server package and create the database to be used.

root@rt:~# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 584
Server version: 5.1.41-3ubuntu12.10 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database rtdb;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on rtdb.* to 'rt'@'localhost' identified by 'SECRETPASSWORD';
Query OK, 0 rows affected (0.03 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

Next, I had to modify /etc/request-tracker3.8/RT_SiteConfig.d/50-debconf to suit our custom environment. I also had to reconfigure /etc/request-tracker3.8/RT_SiteConfig.d/51-dbconfig-common to use mysql with the appropriate values for the database that was created.

# THE DATABASE:
# generated by dbconfig-common

# map from dbconfig-common database types to their names as known by RT
my %typemap = (
    mysql   => 'mysql',
    pgsql   => 'Pg',
    sqlite3 => 'SQLite',
);

Set($DatabaseType, $typemap{mysql} || "UNKNOWN");

Set($DatabaseHost, 'localhost');
Set($DatabasePort, '3306');

Set($DatabaseUser , 'rt');
Set($DatabasePassword , 'SECRETPASSWORD');

# SQLite needs a special case, since $DatabaseName must be a full pathname
#my $dbc_dbname = 'rtdb'; if ( "sqlite3" eq "sqlite3" ) { Set ($DatabaseName, '/var/lib/dbconfig-common/sqlite3/request-tracker3.8' . '/' . $dbc_dbname); } else { Set ($DatabaseName, $dbc_dbname); }
Set ($DatabaseName, 'rtdb');

By default, the RT install uses a simple sqlite database. We just switched it to use our mysql database that we created in the previous step. Once that is complete, you need to update the SiteConfig by running update-rt-siteconfig. Then you can move on to configuring nginx.

Here is the nginx configuration that was necessary to get all aspects (as far as I’ve tested) working with RT:

server {
        listen          [::]:80;
        server_name     rt.siriad.com;
        root            /usr/share/request-tracker3.8/html;

        location / {
                index           index.html;
                fastcgi_pass    unix:/var/run/rt/rt.sock;
                include         /etc/nginx/fastcgi_params;
                fastcgi_param   PATH_INFO       $fastcgi_script_name;
        }

        location ~* .+\.(html|js|css)$  {
                index           index.html;
                fastcgi_pass    unix:/var/run/rt/rt.sock;
                include         /etc/nginx/fastcgi_params;
                fastcgi_param   PATH_INFO       $fastcgi_script_name;
        }

        location /NoAuth/images/ {
                alias /usr/share/request-tracker3.8/html/NoAuth/images/;
        }
}

Here is the upstart script located at /etc/init/rt-fcgi.conf:

# rt-fcgi - test
start on runlevel [12345]
stop on runlevel [0]
respawn

env FCGI_SOCKET_PATH=/var/run/rt/rt.sock

exec su www-data -c "/usr/share/request-tracker3.8/libexec/mason_handler.fcgi"

Once all those are in place, the only thing you need to do is service rt-fcgi start and restart nginx. Then you should be able to login using the default RT username/password.

This post discusses the nginx proxy module. I recently setup Nginx as a reverse caching proxy for various sites. Every configuration example I came across online failed to mention using the proxy_cache_key directive. Therefore, I originally ended up with something like this:

# cat /etc/nginx/sites-available/siriad
# You may add here your
# server {
#       ...
# }
# statements for each of your virtual hosts to this file

server {
        listen   [::]:80;
        server_name siriad.com;
        rewrite ^/(.*) http://www.siriad.com/$1 permanent;
}

server {
        listen   [::]:80;
        server_name     www.siriad.com
                        testing.siriad.com;

        access_log      /var/log/nginx/siriad.com/access.log;
        error_log       /var/log/nginx/siriad.com/error.log;

        location / {
                proxy_pass              http://backend;
                proxy_set_header        Host $host;
                proxy_cache             siriad;
                proxy_cache_valid       200 1d;
                proxy_cache_use_stale   error timeout invalid_header updating
                                        http_500 http_502 http_503 http_504;
        }
}

This led to some odd behavior. When I would load www.siriad.com and subsequently load testing.siriad.com, I would end up with the cached content from www.siriad.com for both requests. The cache was working, but was not distinguishing between the two hosts. I spent some time trying different configurations thinking that this was a problem caused by me since I had trouble finding any information on it.

It turns out, this is exactly the use case for the proxy_cache_key directive. By adding the following line, I made sure that the hostname was included in the key used to cache the request so that there were no key collisions during the process.

                proxy_cache_key         "$scheme$host$request_uri";

I was able to find this information after searching around DDG for quite a while. I finally came across this forum post. The result of the above configuration is a working reverse caching proxy using Nginx for siriad.com as well as testing.siriad.com. I am hoping this post is slightly more searchable than the results I was getting while trying to find the answer to this problem.

I recently setup a VPS with Arp Networks. By default they set you up with your own IPv6 block. All you need to do is configure it to meet your needs. I reinstalled the VPS with Ubuntu Lucid (which is not one of their default OSes) so I needed to reconfigure my interface to use one of my assigned v6 addresses. The process is extremely easy. Here is the relevant section of my /etc/network/interfaces file:

iface eth0 inet6 static
        address 2607:f2f8:a230::2
        gateway 2607:f2f8:a230::1
        netmask 64

After restarting networking, I was able to reach the machine via IPv6. You can verify the results of this by testing soljerome.com at http://ipv6-test.com/validate.php. Now that I’m IPv6 ready, I just wish that Comcast would finish their dual-stack rollout so I can use it natively from home. Seeing as how the IANA just allocated the final five /8 blocks of IPv4 address space, I’m hoping that the rollout happens sooner rather than later.

I was recently setting up DBStats for a Bcfg2 installation and was having some serious performance issues when a client was uploading statistics to the server.

hwfwrv003.web.e.uh.edu:probe:current-kernel:['2.6.18-194.26.1.el5']
hwfwrv003.web.e.uh.edu:probe:groups:['group:rpm', 'group:linux', 'group:redhat', 'group:redhat-5Server', 'group:redhat-5', 'group:x86_64']
Generated config for hwfwrv003.web.e.uh.edu in 0.044s
Handled 1 events in 0.000s
Client hwfwrv003.web.e.uh.edu reported state clean
Imported data for hwfwrv003.web.e.uh.edu in 139.942095041 seconds

This is drastically slower than normal. So, I remounted the sqlite database on a ramdisk.

# losetup /dev/loop0 /bcfg2/bcfg2.sqlite
# mount -t ramfs /dev/loop0 /bcfg2/
# mount | grep ramfs
/dev/loop0 on /bcfg2 type ramfs (rw)

Here is the time it took once I moved the sqlite database to a ramdisk.

hwfwrv003.web.e.uh.edu:probe:current-kernel:['2.6.18-194.26.1.el5']
hwfwrv003.web.e.uh.edu:probe:groups:['group:rpm', 'group:linux', 'group:redhat', 'group:redhat-5Server', 'group:redhat-5', 'gr
oup:x86_64']
Generated config for hwfwrv003.web.e.uh.edu in 0.074s
Handled 1 events in 0.000s
Client hwfwrv003.web.e.uh.edu reported state clean
Imported data for hwfwrv003.web.e.uh.edu in 1.16791296005 seconds

That’s faster by a factor of almost 120! As you can see, something is very odd with the performance hit we are taking when using an ext4 filesystem. Just for comparison, I created an ext3 partition to hold the sqlite database.

# mount | grep foo
/dev/loop1 on /foo type ext3 (rw)
# ls /foo/
bcfg2.sqlite

Here is the same client update again when using ext3 to hold the sqlite database.

hwfwrv003.web.e.uh.edu:probe:current-kernel:['2.6.18-194.26.1.el5']
hwfwrv003.web.e.uh.edu:probe:groups:['group:rpm', 'group:linux', 'group:redhat', 'group:redhat-5Server', 'group:redhat-5', 'gr
oup:x86_64']
Generated config for hwfwrv003.web.e.uh.edu in 0.037s
Handled 1 events in 0.000s
Client hwfwrv003.web.e.uh.edu reported state clean
Imported data for hwfwrv003.web.e.uh.edu in 1.60297989845 seconds

I was finally able to track this down to a change in the default kernel configuration used by Ubuntu for ext4 filesystems. The change is detailed at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/588069. Ubuntu apparently decided it was a good idea to turn on barriers by default in 10.04 (Lucid). Luckily, I was able to remount the ext4 partition without barriers (-o barrier=0) and the performance dropped back down to something more reasonable.

hwfwrv003.web.e.uh.edu:probe:current-kernel:['2.6.18-194.26.1.el5']
hwfwrv003.web.e.uh.edu:probe:groups:['group:rpm', 'group:linux', 'group:redhat', 'group:redhat-5Server', 'group:redhat-5', 'gr
oup:x86_64']
Generated config for hwfwrv003.web.e.uh.edu in 0.038s
Handled 1 events in 0.000s
Client hwfwrv003.web.e.uh.edu reported state clean
Imported data for hwfwrv003.web.e.uh.edu in 6.47736501694 seconds

That’s still much slower than ext3, but it’s at least acceptable in this particular case.

While I can understand the reasoning behind changing something like this, it does not appear to be a good idea to drastically reduce the performance of a LTS release without at least warning people VERY LOUDLY.

More information about this can be found at http://lwn.net/Articles/283161/.

I have recently started using ssh multiplexing and thought I’d share this technique with everyone. You will find this technique especially useful in cases where the initial connection negotiation takes longer than expected. You may also find it useful if the remote host forces password authentication and you get tired of typing your password repeatedly.

The first thing you need to do is copy the following lines to your ~/.ssh/config (or the global ssh_config):

Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h:%p

After doing this, go ahead and test that things are working:

solj@abbysplaything $ ssh -f pjacosta-desktop sleep 60
solj@abbysplaything $ ls -l /tmp/solj*
srw------- 1 solj solj 0 Nov 14 13:44 /tmp/solj@pjacosta-desktop:22

Here you can see that a socket has been created for my user which can be reused by any additional ssh connections which go to the same user/host/port combination. Not only does this bypass future negotiations, but it also prevents opening additional connections unnecessarily. This has also really helped out at work because now when I log into a remote machine with an extremely high load, I can simply use the existing connection if I need to open up multiple sessions.

I haven’t found any downsides to using this sort of multiplexing and it certainly has some upside. This feature doesn’t appear to be very popular or publicized, but I think that it provides really useful functionality.

The new Bcfg2 VPS is finally setup and active. After a little bit of tweaking, I was able to get the documentation building from the master git branch every five minutes. The current setup is such that the documentation for the latest stable version of Bcfg2 can be found at http://docs.bcfg2.org/ while the latest development branch documentation is at http://docs.bcfg2.org/dev.

I plan on publishing the configuration for the web server soon, however, I want to do this at the same time as the common Bcfg2 repository so that I can finally resolve both these issues. I’m also thinking that it will probably be a good idea to create a debian repository on the VPS so that we can automate the building of packages. Currently our mirror at http://debian.bcfg2.org is very out of date.

My goal is to try and do most of this during the next code sprint. I have found that these types of things are difficult to work on unless I set a specific time. I am hoping that by next year we will finally have a real working example of a live Bcfg2 repository for people to see and use. I think that having these examples will lower the barrier significantly as new users are often tripped up by minor mistakes which can be mitigated more easily if you begin from a working example.

That’s all for now. More to come once I get started working on this.

I have just started creating a new sample Bcfg2 repository on github. This post details the strategy used in this repository in order to make it easy to pull in updates and merge them seamlessly with your running Bcfg2 instance.

The primary goal for this repository is to be nondestructive when someone tries to pull in the changes from upstream. However, in order to do this, there is a slightly more complicated repository structure involved.

The first thing you’ll probably want to do is grab a copy of the repository:

git clone git://github.com/solj/bcfg2-repo.git

As you inspect the repository, you’ll notice that I have tried to make use of Xinclude in order to prevent overwriting custom repository files. The idea is that you will have 3 separate files (file.xml, upstreamfile.xml, and localfile.xml). The basic format for file.xml will be a comment with information about the purpose of the file. Then it will xinclude localfile.xml and have a comment containing the xinclude for upstreamfile.xml. The reasoning behind this is that you will most often have slight changes in your local repository from what is contained upstream. Therefore, you will most likely populate localfile.xml and merge in any changes from upstreamfile.xml manually.

This appears to be working out well so far (although I haven’t really added enough content yet). I am also considering adding paths to default configuration files (e.g. ssh). I am thinking about populating them using group-specific files so that they can be overridden by adding a group file with a higher priority. I am also considering possibly adding a separate layout under Cfg/TCheetah/TGenshi and using altsrc to bind the entries from the Bundles.

I’m hoping this all works out and any comments/criticisms are welcome. I know that the useful examples out there are sparse and widespread. I’m hoping that we can get something together which allows people to collaborate on useful configurations easily so that the initial barrier to using Bcfg2 is decreased.

The xargs page on wikipedia talks mostly about the benefits of using it when running tasks on large lists. However, I have found it to be useful in other situations as well.

Oftentimes I am tasked with suspending accounts at work due to CPU (over)use. During this process, I frequently come across accounts which are processing Awstats statistics for some excessive number of domains (as this is the default setting in CPanel). This is normally because the number of domains for a particular user is something > 1500. In order to kill off the statistics processing, I usually just find and kill all the processes running for that user. I used to use the following to do that:

for p in `ps aux | grep ^user | awk '{print $2}'`; do kill $p; done

This is actually pretty simple and works quite well. However, it normally requires use of the Home/End keys along with ‘`’ because it’s usually after I get the process listing that I realize awstats is running. This is where xargs comes in. Usually the command typed just before the above is this:

ps aux | grep ^user

From there, it would be nice to awk out the process ids and pipe them to kill. Here is some sample output of what I’m talking about:

$ ps aux | grep ^solj | awk '{print $2}'
2190
2205
2237
2240

So, the result is simply a newline-delimited listing of the various process ids running for a particular user. Note that because I am on Linux, awk is actually gawk so I don’t have to worry about the newline separation being an issue. Therefore, I finally came up with the following elegant solution to my problem:

ps aux | grep ^solj | awk '{print $2}' | xargs kill

As you can see, this makes it extremely simple to go from the previous command to the one needed to kill off all the user’s processes. No special characters modifications at the beginning of the line are necessary. While this may seem like a miniscule amount of time saved, it actually adds up quite a bit with the number of suspensions made in this manner :-) . I look forward to using xargs for inode abuse as well, but we’ll save that for another post…

I recently set up this blog on a virtual machine which is running on my home computer. Since soljerome.com was already running elsewhere, I decided to serve up the content using the existing apache instance at soljerome.com by setting up apache using a reverse proxy.

This particular setup is on Ubuntu 8.04 as that is the distribution running on my web server. My web server is able to view the virtual machine using an internal IP address of 10.10.10.34. Here is the /etc/hosts entry

10.10.10.34 www.solnet www blog.solnet blog

So, I am able to view the blog from the web server by browsing to http://blog.solnet. Therefore, I needed to tell Apache was to take a URL like http://soljerome.com/blog, internally request http://blog.solnet, and give the result back to the viewer.

The first thing I needed to do was install the Apache mod_proxy module

aptitude install libapache2-mod-proxy-html

Then I enabled the proxy module by running

a2enmod proxy_html

After the module was enabled, I added the following lines to /etc/apache2/sites-available/default

    ProxyRequests off    

    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyPass /blog http://blog.solnet
    ProxyVia off

Once I restarted Apache, I was able to browse to http://soljerome.com/blog successfully. There are still a few things that I have yet to get working properly (although I think most are due to bugs in WordPress). Some of the wp-admin links work but redirect to the internal address which forces me to have to click the back button on my browser (annoying). Also, trying to setup the admin interface to use SSL has proved to be a problem.

© 2012 Sol's blog Suffusion theme by Sayontan Sinha
  • Google+
  • LinkedIn