Launching Linux Fireball

Since late 2011, I've found myself doing a lot more Linux administration than Mac. In light of that, I am launching a new Linux blog that will have a lot of overlap with Command Line Mac.

Mostly for my own convenience, I'll be copying a lot of the Unix/Linux only posts over to the new blog. I'll still be posting here when I run across something interesting in the Mac world, but I expect the other blog to be much more active this year.

Adventures with Airport Express

After the failure of my cable TV/Internet service last month, and the slow response of the cable company, I dumped cable in favor of satellite+DSL. It was an inconvenient process to say the least. When it was done, I needed to connect the satellite DVR to the wireless DSL router to enable additional features.

The DVR had a USB wireless adapter, but because of the location of router, the signal was not quite strong enough to make the connection work. I had a spare Airport Express and my first thought was to use it to extend the DSL router network. Before doing any research, I made manual changes to the Airport Express that rendered it a brick. Doh!

I found a helpful article at Apple Support that let me reset it to factory settings. Then, I finally did the research and found that the plain Express can't be used to extend a 2WIRE wireless router.

The next strategy was to set up a separate wireless network and plug the Airport Express into one of the wired ports on the router. The Airport Express provided a stronger signal that the 2WIRE and allowed me to connect the DVR to the Internet. The only modification I had to make from a basic configuration was to turn off network address translation (NAT) on the Airport Express to avoid a double NAT situation since the 2WIRE also provided NAT. The Airport Express actually detected the problem and made the suggestion when I connected to it with the Airport Utility. Apple really has some very nicely thought out software in their hardware.

Ruby: Sorting an array with multiple fields

I was working on a project today and needed to make some changes to the way an array on objects was sorted before being displayed on the screen. The main reason I even bothered to post this is to heap a little more praise on Ruby for working the way I expected it to work.

Here was my object (not tied to a database, just created for convenience):

class Force
attr_accessor :id
attr_accessor :last_name
attr_accessor :first_name
attr_accessor :rank
attr_accessor :shift
attr_accessor :promoted_on
attr_accessor :phone
attr_accessor :totalcount
end

I needed to sort an array of these objects by rank ascending, shift ascending, then promoted_on (date) descending.

I sorted them by concatenating the fields together and making the right comparisons. This only works because the first two fields, rank and shift, cannot be empty. Otherwise, a simple concatenation would create undesired results.

Sorting all fields ascending looks like this:

@forces = @forces.sort {|x,y| x.shift + x.rank + x.promoted_on.to_s <=> y.shift + y.rank + y.promoted_on.to_s}

To sort the promoted_on field in descending order, reverse the x,y sides for that field:

@forces = @forces.sort {|x,y| x.shift + x.rank + y.promoted_on.to_s <=> y.shift + y.rank + x.promoted_on.to_s}

No need for a special sort coding block. Ruby FTW!

Steve Jobs RIP

Steve Jobs RIP

The Chromium OS experiment

Google's Chromium OS

I downloaded a bootable ISO of the Google Chromium OS release candidate. Then, fired it up in VMware Player and took a look around. It appears to be based on OpenSUSE Linux 11.4. SUSE was one of the distributions I cut my teeth on in the Linux world. I always appreciated the 9 pound manual of documentation that came with it. Very helpful for beginners and a good reference to their tools.

I might have made a mistake when setting up the virtual machine because it complained of low memory right after booting. This could also explain the slow performance I experienced. The desktop itself is clean and appears to be optimized to run the Chrome web browser. There was a local word processing application, but I didn't get far enough to test it out, it was just too slow. I could not find a download link at Google, but I found one at a .eu domain. Maybe it was a pirated or tampered version. I am going to postpone further research until I am sure I have a good distribution.

Updated: Turns out Google doesn't provide an image or download of the OS itself, only the source code and compilation instructions. Also, the OS is based on Ubuntu, despite the build I got from Europe based on SUSE. ZD Net suggested that the running OS would be slow so I maybe I am not missing much at this stage of development. I don't want to pay for a Chromebook just to run Chrome. Another reason to keep this on hold for now.

Getting by in Git

Git is the source code management system used for the Linux kernel and many other highly complex projects. It was written by Linux Torvalds after some controversy over the proprietary Bitkeeper program that used to manage the Linux kernel.

I've needed to upgrade my skills recently to use git in place of subversion because that is what my shop decided to use. I've moved all my Rails code into a remote git server and so far, so good.

One improvement is it has fewer "droppings" than subversion. There is no hidden .svn directory in each directory with source code. Only a single .git directory at the root of the project, plus a .gitignore file for files you don't want git to track.

Initialize project tracking
git init

Check out an existing project from remote server
git clone ssh://server/git/project

Add a file for git to track
git add

Add all files from this directory and below for git to track
git add .

Commit all files to local repository
git commit -a -m "message"

Undo changes to a file (re-check out from repository)
git checkout --

Pull files from remote repository and merge with local repository
git pull

Push files to remote repository (must commit first)
git push

Move file or directory to new location
git mv path destination

Remove file or directory from the working tree
git rm path

To create a remote repository from an existing project takes several steps
cd /tmp
git clone --bare /path/to/project (creates a /tmp/project.git directory)
scp project.git to remote server
cd /path/to/project
git remote add origin ssh://server/git/project
git config branch.master.remote origin
git config branch.master.merge refs/heads/master

LVM basics

I've just spent a few hours with iSCSI SAN disks (EqualLogic) and Linux Logical Volume Manager (LVM). The abstraction is even deeper than that, because Linux at work is running under VMware, so it is really VMware talking to the SAN and presenting a SCSI disk to Linux. Since I only get into the LVM weeds a couple of times a year, I thought it would be helpful to list the steps I took to get usable disk space under Linux starting with the raw disk space.

Step One - create a new partition
Create a new partition with FDISK or PARTED. Mark the partition type hex 8E for LVM. In my case, the SCSI disk appeared as /dev/sdb and the partition using all space became /dev/sdb1. LVM is capable of using a raw device (no partition type), but I stayed in familiar partitioning territory.

Step Two - create LVM physical volume
pvcreate /dev/sdb1

Step Three - create LVM volume group in the physical volume
vgcreate new_volume_group /dev/sdb1

Step Four - create LVM logical volume in the volume group
lvcreate --name new_logical_volume --size 100G new_volume_group

Step Five - create a file system on the logical volume
mkfs -t ext4 /dev/mapper/new_volume_group-new_logical_volume
Note: Linux device mapper automatically creates a symlink to the disk in /dev/mapper using the volume group and logical volume names. If you choose more meaningful names than the example, the name won't look so awful.

Step Six - turn off automatic file system checks (optional)
tune2fs -c 0 /dev/mapper/new_volume_group-new_logical_volume

Step Seven - add mount point in /etc/fstab
Once the mount point is list in fstab, mount it manually and it is ready to use.

Chrome Revisited

In my first serious test of Mac Chrome back in August, 2010, it came up short compared to other popular web browsers. But Chrome has been evolving fast. And since Firefox 4.0 was just released, it seemed like the right time to run another informal comparison.

What a difference. Chrome has become more stable, faster, and has made huge strides in available and useful plugins. The plugin gap with Firefox was a glaring issue last time. As a web developer, Firefox was indispensable with the Firebug plugin, letting you drill into the CSS and JavaScript acting on individual DOM components. Chrome now has native developer tools that rival Firebug. I am also fond of the Awesome Screenshot plugin that allows you to capture and annotate a web page from within the browser.

While I have found Firefox 4.0 a nice improvement in looks and rendering speed, I have also found it leaks memory, more so on Windows than Linux or Mac. I can rarely make it through a day without the Windows version locking up. That may be an artifact of the slew of plugins I am running and not the Firefox core.

For the last couple of weeks, I have been using Chrome as my primary browser on all platforms and have been very pleased. Competition is a great thing and I am excited to see browsers evolving again after what seemed like a long period of stagnation.

Android G2 and iTunes music sync

Getting music from iTunes on a Mac onto an Android G2 smart phone was an easy task. Maybe it wasn't so easy with the first generation Android phones, but the G2 I purchased a couple of weeks ago came with everything I needed.

Connecting the phone to the Mac

My main Mac is a two year old MacBook. The G2 came with a USB sync cable and as soon as I attached it, it opened iPhoto to upload pictures just like a camera. But the G2 also displayed a screen to enable it as a USB disk drive. I enabled disk mode, then took a look at the mounted volume in Finder.

One of the icons on the Volume was DoubleTwist. When I double clicked it, it started downloading the latest Mac version of DoubleTwist. After installation, the DoubleTwist interface looked very much like iTunes. It allows you to automatically sync all music from iTunes or just selections.

Since I don't plan to listen to music often on the G2, I chose to only sync playlists. This only syncs the music required in each playlist. There are a lot of options in DoubleTwist and the integration was seamless.

Automating FTP

People frequently need to automate FTP sessions to upload or download files.
Most command line FTP clients, including the FTP client on the Mac, can automatically login to an FTP server by reading the .netrc file in the user home directory. Note that the FTP auto-login file starts with a dot in front of the name (dot netrc).

Syntax of the $HOME/.netrc

The .netrc file can contain more than one auto-login configuration. Each FTP server has a set of commands, the minimum being the login name and password. You can create as many machine sections as you need. Here is a generic example:

machine ftp.server.com
  login myuserID
  password mypassword

Very Important: .netrc permissions!
Since user IDs and passwords are stored in the .netrc file, the FTP client enforces permission checking on it. It must be set so that no groups and no other users can read or write to it. You can set the permissions on it with this command from the Terminal (from your home directory) once the file is created:
chmod 700 .netrc

Adding FTP commands in a BASH script
You can embed FTP commands in a BASH script to upload and download files.
For example, you could create a script file named ftpupload.sh:

#!/bin/bash
# upload a file
/usr/bin/ftp -i ftp.server.com <<ENDOFCOMMANDS
cd backupdir
cd subdir
put datafile
quit
ENDOFCOMMANDS

In this example, I added the -i switch when running FTP to prevent it from prompting on multiple file uploads/downloads, even though it is only uploading one file in the example. I also use the BASH HERE document feature to send commands to FTP. When the script is run, it will auto-login using the information in the .netrc file, change to the right remote directory and upload the datafile.

Scheduling the script with Cron
The last step is to get the BASH script to run unattended, say every day at 5:00 am. The old school UNIX way is to use Cron, but the fancy new Apple way is to use a launchd XML configuration. As long as cron is supported in OS X, I'll stick to the old school way. I leave the launchd configuration as an exercise for the reader.

Add these lines with the command "crontab -e", then save:

# automated FTP upload
0 5 * * * /Users/username/ftpupload.sh

Dumping a Postgresql database remotely with SSH

I ran into a rare problem recently with a large Postgresql database that was filling up the local disks of a server. The database was large, over 100 GB and about 300 million records. There was a lot of churn and it had not been vacuumed in a long time. When I manually ran a vacuum on it, there was not enough working disk space to complete the operation, creating a bind.

What I decided to do instead of using vacuum was to dump it to a remote backup location, then drop the database and restore it from the remote dump. I used SSH to run the remote commands.

Dump a remote Postgresql database to the local machine
ssh user@remote-database-server 'pg_dump database-name -t table-name' > table-name.sql

Restore a remote Postgresql database dump to the local database server
ssh user@backup-machine 'cat table-name.sql' | psql -d database-name

Note that the dump command is run from the backup machine and the restore command is run from the database server.

Also note the single quotes around certain parts of the command.

The Google Chrome experiment

When Google first announced their Chrome browser, packed with a revamped JavaScript engine (V8) and support for offline web apps, I thought I would give it a spin. The first couple of releases were for Windows and Linux -- no Mac version. Those early versions were a little clunky and appeared to offer no better performance than other popular browsers. So I moved on.

When the Mac version became available, it was a much more polished browser. Another theoretical selling point was that each tab ran as a separate process so one crashed tab would not crash the whole browser. I decided to give Chrome a serious work out on my Mac at home.

Things started out well enough and performance was good. I perused the help and learned some of the short cuts. After about three good weeks, something went wrong. I don't know if it was an update, a growing cache, or what, but it started slowing down. Then, it started having problems loading pages from web sites that worked fine in other browsers. It is possible that it even caused wireless network issues, though that is just speculation at the moment. I need to do some more research to see if the problems were related to Chrome.

For now, I am sticking with Firefox as my main browser on the Mac. I'll come back and try Chrome out after the next major release.

Finding IPs connected to your web server

On Mac OS X
note: this also shows outgoing connections from web browsers

Get all IPs connected to your web server:

netstat -nat | sed -n -e '/ESTABLISHED/p' | awk '{print $5}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' | sort

Get all unique IPs connected to your web server:

netstat -nat | sed -n -e '/ESTABLISHED/p' | awk '{print $5}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -n

On Linux
Get all IPs connected to your web server:

netstat -ntu | sed -e 's/::ffff://g' | awk '{print $5}' | cut -d : -f1 | sort -n

Get all unique IPs connected to your web server:

netstat -ntu | sed -e 's/::ffff://g' | awk '{print $5}' | cut -d : -f1 | sort | uniq -c | sort -n

RIP RJD

The Last in Line

The mysterious Data Center Technical Specialist certification

On March 4, I received an email from Novell Technical Training that I had received a Novell certification for "Data Center Technical Specialist". This came as a surprise to me because I had not applied for this certification, taken any tests for this certification, not had I even heardof this certification.

Due to a cross marketing agreement with the Linux Professional Institute, I had applied for and received the Novell Certified Linux Administrator certification a few weeks prior. This seemed legitimate to me, as I had extensive experience with SUSE Linux and my Linux skills are still sharp. I continue to perform Linux server administration as part of my daily work.

However, I am not quite sure what the Data Center Technical Specialist is supposed to represent. Confused, I wrote to Novell Training asking what the certification meant. I received this equally mysterious reply:

Thank you for contacting Novell Training Services. You have received the certification as part of some changes we have made recently to our partner requirements. As part if these changes, some of the exams/certifications you have now count toward the new certification.

I searched the official Novell Certification web site, and this certification does not appear anywhere. I suspect, but can't confirm that it may be part of the Solution Provider program. As such, it is probably more of a value to Novell sales than to an individual technician. I remain somewhat baffled.

Fuzzy string matching in PostgreSQL

A recent project required me to use fuzzy string matching, or sound alike matching, in an application that searched a list of names. It turns out there is a contrib module for the PostgreSQL database called fuzzystrmatch that provides several different matching algorithms.

The task at hand involved rewriting a legacy application, originally in PICK, in Ruby on Rails. The PICK application used a soundex search to find names of people that sounded like the search string.

Three algorithms are available as PostgreSQL functions (after installation of the fuzzystrmatch module). They are soundex(), levenshtein(), and metaphone().

Both soundex and metaphone convert a string into character codes. Soundex uses 4 characters and metaphone uses a configurable number of characters. Levenshtein directly compares two strings and returns an integer indicating how well the two strings match.

After some trial and error, I found that metaphone produced better results than soundex. I didn't test the Levenshtein function.

To improve the results, I added a classic substring search using ILIKE. The combination of ILIKE and metaphone gave me a broad, but reasonably accurate fuzzy string search.

Linux: fuser to find processes on TCP ports

Note: This is for Linux only. The Mac (BSD) version of fuser does not handle TCP/UDP ports.

Once or twice a year, I run into a problem where a process is using a TCP port and I need to find out which one. I am documenting it here for the next time so I don't have to look it up in man pages.

To see all processes, run fuser as root or with sudo.

To list all processes connected to TCP port 22:

fuser -n tcp 22

Rails 2.x scaffolding field types

Dude, where's my CRUD?

One of the powerful features of Rails 1.x was the ability to generate CReate, Update, and Delete (CRUD) admin screens automatically using the scaffolding script built into Rails.

The original scaffolding read the database models and created basic, but usable screens to let you add and edit database records. When the 2.x release of Rails came out, scaffolding lost that power. Now, you have to manually specify each table field and type on the command line when running scaffold. If you don't, the generated screens will be empty.

Worse, a basic reference to all valid field types was missing. Here are all the valid types I have been able to dig up:

string
text (long text, up to 64k, often used for text areas)
datetime
date
integer
binary
boolean
float
decimal (for financial data)
time
timestamp

A mapping of the scaffolding types to data types in corresponding databases can be found on Overooped.

Here is an example of using 2.x scaffolding with data types, run from the Rails application root directory:

ruby script/generate scaffold Modelname name:string title:string employed_on:date remarks:text


Here is an example of using rails 3.x scaffolding with data types, run from the Rails application root directory:

ruby script/rails generate scaffold Modelname name:string title:string employed_on:date remarks:text


You ARE your operating system (among other things)

Whenever you make a choice among products with similar functions, that choice spills over into the realm of social status.

Cars are a common example where social rank often goes with brand, and even within brand, by model, and even within model, by variations, upgrades, and badges. All signify some social status. I became acutely aware of this after purchasing a car that did not fit my image. It was uncomfortable for everyone.

Your operating system

You can see the social aspect tied to an operating system by looking at the Apple "I'm a Mac" campaign, and the weak Microsoft "I'm a PC" campaign response. A choice to run Linux or other operating system also carries connotations and shared group identity. In this sense, you are your operating system.

Your collection of choices

Whether conscious or not, decisions and choices about purchases you make, where choices are available, weave together part of your social tapestry. The schools you attend, where you work, your clothes, your car, where you live, and yes, your operating system. You are those things, at least socially.

The question is, in the context of a consumer society, can a choice be made without the attachment of social status?

Calculating the square root of 2 longhand

There are many numerical methods to calculate square roots. This is the long hand method I learned in junior high. It produces one (accurate) digit at a time, but the working numbers get larger each iteration. Eventually, it bogs down because it is doing math with integers hundreds, then thousands of digits long.

Still, it is fun for tinkering. If you run this Ruby script as is, it will calculate the square root of 2 to 1,000 digits. To adjust the precision, change the number of iterations on this line:

while iterations < 1000

Here is the entire script...

#!/usr/bin/ruby
# calculate a square root of 2 using longhand

def newroot(divisor, doubleroot)
  # calculate new root
  # formula is doubleroot _ * _ = closest to divisor
  # try 9, then 8, then 7 ...

  i = 9
  while i >= 0
    multiple1 = (doubleroot.to_s + i.to_s).to_i
    multiple2 = i
    product = multiple1 * multiple2

    if product <= divisor
      # found new root
      root = i
      # get modulus
      modulus = divisor - product
      break
    end
    i = i - 1
  end


  return root, modulus
end

roots = Array.new

# the nearest root to 2 is 1

root = 1
roots.push(root)
remainder = 2 - root

doubleroot = root * 2
divisor = remainder * 100

iterations = 0

while iterations < 1000
  root,remainder = newroot(divisor, doubleroot)
  roots.push(root)

  # compute new doubleroot 

  return root, modulus
end

roots = Array.new

# the nearest root to 2 is 1

root = 1
roots.push(root)
remainder = 2 - root

doubleroot = root * 2
divisor = remainder * 100

iterations = 0

while iterations < 1000
  root,remainder = newroot(divisor, doubleroot)
  roots.push(root)

  # compute new doubleroot
  doubleroot = roots.to_s.to_i * 2

  # compute divisor
  divisor = remainder * 100

  iterations = iterations + 1

  print "iteration " + iterations.to_s + "\n"
end

print "final roots are " + roots.to_s + "\n"